Multi-view three-dimensional Gaussian densification method and system for adaptive density control

Abstract

The invention belongs to the technical field of three-dimensional scene reconstruction, and particularly discloses a multi-view three-dimensional Gaussian densification method and system for adaptive density control, and the method comprises the following steps: collecting a multi-view original image, and carrying out the preprocessing of the multi-view original image; complexity features are extracted, a pixel-level complexity heat map is generated, and a globally unified three-dimensional complexity field is constructed; performing back projection on the reconstruction residual error, high-frequency inconsistency and depth / geometric consistency cost of each view angle, generating three-dimensional error popularity, determining a candidate newly-added set and a candidate pruned set, generating a weak label to train a lightweight multilayer perceptron classifier, outputting a ternary probability corresponding to newly-added / pruned / maintained, and obtaining a new / pruned / maintained three-dimensional perceptron classifier; and performing Gaussian densification operation on the newly added region. By adopting the technical scheme, fine point adding is carried out on the complex area, effective pruning is carried out on the simple area, and meanwhile, the synthesis quality, the global consistency and the calculation efficiency of the new view angle are improved.

Description

Technical Field

[0001] This invention belongs to the field of 3D scene reconstruction technology, and relates to an adaptive density control multi-view 3D Gaussian densification method and system. Background Technology

[0002] 3D scene reconstruction technology aims to use multi-view images, videos, or sensor data to reconstruct the 3D geometric structure and appearance information of the real world in a computer. It is a core supporting technology in fields such as virtual reality (VR), augmented reality (AR), autonomous driving, and digital twins.

[0003] With the development of deep learning, the representation methods for 3D scene reconstruction have undergone a rapid evolution from explicit representations based on meshes, point clouds, and voxels to implicit neural representations. This development has greatly promoted the accuracy and efficiency of 3D content generation and interaction.

[0004] In recent years, implicit representation methods (such as NeRF—Neural Radiance Fields) have attracted widespread attention due to their high-quality performance in novel perspective synthesis. NeRF implicitly models the color and density of each point in 3D space using a multilayer perceptron (MLP) and utilizes volume rendering equations to generate images from arbitrary perspectives. However, NeRF suffers from slow inference speed and insufficient scalability when dealing with large-scale scenes, which creates a bottleneck in real-time applications and large-scale 3D reconstruction tasks.

[0005] To overcome the shortcomings of NeRF in rendering efficiency, researchers have proposed several improved methods for explicit representation. Among them, 3D Gaussian Splatting (3DGS) is a novel explicit representation method that uses anisotropic 3D Gaussian to fit the scene in space and achieves real-time rendering through rasterization. This method achieves a good balance between rendering speed and quality, supports high frame rate composition of new perspectives, and is particularly suitable for applications such as real-time interaction and dynamic visualization.

[0006] By analyzing existing multi-view 3D reconstruction methods (including NeRF, 3D Gaussian Splatting and its multi-view extension MVGS) and density control techniques based on image complexity modeling, it can be found that they all have certain shortcomings and limitations in terms of density control, complexity utilization, information fusion and resource allocation.

[0007] Traditional single-viewpoint-driven densification mechanisms often rely on the residual of the current viewpoint to determine whether to add Gaussian points. This can lead to overfitting in that viewpoint direction, while exhibiting geometric gaps, structural breaks, or texture blurring in unobserved views. Even in schemes that introduce multi-viewpoint constraints, the densification triggering condition still primarily depends on the residual magnitude, lacking a comprehensive assessment of the scene's geometry and texture detail complexity. This results in insufficient point addition in high-complexity regions and excessive point addition in low-complexity regions, leading to detail loss and wasted computational resources.

[0008] Furthermore, most existing complexity modeling methods remain at the two-dimensional image level, often using features such as the number of segments and the number of categories to measure image complexity. However, these methods have low integration with multi-view 3D projection, and the mapping and utilization of complexity information in 3D space is limited. This disconnect means that densification strategies cannot dynamically adjust according to the 3D complexity of the scene, especially in areas containing reflective surfaces, geometrical abrupt changes, and subtle textures. In these cases, density control strategies often struggle to make optimal judgments, affecting the final reconstruction quality.

[0009] Meanwhile, most existing densification and pruning strategies use uniform global thresholds and rules, lacking a refined discrimination process. This not only easily leads to the accidental deletion of Gaussian points in detailed regions, causing loss of structural details, but also retains a large number of redundant points in flat or simple regions, increasing storage and rendering overhead.

[0010] The multi-view information fusion process also fails to fully consider the differences in information quality among different viewpoints, making it susceptible to interference from low-quality viewpoint noise, thus reducing the stability and detail fidelity of the fusion. In large-scale scenes, the uniform density control strategy can also lead to uneven distribution of Gaussian points, further exacerbating memory consumption and computational pressure. Summary of the Invention

[0011] The purpose of this invention is to provide an adaptive density control method and system for multi-view 3D Gaussian densification, in order to solve the problems of insufficient accuracy and unreasonable resource allocation in existing multi-view 3D reconstruction methods in terms of densification control.

[0012] To achieve the above objectives, the basic solution of this invention is: an adaptive density-controlled multi-view three-dimensional Gaussian densification method, comprising the following steps:

[0013] Acquire raw images from multiple perspectives and perform preprocessing;

[0014] Complexity features are extracted at the two-dimensional image level to generate pixel-level complexity heatmaps, and a globally unified three-dimensional complexity field is constructed through multi-view back-projection fusion.

[0015] The reconstruction residuals, high-frequency inconsistencies, and depth / geometric consistency costs from each perspective are back-projected together to generate a 3D error heatmap.

[0016] Based on the three-dimensional complexity field C 3D Based on the 3D error heat H, candidate new set and candidate pruning set are determined, and weak labels are generated for supervised learning;

[0017] A lightweight multilayer perceptron classifier is trained using weak labels, and the output ternary probabilities correspond to "addition / pruning / preservation". Gaussian density is then applied to the newly added regions.

[0018] The working principle and beneficial effects of this basic scheme are as follows: This technical scheme extracts complexity features such as the number of segments, the number of categories, and high-frequency textures at the two-dimensional image level, generates a pixel-level complexity heatmap, and constructs a globally unified three-dimensional complexity field through multi-view back-projection fusion. In the process of densification and pruning decisions, this complexity field directly serves as the basis for density scheduling, enabling complex regions to obtain more Gaussian points and simple regions to be effectively compressed, thereby achieving refined spatial resource allocation and improving accuracy.

[0019] Furthermore, the method for preprocessing the original images from multiple perspectives is as follows:

[0020] The set of viewpoint images is V = {v}, where each viewpoint provides a color image Iv, as well as camera intrinsic parameters Kv and extrinsic parameters (R). v ,t v );

[0021] The 3D scene uses an anisotropic 3D Gaussian set. It means that a single Gaussian G i The parameters include the center position. Symmetric positive definite covariance Color / radiance parameter c i (d) and opacity α i ∈(0,1), c i (d) can be represented by constant color or low-order spherical harmonics based on the viewing direction d;

[0022] The rendering of pixel p at viewpoint v employs 2D Gaussian rasterization and forward composite, making w i,v (p) represents the two-dimensional Gaussian weights projected onto the pixels, then the color composite image... for:

[0023]

[0024] Where p represents a two-dimensional pixel, T i-1,v (p) represents the transmittance up to the (i-1)th Gaussian, reflecting the degree to which pixel p is absorbed by the preceding Gaussian, with an initial value T. 0,v(p) = 1, d v,p This represents the line of sight from camera v to pixel p, where k is the k-th pixel. <i;α k,v (p) is defined the same as α i,v (p); α i,v (p)=1-exp(-τw i,v (p)) represents the opacity contribution of the i-th Gaussian at pixel p, ranging from [0,1]; τ>0 represents the mapping of the medium accumulation coefficient, w i,v Let i represent the two-dimensional weights of i Gaussian vectors projected onto pixel p from viewpoint v;

[0025] The two-dimensional weights are obtained by linearizing the projection of the three-dimensional ellipsoid onto the imaging plane, with the projection center denoted as μ′. i,v The projection covariance is Σ′ i,v ,but:

[0026]

[0027] Preprocessing of the original images from multiple perspectives ensures high-speed synthesis on modern GPUs using a rasterization pipeline, providing a computational foundation for subsequent large-scale adaptive density control.

[0028] Furthermore, the following method is used to extract complexity features at the two-dimensional image level, generate pixel-level complexity heatmaps, and construct a globally unified three-dimensional complexity field through multi-view backprojection fusion:

[0029] In each viewpoint, the preprocessed original image is subjected to silent segmentation and named category recognition, yielding the number of segmented regions num_seg(v) and the number of categories num, respectively. class(v) ;

[0030] Interpretable complexity metrics are used to generate complexity scalars and complexity heatmaps for each perspective, providing perspective-level complexity. Defined as:

[0031]

[0032] Where w1, w2>0 are weights, and the least squares method is used to fit the data on the representative validation set.

[0033] Pixel-level complexity heatmap C v (p) integrates segmentation boundary density, semantic category density, and high-frequency information in the frequency domain, and is specifically constructed as follows:

[0034] C v (p)=λ s E v (p)+λ f H v (p)+λ l Lv (p),

[0035] Among them, E v (p) represents the edge saliency obtained from the gradient field of the segmentation boundary, H v (p) represents the high-frequency intensity derived by wavelet or Sobel, L v (p) is the Laplace difference plot, λ s ,λ f ,λ l ≥0 is the weight and is normalized;

[0036] The full-graph level complexity heatmap C v Back-projecting to a unified world coordinate system, the depth D can be obtained using known camera interior and exterior parameters. v (p) projects pixel p back onto a 3D point. Complexity fields are formed in the three-dimensional domain using voxel stacks or kernel density estimation.

[0037]

[0038] Where Q is the metric tensor, Z is the normalization factor; the set of viewpoint images is V = {v}; Ω v Let θ represent the set of pixels at viewpoint v, i.e., the index range of all pixels on the imaging plane of camera v; Θ(·) represents the kernel function (such as the Gaussian kernel or the Epanechnikov kernel), used to smoothly propagate the complexity value corresponding to the pixel to the three-dimensional space; X represents the point X enumerated in the entire three-dimensional point set, ||XX v,p || Q : 3D point X and 3D point X obtained by back projection of pixel p v,p The distance, with the weighted norm of the metric tensor Q; X v,p This represents the coordinates of a 3D point in 3D space projected from pixel p in the camera's viewpoint v. This represents the transpose of the rotation matrix of the camera viewpoint v (3×3, which transforms world coordinates to camera coordinates); This represents the inverse of the intrinsic parameter matrix of the camera viewpoint v (3×3, containing focal length and principal point coordinates for the transformation from pixel coordinates to camera normalized coordinates); Represents pixel p = (u, v) in homogeneous coordinate form; D v (p) represents the depth value of pixel p at the viewpoint v, and represents the distance from the camera center along the ray from that pixel to the scene surface; t v The translation vector (3×1, representing the camera's position in the world coordinate system) represents the camera's viewpoint v.

[0039] The three-dimensional complexity field measures the "modeling difficulty" at various points in the global coordinate space, providing accurate signals for CRD and subsequent discrimination.

[0040] Furthermore, the reconstruction residuals, high-frequency inconsistencies, and depth / geometric consistency costs from each perspective are back-projected together to generate a 3D error heatmap, specifically:

[0041] Define the pixel reconstruction residual under viewpoint v as Define structural consistency penalty s v (p), then the three-dimensional error heat for:

[0042]

[0043] Where p represents a two-dimensional pixel; with I v These represent the synthesized and ground truth images, respectively; Q is the metric tensor; β1 and β2 represent non-negative weight coefficients, which control the residual term r, respectively. v (p) and consistency term s v (p); Θ(·) represents the kernel function, X v,p s represents a three-dimensional point. v (p): The structural consistency penalty term for pixel p under viewpoint v, such as gradient direction difference, normal consistency error or multi-disparity cost; X: The point X enumerated in the entire 3D point set.

[0044] A complexity-based consistency screening is performed, and the reconstruction residuals, high-frequency inconsistencies, and depth / geometric consistency costs from each perspective are back-projected together to form a multi-source 3D error stack.

[0045] Furthermore, based on the three-dimensional complexity field C 3D The method for determining the candidate addition set and candidate pruning set based on the 3D error heat H, and generating weak labels for supervised learning, is as follows:

[0046] When the three-dimensional complexity field C 3D When the 3D error heat H and the Gaussian mass both reach a local maximum in a certain spatial neighborhood, they are considered candidate densification locations, triggering the addition of a new Gaussian.

[0047] Pruning is performed on Gaussians that contribute very little to the synthesis while maintaining visual quality.

[0048] In the process of densification and pruning decisions, the complexity field and the three-dimensional error heat directly serve as the basis for density scheduling, enabling complex regions to obtain more Gaussian points and simple regions to be effectively compressed, thereby achieving refined spatial resource allocation.

[0049] Furthermore, when a new Gaussian is triggered, two strategies are employed:

[0050] One method is splitting, which involves selecting the Gaussian G with the greatest influence near the candidate position. i And split it into several sub-Gaussians The center and covariance of the sub-Gaussian are as follows Initialization, ∑ k ρ k =1, while maintaining the conservation of color and opacity. Where μ i ,Σ i ,α i These are Gaussian center, covariance, and opacity, respectively; among them, This represents the center position of the k-th sub-Gaussian, calculated by adding an offset δμ to the original center μi. (k) We obtain; μi represents the center position (three-dimensional spatial coordinates) of the original Gaussian Gi; δμ (k) This represents the offset of the sub-Gaussian relative to the original center, used to split the original Gaussian so that the sub-Gaussian covers different spatial sub-regions; Let the covariance matrix of the k-th sub-Gaussian be represented by a scaling factor ρ. k ρ is obtained by scaling the original covariance. k This represents the non-negative scaling factor, used to distribute the variance range of the original Gaussian to each sub-Gaussian; the constraint condition is ∑ k ρ k =1, ensuring the conservation of population variance; Σ i Represents the covariance matrix of the original Gaussian, describing its shape and scale; Let the opacity of the k-th sub-Gaussian be such that it satisfies the following condition: That is, the total opacity is approximately conserved before and after the split; α i This represents the opacity of the original Gaussian shape, controlling its contribution to rendering compositing.

[0051] The second method is injection, which involves directly injecting into the argmax value. X A new Gaussian is placed at H(X) along the principal curvature direction. The covariance is initialized by slightly stretching along the 3D complexity gradient direction to capture slender structures, and local density constraints are introduced. when When the upper limit is exceeded, new additions are suppressed, forming spatial self-conservation. X: the point X enumerated in the entire three-dimensional point set, ∈ represents the neighborhood threshold, which is used to define the criterion of proximity to a certain Gaussian.

[0052] A local density constraint mechanism is introduced during the densification process to monitor the Gaussian density of the neighborhood in real time. When the density exceeds the set upper limit, new additions are suppressed. In the training convergence phase, a global pruning and covariance shrinkage are performed in conjunction with a sparse regularization strategy, thereby effectively reducing memory usage and rendering computation while maintaining visual quality.

[0053] Furthermore, the specific method for pruning Gaussians that contribute minimally to synthesis while maintaining visual quality is as follows:

[0054] A multi-view cross-ray densification mechanism is introduced to uniformly project high-error regions from multiple views onto three-dimensional space, defining the average visibility—contribution γ—of a single Gaussian at viewpoint v. i,v ,for:

[0055]

[0056] Among them, Ω v T represents the set of pixels at viewpoint v, that is, the index range of all pixels on the imaging plane of camera v. i-1,v (p) represents the transmittance up to the (i-1)th Gaussian digit at pixel p; α i,v (p) represents the opacity of the i-th Gaussian at pixel p in viewpoint v, with a value range of [0,1]; p represents a two-dimensional pixel, and the viewpoint image set is V={v};

[0057] When average visibility contribution Continuously below the threshold and 3D complexity Simultaneously, if the value is low and there is high redundancy around the Gaussian, pruning is performed.

[0058] To avoid accidentally deleting boundary details, a covariance anisotropy ratio is introduced. Consistency score ζ of the boundary i , in η i or ζ i When the threshold is high, the retention threshold is increased to reflect the design principle of "retaining the boundary first".

[0059] A multi-view cross-ray densification mechanism is introduced to uniformly project high-error regions from multiple views onto three-dimensional space. By detecting spatial intersections, regions that are difficult to reconstruct from all perspectives are selected. Only in these regions is the operation of adding Gaussian points performed, which fundamentally reduces the overfitting and redundancy problems caused by single-view driven densification.

[0060] Furthermore, a lightweight multilayer perceptron classifier is:

[0061] Its input is a single Gaussian multidimensional statistical eigenvector z. i :

[0062]

[0063] in, and These are multi-view weighted statistics of the average reconstruction residual and consistency cost within the Gaussian projection influence domain, respectively; μ i ,Σ i These are the Gaussian center and covariance, respectively. Contribution to average visibility; η i ζ is the Gaussian anisotropy ratio; i tr represents the boundary consistency score; tr represents the trace of the covariance matrix, which is equal to the sum of the variances of the three principal directions, i.e., the size of the Gaussian global scale; det represents the determinant of the covariance matrix, which is equal to the volume scale, i.e., the volume of the three-dimensional space covered by the Gaussian.

[0064] The multilayer perceptron classifier outputs a ternary probability f. θ (z i )∈Δ 2 Corresponding to "addition / pruning / preservation", the training is jointly performed using cross-entropy loss and delay consistency regularization term:

[0065]

[0066] Among them, y i The pseudo-labels are generated jointly by coarse discrimination and performance rewards, λ d Smoothing the decision time is controlled to improve training stability; i represents the Gaussian index, which represents the summation over all training samples; The training loss function for the classifier in the fine-discrimination stage consists of a cross-entropy term and a consistency regularization term; λ d This represents the regularization weight coefficient, which controls the importance of the delay consistency regularization term in the total loss. For the feature representation of the same Gaussian at iteration step t; This indicates that the expectation of the loss function is taken over all samples to obtain the overall training objective; z i Let θ represent the feature representation of the i-th Gaussian; θ represents the set of trainable parameters of the layer perceptron classifier MLP; l represents the classification loss for a single sample.

[0067] In the early stages of training, coarse discrimination is used to achieve fast and high-recall candidate selection, while in the later stages, an MLP classification model trained with multidimensional statistical features is used to achieve high-precision decision-making, thus achieving a balance between detail preservation and redundancy reduction.

[0068] Furthermore, when training the lightweight multilayer perceptron classifier, a multi-view weighted fusion method based on information quality is adopted to calculate the gradient strength and high-frequency energy of each view and dynamically allocate the view weights λ. v ,for:

[0069]

[0070] Wherein, the set of viewpoint images is V = {v}, and u represents the index symbol, which represents any viewpoint in the set V and is used for denominator normalization; The power amplification of the viewpoint quality score is used to highlight high-quality views and suppress low-quality views in multi-view weighted fusion; τ≥1 is the weight amplification factor, which controls the tilt towards high-information views. The overall training objective for each viewpoint consists of luminosity, consistency, and regularization terms.

[0071]

[0072] By explicitly coupling two-dimensional complexity measurement with three-dimensional density control, the question of "in which regions to increase / decrease Gaussian" is transformed from a heuristic into an optimizable decision problem. The training objective is then written as an energy functional with complexity weights and sparse regularization.

[0073]

[0074] in, For the depth approximation obtained from synthesis, This represents a function that controls the covariance from diverging. It is a sparsity constraint regularization that penalizes the opacity αi of all Gaussians, prompting training to automatically discard redundant Gaussians. Its weighted form encourages sparsity. The constant coefficients α1 to α5 focus on global structure in the early stage of training, and increase the weight of perception and sparsity regularization in the later stage to focus on details and compression. α6 is a weighted complexity regularization. with I v These are the synthetic and ground truth images, respectively. D v For the corresponding image depth; It is a density penalty / incentive term driven by complexity (with a negative gradient in the high complexity region and a positive gradient in the low complexity region); The overall training objective function is represented by a weighted combination of photometric, consistency, depth supervision, and regularization terms. SSIM is the structural similarity index, which measures the structural consistency between synthetic and real images. This represents the covariance regularization term, which constrains the Gaussian covariance matrix to prevent it from diverging and avoids the infinite expansion of Gaussian points. Represents the sparsity regularization term, based on Gaussian point opacity α. i This encourages the model to automatically compress redundant Gaussian points, making the representation more compact.

[0075] In the process of Gaussian parameter optimization, this invention adopts a multi-view weighted fusion method based on information quality. By calculating the gradient intensity and high-frequency energy of each viewpoint, the viewpoint weights are dynamically allocated, so that the viewpoint with rich texture and clear edges contributes more to the optimization, while the interference of low-quality viewpoints is suppressed, effectively improving the detail fidelity and fusion robustness.

[0076] Furthermore, when training a lightweight multilayer perceptron classifier, the influence of the three-dimensional complexity field on a single Gaussian is defined as the weighted integral of its support domain. for:

[0077]

[0078] in, For a three-dimensional complexity field, μ i ,Σ i These are the Gaussian center and covariance, respectively. Let X represent the effective complexity value of the i-th Gaussian point, where X is the point X enumerated in the entire 3D point set, and φ is the value of the effective complexity value. i (X) represents μ i ,Σ i is the three-dimensional Gaussian distribution kernel function with parameters, describing the influence range of the Gaussian point iii on its neighborhood in space; det represents the determinant of the covariance matrix, which is equal to the volume scale, i.e., the three-dimensional spatial volume covered by the Gaussian, and i is the index of the Gaussian point;

[0079] Numerically, it is approximated using importance sampling or ellipsoidal simplification, and... After combining the statistics, the Gaussian-level trigger function ψ is given. i :

[0080]

[0081] Where, ρ i Let σ be the neighborhood redundancy, and α be the Sigmoid function. c α r α γ α ρ >0, α c Complexity features The weights; α r Reconstructing residuals The weights; α γ Visibility contribution The weights; α ρ Neighborhood redundancy ρ i The weights; This represents the average reconstruction residual within the influence domain of the Gaussian projection; Contribution to average visibility; ρ i Neighborhood redundancy;

[0082] When ψ i Exceeding the dynamic threshold τ t At that time, enter the new candidate list;

[0083] When ψ i Below τ p and Smaller ones are then considered for pruning;

[0084] The threshold is used to perform annealing scheduling according to the training step. In the early stage, exploration and addition are encouraged, and in the later stage, it tends to be stable and convergent.

[0085] In order to enable complexity cues to directly affect Gaussian-level density control, the influence of the three-dimensional complexity field on a single Gaussian is defined as the weighted integral of its supporting domain.

[0086] The present invention also provides an adaptive density control multi-view 3D Gaussian densification system based on the method described in the present invention, comprising a data acquisition and preprocessing module, a 3D complexity field construction and fusion module, a coarse discrimination driven CRD, a fine discrimination MLP classifier, and a multi-view weighted optimization and rendering output module connected in sequence.

[0087] The data acquisition and preprocessing module is used to acquire raw images from multiple perspectives and perform preprocessing.

[0088] The three-dimensional complexity field construction and fusion module is used to construct a globally unified three-dimensional complexity field and generate a three-dimensional error heat.

[0089] The coarse-discriminative cross-ray denser CRD and the fine-discriminative multilayer perceptron (MLP) classifier are used to generate weak labels for supervised learning.

[0090] The multi-view weighted optimization and rendering output module is used to train a lightweight multilayer perceptron classifier using weak labels, outputting ternary probabilities corresponding to "addition / pruning / preservation", and performing Gaussian density operation on the added region.

[0091] This system combines complexity modeling, a two-stage discrimination mechanism, and a multi-view weighted fusion strategy to achieve precise point addition for complex regions and effective pruning for simple regions, thereby simultaneously improving the quality of new perspective synthesis, global consistency, and computational efficiency. Attached Figure Description

[0092] Figure 1 This is a flowchart illustrating the multi-view three-dimensional Gaussian denserization method with adaptive density control according to the present invention. Detailed Implementation

[0093] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0094] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0095] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0096] This invention discloses an adaptive density-controlled multi-view 3D Gaussian densification method. Based on scene complexity, it combines complexity modeling, a two-stage discrimination mechanism, and a multi-view weighted fusion strategy to achieve precise point addition in complex regions and effective pruning in simple regions, thereby simultaneously improving the synthesis quality of new perspectives, global consistency, and computational efficiency. Figure 1 As shown, the adaptive density-controlled multi-view 3D Gaussian densification method includes the following steps:

[0097] Acquire raw images from multiple perspectives and perform preprocessing;

[0098] Complexity features are extracted at the two-dimensional image level to generate pixel-level complexity heatmaps, and a globally unified three-dimensional complexity field is constructed through multi-view back-projection fusion.

[0099] The reconstruction residuals, high-frequency inconsistencies, and depth / geometric consistency costs from each perspective are back-projected together to generate a 3D error heatmap.

[0100] Based on the three-dimensional complexity field C 3D Based on the 3D error heat H, candidate new set and candidate pruning set are determined, and weak labels are generated for supervised learning;

[0101] A lightweight multilayer perceptron classifier is trained using weak labels, and the output ternary probabilities correspond to "addition / pruning / preservation". Gaussian density is then applied to the newly added regions.

[0102] This invention achieves adaptive density control—"adding points where they should be added"—without altering the advantages of real-time 3D Gaussian rendering, through structured modular design, continuous 3D complexity field construction, density triggering combined with CRD, and fine discrimination transitioning from heuristic to learning-based methods.

[0103] Specifically, regions with high complexity and large errors achieve a more refined Gaussian distribution, while simpler regions are effectively pruned. This results in reconstructions that outperform existing methods in both subjective aesthetics and objective metrics, while significantly reducing redundant computation and memory consumption. This method can be implemented independently as a complete end-to-end system or embedded as a plug-in module into existing 3DGS / MVGS training pipelines, enabling smooth upgrades and engineering deployments without altering the core rendering and optimization interfaces.

[0104] In a preferred embodiment of the present invention, the method for preprocessing the multi-view original image is as follows:

[0105] Assume there are several calibrated imaging viewpoints, denoted as the viewpoint image set V = {v}, where each viewpoint provides a color image Iv, as well as camera intrinsic parameters Kv and extrinsic parameters (R). v , t v );

[0106] The 3D scene uses an anisotropic 3D Gaussian set. It means that a single Gaussian G i The parameters include the center position. Symmetric positive definite covariance Color / radiance parameter c i (d) and opacity α i ∈(0,1), c i (d) can be represented by constant color or low-order spherical harmonics based on the viewing direction d;

[0107] The rendering of pixel p at viewpoint v employs 2D Gaussian rasterization and forward composite, making w i,v (p) represents the two-dimensional Gaussian weights projected onto the pixels, then the color composite image... for:

[0108]

[0109] Where p represents a two-dimensional pixel, T i-1,v (p) represents the transmittance up to the (i-1)th Gaussian, reflecting the degree to which pixel p is absorbed by the preceding Gaussian, with an initial value T. 0,v (p) = 1, d v,p This represents the line of sight from camera v to pixel p, where k is the k-th pixel. <i;α k,v (p) is defined the same as α i,v (p); α i,v (p)=1-exp(-τw i,v (p)) represents the opacity contribution of the i-th Gaussian at pixel p, ranging from [0,1]; τ>0 represents the mapping of the medium accumulation coefficient, wi,v Let i represent the two-dimensional weights of i Gaussian vectors projected onto pixel p from viewpoint v;

[0110] The two-dimensional weights are obtained by linearizing the projection of the three-dimensional ellipsoid onto the imaging plane, with the projection center denoted as μ′. i,v The projection covariance is Σ′ i,v ,but:

[0111]

[0112] This explicit representation ensures high-speed synthesis on modern GPUs using rasterization pipelines, providing a computational foundation for subsequent large-scale adaptive density control.

[0113] In a preferred embodiment of the present invention, the method for extracting complexity features at the two-dimensional image level, generating a pixel-level complexity heatmap, and constructing a globally unified three-dimensional complexity field through multi-view back-projection fusion is as follows:

[0114] In terms of complexity modeling, the preprocessed original image is subjected to promptless segmentation and named category recognition from each viewpoint (promptless segmentation refers to the automatic division of the input image into several semantically reasonable regions without the need for manual "hints" or "prior information" (such as target location, category label, mask). Here, the preprocessed image is automatically segmented into several candidate regions to obtain the number of regions and boundary information, which are used to measure image complexity (e.g., "more segmentation blocks → higher complexity").

[0115] Nameable category recognition: This refers to mapping segmented regions or the entire image to interpretable category names using a semantic recognition model. A common method is the visual-language model, such as CLIP, which aligns the semantic space of an image with that of natural language. Here, it refers to counting the number of semantic categories appearing in the image (as another indicator of complexity), obtaining the number of segmented regions num_seg(v) (the number of segmented regions of the image at viewpoint v is obtained using a SAM model) and the number of categories num. class(v) (The number of semantic categories appearing under viewpoint v is obtained by segmentation using the CLIP model);

[0116] Based on prior knowledge, an interpretable complexity metric is used to generate a complexity scalar (which represents the number of image segments and semantic categories at a given viewpoint, used to measure the overall complexity of that viewpoint) and a complexity heatmap for each viewpoint, representing viewpoint-level complexity. Defined as:

[0117]

[0118] Where w1, w2>0 are weights, and the least squares method is used to fit the data on the representative validation set.

[0119] Pixel-level complexity heatmap C v (p) integrates segmentation boundary density, semantic category density, and high-frequency information in the frequency domain, and is specifically constructed as follows:

[0120] C v (p)=λ s E v (p)+λ f H v (p)+λ l L v (p),

[0121] Among them, E v (p) represents the edge saliency obtained from the gradient field of the segmentation boundary, H v (p) represents the high-frequency intensity derived by wavelet or Sobel, L v (p) is the Laplace difference plot, λ s ,λ f ,λ l ≥0 is the weight and is normalized;

[0122] To enable complexity to guide 3D density control, the full-graph level complexity heatmap C is used. v (C v It is C v (p) is a collective term for the set of coordinates. The depth D can be obtained by back-projecting the data into a unified world coordinate system using known camera interior and exterior parameters. v (p) (e.g., obtained from multi-view stereo, depth networks, or rendering depth approximation), projecting pixel p back onto a 3D point. Complexity fields are formed in the three-dimensional domain using voxel stacks or kernel density estimation.

[0123]

[0124] Where Q is the metric tensor, Z is the normalization factor; the set of viewpoint images is V = {v}; Ω v Let θ represent the set of pixels at viewpoint v, i.e., the index range of all pixels on the imaging plane of camera v; Θ(·) represents the kernel function (such as the Gaussian kernel or the Epanechnikov kernel), used to smoothly propagate the complexity value corresponding to the pixel to the three-dimensional space; X represents the point X enumerated in the entire three-dimensional point set, ||XX v,p || Q : 3D point X and 3D point X obtained by back projection of pixel p v,p The distance, with the weighted norm of the metric tensor Q; X v,p This represents the coordinates of a 3D point in 3D space projected from pixel p in the camera's viewpoint v. This represents the transpose of the rotation matrix of the camera viewpoint v (3×3, which transforms world coordinates to camera coordinates); This represents the inverse of the intrinsic parameter matrix of the camera viewpoint v (3×3, containing focal length and principal point coordinates for the transformation from pixel coordinates to camera normalized coordinates); Represents pixel p = (u, v) in homogeneous coordinate form; D v (p) represents the depth value of pixel p at the viewpoint v, and represents the distance from the camera center along the ray from that pixel to the scene surface; t v This represents the translation vector of the camera's viewpoint v (3×1, representing the camera's position in the world coordinate system). This three-dimensional complexity field measures the "modeling difficulty" at various points in the global coordinate system, providing accurate signals for CRD and subsequent discrimination.

[0125] In a preferred embodiment of the present invention, the reconstruction residuals, high-frequency inconsistencies, and depth / geometric consistency costs from each viewpoint are back-projected together to generate a three-dimensional error heatmap, specifically as follows:

[0126] In the densification stage, the present invention first performs a complexity-based consistency screening, which back-projects the reconstruction residuals, high-frequency inconsistencies and depth / geometric consistency costs from each viewpoint to form a multi-source 3D error stack.

[0127] Define the pixel reconstruction residual under viewpoint v as Define structural consistency penalty s v (p) (which can be estimated by gradient direction consistency, normal consistency, or parallax consistency), then the 3D error heat... for:

[0128]

[0129] Where p represents a two-dimensional pixel; with I v These represent the synthesized and ground truth images, respectively; Q is the metric tensor; β1 and β2 represent non-negative weight coefficients, which control the residual term r, respectively. v (p) and consistency term s v (p); Θ(·) represents the kernel function, X v,p s represents a three-dimensional point. v (p): The structural consistency penalty term for pixel p under viewpoint v, such as gradient direction difference, normal consistency error or multi-disparity cost; X: The point X enumerated in the entire 3D point set.

[0130] In a preferred embodiment of the present invention, based on the three-dimensional complexity field C 3DUsing the 3D error heat H, candidate addition and pruning sets are determined, and weak labels are generated (using the 3D complexity field and 3D error heat, addition / pruning / retention labels for candidate points are automatically generated through thresholds / rules; these automatically generated labels are the weak labels, which are then used to supervise the training of the fine discriminant network). The method for supervised learning is as follows:

[0131] When the three-dimensional complexity field C 3D When the 3D error heat H and the Gaussian mass both reach a local maximum in a certain spatial neighborhood, they are considered candidate densification locations, triggering the addition of a new Gaussian.

[0132] Pruning is performed on Gaussians that contribute very little to the synthesis while maintaining visual quality.

[0133] This mechanism is equivalent to the continuum realization of "cross-ray densification": only those spatial points that exhibit high complexity and high error under the combined effect of multiple perspectives are allowed to trigger the addition of new Gaussians.

[0134] In the process of densification and pruning decisions, the complexity field and the three-dimensional error heat directly serve as the basis for density scheduling, enabling complex regions to obtain more Gaussian points and simple regions to be effectively compressed, thereby achieving refined spatial resource allocation.

[0135] In a preferred embodiment of the present invention, two strategies are employed when a new Gaussian is triggered:

[0136] One method is splitting, which involves selecting the Gaussian atom G that has the greatest impact near the candidate position (identifying the Gaussian atom G that contributes the most to the rendering result or error of the pixel / region near the candidate position). i This is treated as a splitting object. This can be measured using Gaussian G(s) (based on pixel contribution weights, error gradient magnitude, or cumulative residual contribution). i And split it into several sub-Gaussians The center and covariance of the sub-Gaussian are as follows Initialization, ∑ k ρ k =1, while maintaining the conservation of color and opacity. Where μ i ,Σ i ,α i These are Gaussian center, covariance, and opacity, respectively; among them, This represents the center position of the k-th sub-Gaussian, calculated by adding an offset δμ to the original center μi. (k) We obtain; μi represents the center position (three-dimensional spatial coordinates) of the original Gaussian Gi; δμ (k) This represents the offset of the sub-Gaussian relative to the original center, used to split the original Gaussian so that the sub-Gaussian covers different spatial sub-regions; Let the covariance matrix of the k-th sub-Gaussian be represented by a scaling factor ρ. k ρ is obtained by scaling the original covariance. k This represents the non-negative scaling factor, used to distribute the variance range of the original Gaussian to each sub-Gaussian; the constraint condition is ∑ k ρ k =1, ensuring the conservation of population variance; Σ i Represents the covariance matrix of the original Gaussian, describing its shape and scale; Let the opacity of the k-th sub-Gaussian be such that it satisfies the following condition: That is, the total opacity is approximately conserved before and after the split; α i This represents the opacity of the original Gaussian shape, controlling its contribution to rendering compositing.

[0137] The second method is injection, which involves directly injecting into the argmax value. X A new Gaussian is placed at H(X) along the principal curvature direction, and the covariance initialization is slightly stretched along the 3D complexity gradient direction to capture the slender structure. To avoid over-densification, a local density constraint is introduced. when When the upper limit is exceeded, new additions are suppressed, forming spatial self-conservation. X: the point X enumerated in the entire three-dimensional point set, ∈ represents the neighborhood threshold, which is used to define the criterion of proximity to a certain Gaussian.

[0138] A local density constraint mechanism is introduced during the densification process to monitor the Gaussian density of the neighborhood in real time. When the density exceeds the set upper limit, new additions are suppressed. In the training convergence phase, a global pruning and covariance shrinkage are performed in conjunction with a sparse regularization strategy, thereby effectively reducing memory usage and rendering computation while maintaining visual quality.

[0139] In a preferred embodiment of the present invention, the specific method for pruning Gaussians that contribute minimally to the synthesis while maintaining visual quality is as follows:

[0140] The pruning dual to densification needs to remove Gaussians that contribute very little to the synthesis while maintaining visual quality. A multi-view cross-ray densification mechanism is introduced to uniformly project high-error regions from multiple views onto three-dimensional space. The average visibility—contribution γ—of a single Gaussian at viewpoint v is defined. i,v ,for:

[0141]

[0142] Among them, Ω v T represents the set of pixels at viewpoint v, that is, the index range of all pixels on the imaging plane of camera v. i-1,v (p) represents the transmittance up to the (i-1)th Gaussian digit at pixel p; α i,v(p) represents the opacity of the i-th Gaussian at pixel p in viewpoint v, with a value range of [0,1]; p represents a two-dimensional pixel, and the viewpoint image set is V={v};

[0143] When average visibility contribution Continuously below the threshold ( This represents the average visibility contribution of Gaussian scalars across multiple viewpoints. A threshold is set, such as 0.05 or 0.1, indicating that if a Gaussian scalar is almost invisible in most viewpoints (meaning a very low average visibility contribution), it is considered to have limited contribution to the rendering result and can be marked as a candidate for pruning. (And 3D complexity...) At the same time, it has a lower (average visibility contribution) A lower value indicates that this Gaussian is almost invisible from most viewpoints, contributing little and having virtually no impact on the final rendering. Complexity A lower value indicates that the spatial region containing this Gaussian point has relatively simple geometry and texture, lacking complex details. Even if the Gaussian point is removed, the impact on visual quality is minimal. Simultaneously, a lower value means that the Gaussian point is considered a "redundant point" and pruning is only performed when both conditions are met. Furthermore, a higher redundancy exists around the Gaussian point (using neighborhood statistics, defining the neighborhood radius ε, and counting the Gaussian center μ). i The number of Gaussians in the vicinity is set to 10 in the experiment, which is the redundancy of the number of Gaussians within a certain neighborhood radius. The neighborhood radius is set using the relative scaling method, which is generally set to 1.5 times the square root of the Gaussian radius. Pruning is then performed.

[0144] To avoid accidentally deleting boundary details, a covariance anisotropy ratio is introduced. Consistency score ζ of the boundary i (by E) v (obtained from projection statistics), in η i or ζ i When the threshold is high, the retention threshold is increased (an adaptively increased lower limit for pruning, used to force an increase in the survival probability in boundary regions or in cases of high anisotropy Gaussian, thereby realizing the design idea of ​​boundary-priority retention), reflecting the design of "boundary-priority retention".

[0145] A multi-view cross-ray densification mechanism is introduced to uniformly project high-error regions from multiple views onto three-dimensional space. By detecting spatial intersections, regions that are difficult to reconstruct from all perspectives are selected. Only in these regions is the operation of adding Gaussian points performed, which fundamentally reduces the overfitting and redundancy problems caused by single-view driven densification.

[0146] In a preferred embodiment of the present invention, the discrimination process employs a two-stage collaborative mechanism of "coarse-fine". The coarse discrimination stage is based on... Joint peak detection with H provides candidate addition and candidate pruning sets, and generates weak labels for supervised learning; the fine discrimination stage trains a lightweight multilayer perceptron classifier f. θ .

[0147] A lightweight multilayer perceptron classifier is:

[0148] Its input is a single Gaussian multidimensional statistical eigenvector z. i :

[0149]

[0150] in, and These are multi-view weighted statistics of the average reconstruction residual and consistency cost within the Gaussian projection influence domain, respectively; μ i ,Σ i These are the Gaussian center and covariance, respectively. Contribution to average visibility; η i ζ is the Gaussian anisotropy ratio; i tr represents the boundary consistency score; tr represents the trace of the covariance matrix, which is equal to the sum of the variances of the three principal directions, i.e., the size of the Gaussian global scale; det represents the determinant of the covariance matrix, which is equal to the volume scale, i.e., the volume of the three-dimensional space covered by the Gaussian.

[0151] The multilayer perceptron classifier outputs a ternary probability f. θ (z i )∈Δ 2 Corresponding to "addition / pruning / preservation", the training is jointly performed using cross-entropy loss and delay consistency regularization term:

[0152]

[0153] Among them, y i The pseudo-labels are generated jointly by coarse discrimination and performance rewards, λ d Smoothing the decision time is controlled to improve training stability; i represents the Gaussian index, which represents the summation over all training samples; The training loss function for the classifier in the fine-discrimination stage consists of a cross-entropy term and a consistency regularization term; λ d This represents the regularization weight coefficient, which controls the importance of the delay consistency regularization term in the total loss. For the feature representation of the same Gaussian at iteration step t; This indicates that the expectation of the loss function is taken over all samples to obtain the overall training objective; z i θ represents the feature representation of the i-th Gaussian; θ represents the set of trainable parameters of the Layer Perceptron Classifier (MLP), such as complexity. Redundancy, etc.; l represents the classification loss for a single sample. Fine discrimination replaces coarse discrimination and assumes the main responsibility for density control in later iterations, shifting decision-making from heuristic to data-driven.

[0154] In the early stages of training, coarse discrimination is used to achieve fast and high-recall candidate selection, while in the later stages, an MLP classification model trained with multidimensional statistical features is used to achieve high-precision decision-making, thus achieving a balance between detail preservation and redundancy reduction.

[0155] In a preferred embodiment of the present invention, regarding multi-view information fusion and loss design, the present invention emphasizes the reality of uneven quality of view information and adopts a gradient energy-driven view weight λ. v When training a lightweight multilayer perceptron classifier, a multi-view weighted fusion method based on information quality is adopted to calculate the gradient intensity and high-frequency energy of each viewpoint. (After grayscale conversion and optional smoothing of the images from each viewpoint, the gradient magnitude is calculated using the Sobel operator, and the gradient intensity Gv is obtained by pixel averaging or quantiles; the high-frequency energy Hv is calculated using Laplacian energy; then Gv and Hv are normalized and calculated according to...) Fusion, ultimately with λ v =softmax(S v / T) yields the viewpoint weights, which are used for multi-view weighting in training and parameter optimization; the viewpoint weights λ are dynamically allocated. v ,for:

[0156]

[0157] Wherein, the set of viewpoint images is V = {v}, and u represents the index symbol, which represents any viewpoint in the set V and is used for denominator normalization; The power amplification of the viewpoint quality score is used to highlight high-quality views and suppress low-quality views in multi-view weighted fusion; τ≥1 is the weight amplification factor, which controls the tilt towards high-information views. The overall training objective for each viewpoint consists of luminosity, consistency, and regularization terms.

[0158]

[0159] By explicitly coupling two-dimensional complexity measurement with three-dimensional density control, the question of "in which regions to increase / decrease Gaussian" is transformed from a heuristic into an optimizable decision problem. The training objective is then written as an energy functional with complexity weights and sparse regularization.

[0160]

[0161] in, For the depth approximation obtained from synthesis, This represents a function that controls the covariance from diverging. It is a sparse constraint regularization that penalizes the opacity αi of all Gaussians, prompting training to automatically discard redundant Gaussians. Its weighted form encourages sparsity. The constant coefficients α1 to α5 prioritize global structure in the early stages of training (α1 and α3 are larger), and increase the weights of perception and sparse regularization in the later stages to focus on details and compression (the model is still in the coarse stage, at which point details are difficult to reconstruct, so priority is given to ensuring the general geometry and light consistency of the scene. Specifically, in the early stages of training, a1, a2, and a3 take relatively large values, while a4 and a5 take smaller values. This allows the model to build the overall structure first, avoiding premature and overly strong regularization that would cause the model to have difficulty converging. It is a two-stage constant); α6 is a weighted complexity regularization. with I v These are the synthetic and ground truth images, respectively. D v For the corresponding image depth; It is a density penalty / incentive term driven by complexity (with a negative gradient in the high complexity region and a positive gradient in the low complexity region); The overall training objective function is represented by a weighted combination of photometric, consistency, depth supervision, and regularization terms. SSIM is the structural similarity index, which measures the structural consistency between synthetic and real images. This represents the covariance regularization term, which constrains the Gaussian covariance matrix to prevent it from diverging and avoids the infinite expansion of Gaussian points. Represents the sparsity regularization term, based on Gaussian point opacity α. i This encourages the model to automatically compress redundant Gaussian points, making the representation more compact.

[0162] In the process of Gaussian parameter optimization, this invention adopts a multi-view weighted fusion method based on information quality. By calculating the gradient intensity and high-frequency energy of each viewpoint, the viewpoint weights are dynamically allocated, so that the viewpoint with rich texture and clear edges contributes more to the optimization, while the interference of low-quality viewpoints is suppressed, effectively improving the detail fidelity and fusion robustness.

[0163] In a preferred embodiment of the present invention, in order to enable complexity cues to directly affect Gaussian-level density control, when training a lightweight multilayer perceptron classifier, the influence of the three-dimensional complexity field on a single Gaussian is defined as the weighted integral of its support domain. for:

[0164]

[0165] in, For a three-dimensional complexity field, μ i ,Σ i These are the Gaussian center and covariance, respectively. Let X represent the effective complexity value of the i-th Gaussian point, where X is the point X enumerated in the entire 3D point set, and φ is the value of the effective complexity value. i(X) represents μ i ,Σ i is the three-dimensional Gaussian distribution kernel function with parameters, describing the influence range of the Gaussian point iii on its neighborhood in space; det represents the determinant of the covariance matrix, which is equal to the volume scale, i.e., the three-dimensional spatial volume covered by the Gaussian, and i is the index of the Gaussian point;

[0166] Numerically, it is approximated using importance sampling or ellipsoidal simplification, and... After combining the statistics, the Gaussian-level trigger function ψ is given. i :

[0167]

[0168] Where, ρ i Let σ be the neighborhood redundancy, and α be the Sigmoid function. c α r α γ α ρ >0, α c Complexity features The weights; α r Reconstructing residuals The weights; α γ Visibility contribution The weights; α ρ Neighborhood redundancy ρ i The weights; This represents the average reconstruction residual within the influence domain of the Gaussian projection; Contribution to average visibility; ρ i Neighborhood redundancy;

[0169] When ψ i Exceeding the dynamic threshold τ t At that time, enter the new candidate list;

[0170] When ψ i Below τ p and Smaller ( The Gaussian Gi's contribution to average visibility across multiple views is relatively small, meaning that the Gaussian is almost invisible and has a limited contribution in most views. Two methods are used for this determination: one is a fixed threshold, and the other is... One approach is to consider visibility to be low; another is the percentile distribution method, which statistically analyzes all Gaussian distributions. The distribution of Gaussians is used, with the 10th or 20th percentile as the threshold. If the Gaussian falls in the lower quantile interval, it is considered to be small and then enters the pruning candidate.

[0171] The threshold is used to perform annealing scheduling according to the training step. In the early stage, exploration and addition are encouraged, and in the later stage, it tends to be stable and convergent.

[0172] In terms of the training process, this invention adopts a phased resolution adaptive strategy. The first few iterations are performed at a lower resolution, globally laying out the geometric skeleton and quickly converging the coarse structure;

[0173] When the average residual drops to a set percentage, the resolution is automatically increased, and the coupling triggering of CRD and complexity mapping is enabled, which, together with the fine-discrimination MLP, performs fine addition and pruning.

[0174] To avoid parameter oscillations, fine discrimination initially adopts a teacher-student distillation update, that is, using coarse discrimination confidence as a soft label, and gradually transitions to density control driven entirely by the classifier.

[0175] At the optimizer level, different learning rates and preconditioning matrices can be used for position and covariance. A smaller step size is used for position to improve geometric stability, while the covariance is determined based on η. i The adjustment makes the scale update of highly anisotropic Gaussians more conservative.

[0176] During the inference and output phases, the system uses the converged Gaussian set g. * Real-time rendering is enabled. On modern GPUs, with the help of batch sorting and early culling strategies, pixel composition can ignore terms with very low transparency, achieving a complexity approximately linear with the number of visible Gaussians.

[0177] If further compression is needed, a one-time global pruning and covariance shrinkage can be performed without affecting subjective quality. The weights are temporarily increased to obtain a lightweight model for deployment.

[0178] In terms of implementation, the first implementation focuses on indoor high-texture scenes. During the data acquisition phase, structured light or binocular vision is used to obtain reliable depth priors, and the λ value is increased in the complexity modeling. f ,λ l The proportion of the weight is to highlight the texture and edges, and the CRD stage is more inclined to generate slender Gaussians along the segmentation boundary.

[0179] To accommodate mirrored and semi-transparent materials, the color parameter c i (d) A lightweight regression with second-order spherical harmonics and a viewpoint dependency is employed to reduce reflection-induced multi-viewpoint inconsistencies. This approach can more fully fill in high-frequency details under the same sampling budget, and both subjective and objective metrics are significantly better than the baseline without the complexity mechanism.

[0180] The second implementation is geared towards large-scale outdoor scenes. Since the depth prior may be coarser, the construction of the 3D complexity field employs a larger kernel bandwidth and voxel fusion.

[0181] To suppress redundant additions, a height-density attenuation factor based on terrain priors is introduced. A conservative strategy is applied to large planar areas on the ground, adding points only near significant peaks of H. Multi-view weight λ v The exponent K is moderately increased to reduce interference from distant, low-resolution perspectives; under this condition, the system can still maintain the visibility continuity of narrow structures such as building facades, railings, and power lines while ensuring smooth rendering.

[0182] The present invention also provides an adaptive density control multi-view 3D Gaussian densification system based on the method described in the present invention, comprising a data acquisition and preprocessing module, a 3D complexity field construction and fusion module, a coarse discrimination driven CRD, a fine discrimination MLP classifier, and a multi-view weighted optimization and rendering output module connected in sequence.

[0183] The data acquisition and preprocessing module is used to collect and preprocess raw images from multiple perspectives. The 3D complexity field construction and fusion module is used to construct a globally unified 3D complexity field and generate 3D error heatmaps. The module employs a coarse-discriminatory CRD (Cross-ray Densification) to locate regions with high residuals in 3D space through multi-view cross-projection, and then adds Gaussian points to these regions. This avoids overfitting caused by single-view driving by concentrating the added Gaussian points in regions where consistency across multiple views is difficult. A fine-discriminatory MLP (Multi-Layer Perceptron) is used to generate weak labels for supervised learning. It is a lightweight neural network composed of several fully connected layers used for mapping feature inputs to classification outputs. The MLP acts as a discriminative classifier, performing ternary classification (addition / pruning / retention) on each Gaussian feature vector.

[0184] The multi-view weighted optimization and rendering output module is used to train a lightweight multilayer perceptron classifier using weak labels, outputting ternary probabilities corresponding to "addition / pruning / preservation", and performing Gaussian density operation on the added regions.

[0185] Each module works in a closed-loop training and inference process by coupling with the camera geometric model through a unified set of Gaussian parameters.

[0186] This invention introduces a 2D-to-3D complexity mapping, cross ray densification (CRD), and a two-stage discrimination (coarse-fine) mechanism within the overall framework of multi-view 3D Gaussian rendering (MVGS). In parameter optimization, a multi-view information quality weighting strategy is adopted to ensure that the decision to add / prune Gaussian points is consistent with the real complexity of the scene, thereby simultaneously improving the synthesis quality of new perspectives, global geometric consistency, and computational efficiency.

[0187] Preferably, without changing the core idea of ​​the invention, different alternatives can be used for each component module, implementation method, and execution step. In addition to using the SAM model and FC-CLIP to extract the number of segments and categories, the complexity modeling part can also use other segmentation and semantic recognition models, such as Mask2Former, DeepLabV3+, and SegFormer, to generate pixel-level complexity heatmaps. The extraction of high-frequency features can also be replaced by Fourier transform, Gabor filtering, or the Laplacian of Gaussian operator to meet the requirements of different hardware platforms and computational overhead.

[0188] Besides the back-projection method, the construction of the three-dimensional complexity field can also employ a voxel integration method based on dense deep fusion, or obtain the complexity distribution by interpolating the results of multi-view sparse matching in voxel space. Regarding the complexity and residual fusion strategy, the weighted summation in this invention can be replaced by joint optimization under a multi-task learning framework, where the network simultaneously regresses the density probability and complexity estimate, thus eliminating the need for a separate complexity field construction step.

[0189] Regarding the densification and pruning discrimination mechanisms, the coarse discrimination stage can be changed from fixed threshold detection to multi-scale local maximum detection based on hierarchical grid search, or the density adjustment direction can be inferred directly from the Gaussian neighborhood graph through a graph convolutional network. The MLP model in the fine discrimination stage can also be replaced with support vector machines, random forests, gradient boosting trees, or lightweight Transformers to adapt to the accuracy and inference speed requirements of different scenarios. In the initialization strategy for new Gaussian points, in addition to orientation along the complexity gradient or principal curvature direction, covariance can be set based on the normal distribution of the local surface fitting, or new points can be generated by sampling the neighborhood distribution parameters of existing Gaussian points.

[0190] In addition to static weights based on gradient strength, multi-view weighted fusion methods can also employ dynamic fusion based on self-attention mechanisms, where the network adaptively generates weights according to the feature distribution of each viewpoint; or introduce a weighting method based on information entropy, making the weights directly related to the information gain of that viewpoint in the reconstruction task.

[0191] Regarding local density constraints and sparsity regularization, density monitoring can be replaced by spatial sampling consistency checks. For example, by comparing the consistency of rendered images before and after pruning through random projection, Gaussian points can be retained or deleted. The sparsity regularization term can also be replaced by group sparse or structural sparse strategies, so that the pruning process is associated with the spatial distribution or semantic labels of Gaussians.

[0192] The core idea of ​​this invention—multi-view adaptive density control driven by scene complexity—is not only applicable to rendering frameworks based on 3D Gaussian representation, but can also be extended to other explicit and implicit 3D representation methods. For example, in frameworks such as voxel-based volume rendering, triangular mesh-based surface reconstruction, and multi-plane MPI representation, the complexity field can be introduced into the density scheduling and resource allocation process, thereby achieving similar details enhancement and efficiency optimization effects under different 3D representation forms.

[0193] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0194] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A multi-view three-dimensional Gaussian denserization method with adaptive density control, characterized in that, Includes the following steps: Acquire raw images from multiple perspectives and perform preprocessing; Complexity features are extracted at the two-dimensional image level to generate pixel-level complexity heatmaps, and a globally unified three-dimensional complexity field is constructed through multi-view back-projection fusion. The reconstruction residuals, high-frequency inconsistencies, and depth / geometric consistency costs from each perspective are back-projected together to generate a 3D error heatmap. Based on the three-dimensional complexity field C 3D Based on the 3D error heat H, candidate new set and candidate pruning set are determined, and weak labels are generated for supervised learning; A lightweight multilayer perceptron classifier is trained using weak labels, and the output ternary probabilities correspond to "addition / pruning / preservation". Gaussian density is then applied to the newly added regions.

2. The adaptive density-controlled multi-view three-dimensional Gaussian denserization method as described in claim 1, characterized in that, The method for preprocessing original images from multiple perspectives is as follows: The set of viewpoint images is V = {v}, where each viewpoint provides a color image Iv, as well as camera intrinsic parameters Kv and extrinsic parameters (R). v ,t v ); The rendering of pixel p at viewpoint v employs 2D Gaussian rasterization and forward composite, making w i,v (p) represents the two-dimensional Gaussian weights projected onto the pixels, then the color composite image... for: Where p represents a two-dimensional pixel, T i-1,v (p) represents the transmittance up to the (i-1)th Gaussian, reflecting the degree to which pixel p is absorbed by the preceding Gaussian, with an initial value T. 0,v (p) = 1, d v,p This represents the line of sight from camera v to pixel p, where k is the k-th pixel. <i;α k,v (p) is defined the same as α i,v (p); α i,v (p)=1-exp(-τw i,v (p)) represents the contribution of the i-th Gaussian to the opacity of pixel p, ranging from [0,1]; τ>0 represents the mapping of the medium accumulation coefficient rate; The two-dimensional weights are obtained by linearizing the projection of the three-dimensional ellipsoid onto the imaging plane, with the projection center denoted as μ′. i,v The projection covariance is Σ′ i,v ,but:

3. The adaptive density-controlled multi-view three-dimensional Gaussian denserization method as described in claim 1, characterized in that, The method for extracting complexity features at the two-dimensional image level, generating pixel-level complexity heatmaps, and constructing a globally unified three-dimensional complexity field through multi-view backprojection fusion is as follows: In each viewpoint, the preprocessed original image is subjected to silent segmentation and named category recognition, yielding the number of segmented regions num_seg(v) and the number of categories num, respectively. class(v) ; Complexity metrics are used to generate complexity and complexity heatmaps for each perspective, providing perspective-level complexity. Defined as: Where w1, w2>0 are weights, and the least squares method is used to fit the data on the representative validation set. Pixel-level complexity heat value C v (p) integrates segmentation boundary density, semantic category density, and high-frequency information in the frequency domain, and is specifically constructed as follows: C v (p)=λ s E v (p)+λ f H v (p)+λ l L v (p), Among them, E v (p) represents the edge saliency obtained from the gradient field of the segmentation boundary, H v (p) represents the high-frequency intensity derived by wavelet or Sobel, L v (p) is the Laplace difference plot, λ s ,λ f ,λ l ≥0 is the weight and is normalized; The full-graph level complexity heatmap C v Back-projecting to a unified world coordinate system, the depth D can be obtained using known camera interior and exterior parameters. v (p) projects pixel p back onto a 3D point. Complexity fields are formed in the three-dimensional domain using voxel stacks or kernel density estimation. Where Q is the metric tensor, Z is the normalization factor; the set of viewpoint images is V = {v}; Ω v Let represent the set of pixels at viewpoint v, i.e., the index range of all pixels on the imaging plane of camera v; Θ(·) represents the kernel function, used to smoothly propagate the complexity value corresponding to a pixel to 3D space; X represents the enumerated point X in the entire 3D point set, ||XX v,p || Q : 3D point X and 3D point X obtained by back projection of pixel p v,p The distance, with the weighted norm of the metric tensor Q; X v,p This represents the coordinates of a 3D point in 3D space projected from pixel p in the camera's viewpoint v. This represents the transpose of the rotation matrix of the camera's viewpoint v; This represents the inverse of the intrinsic parameter matrix of the camera's viewpoint v; Represents pixel p = (u, v) in homogeneous coordinate form; D v (p) represents the depth value of pixel p at the viewpoint v, and represents the distance from the camera center along the ray from that pixel to the scene surface; t v This represents the translation vector of the camera's viewpoint v.

4. The adaptive density-controlled multi-view three-dimensional Gaussian denserization method as described in claim 1, characterized in that, The reconstruction residuals, high-frequency inconsistencies, and depth / geometric consistency costs from each perspective are back-projected together to generate a 3D error heatmap, specifically: Define the pixel reconstruction residual under viewpoint v as Define structural consistency penalty s v (p), then the three-dimensional error heat for: Where p represents a two-dimensional pixel; with I v These represent the synthesized and ground truth images, respectively; Q is the metric tensor; β1 and β2 represent non-negative weight coefficients, which control the residual term r, respectively. v (p) and consistency term s v (p); Θ(·) represents the kernel function, X v,p s represents a three-dimensional point. v (p): The structural consistency penalty term for pixel p under viewpoint v, such as gradient direction difference, normal consistency error or multi-disparity cost; X: The point X enumerated in the entire 3D point set.

5. The adaptive density-controlled multi-view three-dimensional Gaussian denserization method as described in claim 1, characterized in that, Based on the three-dimensional complexity field C 3D The method for determining the candidate addition set and candidate pruning set based on the 3D error heat H, and generating weak labels for supervised learning, is as follows: When the three-dimensional complexity field C 3D When the 3D error heat H and the Gaussian mass both reach a local maximum in a certain spatial neighborhood, they are considered candidate densification locations, triggering the addition of a new Gaussian. Pruning is performed on Gaussians that contribute very little to the synthesis while maintaining visual quality.

6. The adaptive density-controlled multi-view three-dimensional Gaussian denserization method as described in claim 5, characterized in that, When a new Gaussian is triggered, two strategies are used: One method is splitting, which involves selecting the Gaussian G with the greatest influence near the candidate position. i And split it into several sub-Gaussians The center and covariance of the sub-Gaussian are as follows Initialization, ∑ k ρ k =1, while maintaining the conservation of color and opacity. Where μ i ,Σ i ,α i These are Gaussian center, covariance, and opacity, respectively; among them, This represents the center position of the k-th sub-Gaussian, calculated by adding an offset δμ to the original center μi. (k) We obtain; μi represents the center position of the original Gaussian Gi; δμ (k) This represents the offset of the sub-Gaussian relative to the original center, used to split the original Gaussian so that the sub-Gaussian covers different spatial sub-regions; Let the covariance matrix of the k-th sub-Gaussian be represented by a scaling factor ρ. k ρ is obtained by scaling the original covariance. k This represents the non-negative scaling factor, used to distribute the variance range of the original Gaussian to each sub-Gaussian; the constraint condition is ∑ k ρ k =1, ensuring the conservation of population variance; Σ i Represents the covariance matrix of the original Gaussian, describing its shape and scale; Let the opacity of the k-th sub-Gaussian be such that it satisfies the following condition: That is, the total opacity is approximately conserved before and after the split; α i The first is the opacity of the original Gaussian, which controls its contribution to rendering compositing; the second is injection, which is directly applied to argmax. X A new Gaussian is placed at H(X) along the principal curvature direction. The covariance is initialized by slightly stretching along the 3D complexity gradient direction to capture slender structures, and local density constraints are introduced. when When the upper limit is exceeded, new additions are suppressed, forming spatial self-conservation. X: the point X enumerated in the entire three-dimensional point set, ∈ represents the neighborhood threshold, which is used to define the criterion of proximity to a certain Gaussian.

7. The adaptive density-controlled multi-view three-dimensional Gaussian denserization method as described in claim 5, characterized in that, The specific method for pruning Gaussians that contribute very little to the synthesis while maintaining visual quality is as follows: A multi-view cross-ray densification mechanism is introduced to uniformly project high-error regions from multiple views onto three-dimensional space, defining the average visibility—contribution γ—of a single Gaussian at viewpoint v. i,v ,for: Among them, Ω v T represents the set of pixels at viewpoint v, that is, the index range of all pixels on the imaging plane of camera v. i-1,v (p) represents the transmittance up to the (i-1)th Gaussian digit at pixel p; α i,v (p) represents the opacity of the i-th Gaussian at pixel p in viewpoint v, with a value range of [0,1]; p represents a two-dimensional pixel, and the viewpoint image set is V={v}; When average visibility contribution Continuously below the threshold and 3D complexity Simultaneously, if the value is low and there is high redundancy around the Gaussian, pruning is performed. To avoid accidentally deleting boundary details, a covariance anisotropy ratio is introduced. Consistency score ζ of the boundary i , in η i or ζ i When the threshold is high, the retention threshold is increased to reflect the design principle of "retaining the boundary first".

8. The adaptive density-controlled multi-view three-dimensional Gaussian denserization method as described in claim 1, characterized in that, A lightweight multilayer perceptron classifier is: Its input is a single Gaussian multidimensional statistical eigenvector z. i : in, and These are multi-view weighted statistics of the average reconstruction residual and consistency cost within the Gaussian projection influence domain, respectively; μ i ,Σ i These are the Gaussian center and covariance, respectively. Contribution to average visibility; η i ζ is the Gaussian anisotropy ratio; i tr represents the boundary consistency score; tr represents the trace of the covariance matrix, which is equal to the sum of the variances of the three principal directions, i.e., the size of the Gaussian global scale; det represents the determinant of the covariance matrix, which is equal to the volume scale, i.e., the volume of the three-dimensional space covered by the Gaussian. The multilayer perceptron classifier outputs a ternary probability f. θ (z i )∈Δ 2 Corresponding to "addition / pruning / preservation", the training uses cross-entropy loss and latency consistency regularization term jointly: Among them, y i The pseudo-labels are generated jointly by coarse discrimination and performance rewards, λ d Smoothing the decision time is controlled to improve training stability; i represents the Gaussian index, which represents the summation over all training samples; The training loss function for the classifier in the fine-discrimination stage consists of a cross-entropy term and a consistency regularization term; λ d This represents the regularization weight coefficient, which controls the importance of the delay consistency regularization term in the total loss. For the feature representation of the same Gaussian at iteration step t; This indicates that the expectation of the loss function is taken over all samples to obtain the overall training objective; z i Let θ represent the feature representation of the i-th Gaussian; θ represents the set of trainable parameters of the layer perceptron classifier MLP; l represents the classification loss for a single sample.

9. The adaptive density-controlled multi-view three-dimensional Gaussian denserization method as described in claim 1, characterized in that, When training a lightweight multilayer perceptron classifier, a multi-view weighted fusion method based on information quality is adopted to calculate the gradient strength and high-frequency energy of each view and dynamically allocate the view weights λ. v ,for: Wherein, the set of viewpoint images is V = {v}, and u represents the index symbol, which represents any viewpoint in the set V and is used for denominator normalization; The power amplification of the viewpoint quality score is used to highlight high-quality views and suppress low-quality views in multi-view weighted fusion; τ≥1 is the weight amplification factor, which controls the tilt towards high-information views. The overall training objective for each viewpoint consists of luminosity, consistency, and regularization terms. By explicitly coupling two-dimensional complexity measurement with three-dimensional density control, the question of "in which regions to increase / decrease Gaussian" is transformed from a heuristic into an optimizable decision problem. The training objective is then written as an energy functional with complexity weights and sparse regularization. in, For the depth approximation obtained from synthesis, This represents a function that controls the covariance from diverging. It is a sparsity constraint regularization that penalizes the opacity αi of all Gaussians, prompting training to automatically discard redundant Gaussians. Its weighted form encourages sparsity. The constant coefficients α1 to α5 focus on global structure in the early stage of training, and increase the weight of perception and sparsity regularization in the later stage to focus on details and compression. α6 is a weighted complexity regularization. with I v These are the synthetic and ground truth images, respectively. D v For the corresponding image depth; Density penalty / incentive term driven by complexity; The overall training objective function is represented by a weighted combination of photometric, consistency, depth supervision, and regularization terms. SSIM is the structural similarity index, which measures the structural consistency between synthetic and real images. This represents the covariance regularization term, which constrains the Gaussian covariance matrix to prevent it from diverging and avoids the infinite expansion of Gaussian points. Represents the sparsity regularization term, based on Gaussian point opacity α. i This encourages the model to automatically compress redundant Gaussian points, making the representation more compact.

10. The adaptive density-controlled multi-view three-dimensional Gaussian denserization method as described in claim 1, characterized in that, When training a lightweight multilayer perceptron classifier, the effect of a three-dimensional complexity field on a single Gaussian is defined as the weighted integral of its support domain. for: in, For a three-dimensional complexity field, μ i ,Σ i These are the Gaussian center and covariance, respectively. Let X represent the effective complexity value of the i-th Gaussian point, where X is the point X enumerated in the entire 3D point set, and φ is the value of the effective complexity value. i (X) represents μ i ,Σ i is the three-dimensional Gaussian distribution kernel function with parameters, describing the influence range of the Gaussian point iii on its neighborhood in space; det represents the determinant of the covariance matrix, which is equal to the volume scale, i.e., the three-dimensional spatial volume covered by the Gaussian, and i is the index of the Gaussian point; Numerically, it is approximated using importance sampling or ellipsoidal simplification, and... After combining the statistics, the Gaussian-level trigger function ψ is given. i : Where, ρ i Let σ be the neighborhood redundancy, and α be the Sigmoid function. c α r α γ α ρ >0, α c Complexity features The weights; α r Reconstructing residuals The weights; α γ Visibility contribution The weights; α ρ Neighborhood redundancy ρ i The weights; This represents the average reconstruction residual within the influence domain of the Gaussian projection; Contribution to average visibility; ρ i Neighborhood redundancy; When ψ i Exceeding the dynamic threshold τ t At that time, enter the new candidate list; When ψ i Below τ p and Smaller ones are then considered for pruning; The threshold is used to perform annealing scheduling according to the training step. In the early stage, exploration and addition are encouraged, and in the later stage, it tends to be stable and convergent.

11. A multi-view three-dimensional Gaussian denserization system with adaptive density control based on the method of any one of claims 1-10, characterized in that, It includes a data acquisition and preprocessing module, a 3D complexity field construction and fusion module, a coarse-discrimination driven CRD, a fine-discrimination MLP classifier, and a multi-view weighted optimization and rendering output module, which are connected in sequence. The data acquisition and preprocessing module is used to acquire raw images from multiple perspectives and perform preprocessing. The three-dimensional complexity field construction and fusion module is used to construct a globally unified three-dimensional complexity field and generate a three-dimensional error heat. The coarse-discriminative cross-ray denser CRD and the fine-discriminative multilayer perceptron (MLP) classifier are used to generate weak labels for supervised learning. The multi-view weighted optimization and rendering output module is used to train a lightweight multilayer perceptron classifier using weak labels, outputting ternary probabilities corresponding to "addition / pruning / preservation", and performing Gaussian density operation on the added region.

Images