Structure-Aware Adaptive Gaussian Splashing Method

Abstract

This invention provides a structure-aware adaptive Gaussian splashing method, belonging to the field of 3D scene reconstruction and rendering technology. First, multi-view images of the scene and camera parameters are acquired, and 3D Gaussian primitives are initialized. Then, a structure-aware mask is constructed using morphological processing and surface normal consistency constraints to enhance the geometric representation of thin-structure regions and occlusion boundary regions. Next, a cascaded multi-resolution optimization strategy is employed to progressively update the Gaussian primitive parameters. Then, the orientation and anisotropic shape of the Gaussian primitives are adaptively adjusted based on rendering feedback. Finally, a multi-index evaluation strategy is constructed, combining uncertainty, color entropy, gradient intensity, and view coverage to complete the importance assessment and pruning of the Gaussian primitives, obtaining an optimized set of 3D Gaussian primitives for target image rendering. This invention can effectively improve the structural representation and detail restoration effects in complex scenes, rendering high-quality scene images.

Description

Technical Field

[0001] This invention relates to the fields of 3D scene reconstruction and rendering, computer graphics and computer vision, and specifically to a structure-aware adaptive Gaussian splashing method. Background Technology

[0002] 3D scene reconstruction and rendering technology has wide applications in fields such as digital twins, virtual reality, augmented reality, and 3D visualization. Among existing scene representation methods based on multi-view images, the 3D Gaussian splashing method explicitly models the scene using 3D Gaussian primitives and combines it with differentiable rasterization to achieve image rendering, offering advantages such as fast training speed and high rendering efficiency.

[0003] However, existing 3D Gaussian splashing methods still have the following shortcomings in complex scenes: First, they are not good at representing thin structural regions and occluded boundary regions, which easily leads to edge blurring, structural breakage and floating artifacts; second, they do not make sufficient use of structural information such as normal consistency and local geometric continuity, resulting in unstable constraints in structurally sensitive areas; third, existing methods mostly adopt single-scale optimization or fixed optimization methods, which makes it difficult to take into account both global structure restoration and local detail enhancement; fourth, existing pruning methods mostly rely on a single evaluation index, which is prone to accidentally deleting important Gaussian primitives, thereby affecting the expression of scene details and rendering results.

[0004] The aforementioned problems severely restrict the development and application of 3D Gaussian splash scene rendering methods, exposing their shortcomings such as insufficient structural representation in complex scenes, unstable boundary regions, limited detail recovery capabilities, and high model redundancy. Therefore, there is an urgent need for a robust high-fidelity scene rendering method to obtain a high-quality set of 3D Gaussian primitives from multi-view image data and output a high-fidelity target viewpoint rendered image.

[0005] In recent years, three-dimensional Gaussian splashing and related improved methods have made certain research progress in the field of three-dimensional scene representation and rendering. The relevant research papers include: [1] "3D Gaussian Splatting for Real-Time Radiance Field Rendering", [2] "Mip-Splatting: Alias-free 3D Gaussian Splatting", [3] "FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization", [4] "Compact3D: Smaller and Faster Gaussian Splatting with Vector Quantization", [5] "Light Gaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS".

[0006] Chinese patent application CN120355848A discloses a method, apparatus, device, and medium for scene surface reconstruction. The key technology of this solution is to further inversely obtain depth information from an updated 3D Gaussian model, and then construct a scene surface model through a 3D reconstruction algorithm. In other words, its core objective is "surface model reconstruction," and its key processing chain is: rendering the image, updating the Gaussian model, determining the depth, and constructing the surface model; its loss function also mainly revolves around color, semantics, depth, and normal information.

[0007] Chinese patent application CN120655860A discloses a 3D Gaussian sputtering scene reconstruction method based on structure-aware refined Gaussian. The structure-aware refinement of this method is mainly based on the technical route of "external depth / normal estimation + sparse point cloud filtering + normal consistency regularization + structure scoring screening + joint loss training". Specifically, a depth map and normal map are first generated using a Lotus model. Then, statistical filtering and radius filtering are applied to the sparse point cloud. Next, a normal consistency regularization term forces the Gaussian minor axis direction to align with the normal direction. Finally, a comprehensive scoring function is constructed using anisotropy, curvature, visibility count, and opacity variation variance to screen Gaussian points.

[0008] The key points of these three-dimensional Gaussian splashing methods are how to improve real-time rendering efficiency, alleviate aliasing problems, enhance frequency representation capabilities, or compress the number of Gaussian primitives. However, for thin structural regions, occluded boundary regions, and high-frequency detail regions in complex real-world scenes, existing methods still struggle to simultaneously ensure structural representation stability, detail recovery capabilities, and model representation compactness.

[0009] In other words, existing technologies still face the following challenges when applied to complex scene image data: (1) Existing 3D Gaussian splashing methods have low accuracy in representing thin structural regions and occluded boundary regions in complex scenes, making it difficult to meet the application requirements of high-fidelity rendering in fields such as digital twins, digital protection of cultural heritage, virtual reality, augmented reality and immersive display; (2) Existing 3D Gaussian splashing methods do not make sufficient use of prior information on local structures and geometric continuity in complex scenes, resulting in poor scene representation stability and easy occurrence of structural breakage and floating artifacts; (3) Existing 3D Gaussian splashing methods are difficult to balance global structure restoration, local detail enhancement and model redundancy compression in complex scenes, so there is still a lot of room for improvement in rendering quality and representation efficiency. Summary of the Invention

[0010] Purpose of the Invention: The purpose of this invention is to address the shortcomings of existing technologies and provide a structure-aware adaptive Gaussian splashing method. This invention can improve the geometric representation of thin structural regions and occlusion boundary regions in complex scenes, as well as improve the quality of scene detail restoration and model representation compactness. This effectively overcomes the shortcomings of existing 3D Gaussian splashing scene rendering methods and can promote the development of fields such as metaverse, digital twins, digital protection of cultural heritage, virtual reality, and augmented reality.

[0011] Technical solution: The present invention provides a structure-aware adaptive Gaussian splashing method, comprising the following steps:

[0012] Step S1: Input multi-view images of the scene and corresponding camera parameters, and initialize a set of 3D Gaussian primitives and an initial foreground mask based on the sparse reconstruction results; wherein, the multi-view images are denoted as... , Indicates the number of input viewpoints; the set of three-dimensional Gaussian primitives is denoted as . ,in, Indicates the first The central position of a Gaussian primitive, Indicates the first The covariance matrix of a Gaussian primitive. Indicates the first The transparency of a Gaussian primitive. Indicates the first The color attribute of a Gaussian primitive Indicates the number of Gaussian primitives;

[0013] Step S2: Perform structure-aware enhancement on the initial foreground mask to obtain a structure-aware mask, thereby enhancing the geometric representation of thin structure regions and occlusion boundary regions;

[0014] Step S3: Based on structure-aware masking, a cascaded multi-resolution optimization strategy is used to progressively optimize the position parameters, covariance parameters, transparency parameters, and color parameters of the 3D Gaussian primitives from coarse to fine, and output the set of 3D Gaussian primitives after multi-resolution optimization.

[0015] Step S4: Based on the structural differences between the rendered image and the real image from the reference viewpoint in Step S1, adaptively adjust the principal direction and anisotropic shape of the three-dimensional Gaussian primitive.

[0016] Step S5: Integrate Bayesian uncertainty, color information entropy, gradient strength, and view coverage to evaluate the importance of the 3D Gaussian primitives and perform adaptive pruning.

[0017] Step S6: Output the optimized three-dimensional Gaussian primitive set, and render the target view image based on the three-dimensional Gaussian primitive set.

[0018] Further, in step S2, the initial foreground mask is... Perform morphological dilation and erosion operations to obtain a preliminary corrected mask. Specifically, it is expressed as:

[0019]

[0020] in, Indicates the initial foreground mask. This represents the initial correction mask after morphological processing. Represents the structural element corresponding to morphological operations. This indicates the expansion operation. This represents the erosion operation;

[0021] For the pixel regions in the preliminary correction mask, surface normal vectors are estimated based on the local geometric relationships of the corresponding pixels, and the consistency of normal vectors between adjacent pixels is calculated. When the consistency of normal vectors between adjacent pixels does not meet a preset threshold condition, the corresponding pixel region is determined to be a geometrically discontinuous region and removed from the preliminary correction mask. Meanwhile, the consistency of surface normal vectors between adjacent pixels satisfies the following constraints:

[0022] ;

[0023] in, and Representing pixels and pixels The corresponding surface normal vector, Indicates the normal consistency threshold;

[0024] According to the preliminary modified mask The structure-aware mask is obtained by filtering the results based on the consistency of the normal vector. ;in, This represents the final generated structure-aware mask, used to enhance the geometric representation of thin-structure regions and occluded boundary regions.

[0025] Furthermore, the detailed steps of step S3 are as follows:

[0026] Step S3.1: Set up a low-resolution optimization stage. At a lower resolution (preferably 1 / 4 to 1 / 2 of the target resolution) after downsampling, perform joint gradient optimization on the position parameters, covariance parameters, transparency parameters and color parameters of the Gaussian primitives based on the differentiable rendering error to restore the global geometry of the scene.

[0027] Step S3.2: Set up the medium-resolution optimization stage. Based on the resolution change, scale and update the covariance parameters of the 3D Gaussian primitives, and continue to jointly optimize the position parameters, covariance parameters, transparency parameters, and color parameters of the Gaussian primitives to enhance the medium-scale structural representation of the scene; the scaling update of the covariance parameters satisfies:

[0028] ;in, This represents the covariance parameter in the low-resolution stage. This represents the covariance parameter at the medium resolution stage. and These represent the scale coefficients corresponding to the low-resolution and medium-resolution stages, respectively.

[0029] Step S3.3: Set up a high-resolution optimization stage. At a higher resolution close to the target rendering resolution (not less than 1 / 2 of the target resolution), further refine and optimize the 3D Gaussian primitives to improve the expressive ability of boundary areas and high-frequency detail areas.

[0030] Step S3.4: Through progressive training in low-resolution, medium-resolution, and high-resolution optimization stages, cascaded multi-resolution optimization from global structure restoration to local detail enhancement is achieved.

[0031] Furthermore, the first Projecting a three-dimensional Gaussian primitive onto the image plane of the current viewpoint yields its region of influence in the image space. The image gradient is calculated within the affected region, and a gradient structure tensor is constructed. Specifically, it is expressed as:

[0032]

[0033] in, Indicates the first The region of influence of a Gaussian primitive in the image space. Indicates the pixel position in the affected area. Indicates pixel position Image gradient at that location, Indicates the first The gradient structure tensor corresponding to each Gaussian primitive; based on the gradient structure tensor The principal eigenvector determines the first The principal direction of the Gaussian primitive is determined, and the major axis of the Gaussian primitive is aligned with the principal structure direction of the image; based on the gradient structure tensor... eigenvalue distribution adjustment The degree of anisotropy of a Gaussian primitive, wherein the degree of anisotropy satisfies:

[0034] ;

[0035] in, Indicates the first The degree of anisotropy of a Gaussian primitive, Indicates the adjustment factor. Represents the gradient structure tensor The largest eigenvalue, Represents the gradient structure tensor The average eigenvalues;

[0036] The shape parameters of the 3D Gaussian primitive are updated based on the adjusted principal direction and degree of anisotropy, specifically: the shape parameters of the 3D Gaussian primitive are updated according to the adjusted principal direction and degree of anisotropy. The covariance matrix of a Gaussian primitive is decomposed into a rotation matrix and a scale matrix. The rotation matrix is ​​updated according to the principal direction and the scale matrix is ​​updated according to the degree of anisotropy to reconstruct the updated covariance matrix, thereby improving the fitting ability of edge regions and detail regions.

[0037] Here, the covariance matrix is ​​first decomposed into a rotation matrix and a scale matrix. Then, the rotation matrix is ​​updated according to the principal direction. Next, the scale matrix is ​​updated according to the degree of anisotropy. Finally, a new covariance matrix is ​​reconstructed. The final output is the shape parameters of the three-dimensional Gaussian primitive after principal direction adjustment and anisotropy update, and its corresponding updated covariance matrix.

[0038] Furthermore, in step S5, the Bayesian uncertainty is calculated for each three-dimensional Gaussian primitive. Color information entropy Gradient strength and view coverage Based on Bayesian uncertainty Color information entropy Gradient strength and view coverage Construct the first Importance rating of Gaussian primitives Specifically, it is expressed as: ;in, Indicates the first The importance score of each Gaussian primitive , , and These represent the weight coefficients corresponding to Bayesian uncertainty, color information entropy, gradient strength, and view coverage, respectively; the importance scores are based on each 3D Gaussian primitive. All Gaussian primitives are sorted, and adaptive pruning is performed on low-importance Gaussian primitives to reduce the number of redundant Gaussian primitives while preserving scene structural details and rendering quality. The specific steps are as follows:

[0039] Step S5.1: Calculate the Bayesian uncertainty for each three-dimensional Gaussian primitive. Color information entropy Gradient strength and view coverage The Bayesian uncertainty Represented as:

[0040] ;

[0041] in, Indicates the first The Gaussian primitive in the first Prediction results under random sampling or random deactivation This represents the mean of the corresponding prediction results. Indicates the number of samples;

[0042] The color information entropy Represented as:

[0043] ;

[0044] in, Indicates the first The color distribution of the Gaussian primitives falls into the first... The probability of each bin. Indicates the number of color bins;

[0045] The gradient strength Represented as:

[0046] ;

[0047] in, Indicates the first A Gaussian primitive affects the number of pixels within a region;

[0048] The view coverage Represented as:

[0049] ;

[0050] in, Indicates the first The Gaussian primitive in the first An indicator function for visibility from a given viewpoint; it takes the value 1 if visible and 0 otherwise. Indicates the number of input viewpoints;

[0051] Step S5.2: Based on Bayesian uncertainty... Color information entropy Gradient strength and view coverage Construct the first Importance rating of Gaussian primitives :

[0052] ;

[0053] in, Indicates the first The importance score of each Gaussian primitive , , and These represent the weighting coefficients corresponding to Bayesian uncertainty, color information entropy, gradient strength, and view coverage, respectively.

[0054] Step S5.3: Score based on the importance of each three-dimensional Gaussian primitive. Sort all Gaussian primitives and perform adaptive pruning on low-importance Gaussian primitives according to a preset pruning ratio (by...). Sort and remove Gaussian primitives at the bottom of the list, retaining only the minimum percentage to avoid over-pruning, in order to reduce the number of redundant Gaussian primitives and maintain scene structure details and rendering quality.

[0055] Further, the output of step S6 includes an optimized set of three-dimensional Gaussian primitives and a target view image rendered based on the set of three-dimensional Gaussian primitives; wherein, the optimized set of three-dimensional Gaussian primitives is represented as... The target view image is a result of differentiable rendering based on an optimized set of three-dimensional Gaussian primitives, used to characterize the final scene rendering output of the method.

[0056] Beneficial effects: This invention employs a structure-aware enhancement module to correct the initial foreground mask, and combines morphological processing and surface normal consistency constraints to generate a structure-aware mask, thereby enhancing the geometric representation of thin-structure regions and occluded boundary regions. Then, a cascaded multi-resolution optimization strategy is used to progressively optimize parameters such as position, covariance, transparency, and color of the 3D Gaussian primitives from coarse to fine. Simultaneously, the principal direction and anisotropic shape of the Gaussian primitives are adaptively adjusted based on the feedback from the rendered image to improve the fitting ability of edge regions and high-frequency detail regions. Furthermore, multi-dimensional information such as Bayesian uncertainty, color entropy, gradient strength, and view coverage are integrated to evaluate the importance of the Gaussian primitives and perform adaptive pruning. Finally, an optimized set of 3D Gaussian primitives is obtained, and a target view rendering image is output based on this set.

[0057] Compared with the prior art, the advantages of the present invention are as follows:

[0058] (1) The present invention uses a structure perception enhancement method that combines morphological processing with surface normal consistency constraints to effectively improve the mask reliability and geometric continuity of thin structure regions and occluded boundary regions, thereby reducing edge blurring and structural breakage problems.

[0059] (2) The present invention adopts a cascaded multi-resolution optimization strategy. Through a progressive training process from low resolution to high resolution, the parameters of the three-dimensional Gaussian primitives are optimized from coarse to fine. This can take into account both global structure restoration and local detail enhancement, and improve the reconstruction stability and detail expression ability in complex scenes.

[0060] (3) The present invention uses an anisotropic adaptive mechanism driven by rendering feedback to dynamically adjust the main direction and shape parameters of the Gaussian primitive according to the main structure direction of the image, so that the Gaussian primitive can better fit the edge area and high frequency detail area, thereby improving the clarity and structural consistency of the rendering result.

[0061] (4) This invention integrates multi-dimensional information such as Bayesian uncertainty, color information entropy, gradient strength and view coverage for importance assessment and adaptive pruning, which can reduce the number of redundant Gaussian primitives while maintaining scene structure details and rendering quality, and improve model representation compactness and rendering efficiency.

[0062] (5) The present invention can output an optimized set of three-dimensional Gaussian primitives and a target view rendering image, which can provide high-quality scene representation and rendering results for fields such as digital twins, digital protection of cultural heritage, virtual reality, augmented reality and three-dimensional visualization. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0064] Figure 2 This is a sample of image data from an embodiment.

[0065] Figure 3 This is a schematic diagram of the mask in the embodiment.

[0066] Figure 4 The example image is the final output target image sample using the technical solution of the present invention.

[0067] Figure 5 The example image is a target image sample output using existing technical solutions. Detailed Implementation

[0068] The technical solution of the present invention will be described in detail below, but the scope of protection of the present invention is not limited to the embodiments described.

[0069] Existing 3D Gaussian splashing methods often fail to adequately constrain the geometric structure of thin-structured regions, occlusion boundary regions, and high-frequency detail regions when representing scenes and rendering target perspectives in complex scenarios. This leads to problems such as blurred edges, structural breaks, floating artifacts, and insufficient detail recovery in the rendering results, which are particularly pronounced in complex realistic scenes. To address these issues, the structure-aware adaptive Gaussian splashing method and system of this invention effectively improves the structural expressiveness and rendering quality of 3D Gaussian primitives in complex scenes by constructing a structure-aware mask, employing a cascaded multi-resolution optimization strategy, introducing a rendering feedback-driven anisotropic adaptive mechanism, and constructing a multi-dimensional importance evaluation and pruning strategy.

[0070] like Figure 1 As shown, the main idea of ​​this invention is as follows: First, input multi-view image data and camera parameters to initialize 3D Gaussian primitives; then, construct a structure-aware mask to enhance the geometric representation of thin-structure regions and occluded boundary regions; second, employ a cascaded multi-resolution optimization strategy to progressively optimize the 3D Gaussian primitives from coarse to fine; third, adaptively adjust the principal direction and anisotropic shape of the Gaussian primitives based on rendering feedback; subsequently, construct a multi-index evaluation strategy to assess the importance of the Gaussian primitives and perform adaptive pruning; finally, output the optimized set of 3D Gaussian primitives and render the target image. This invention fully utilizes the synergistic effect of structure-aware enhancement, cascaded multi-resolution optimization, adaptive adjustment based on rendering feedback, and multi-dimensional constraint pruning to effectively improve the structural representation capability in complex scenes and the rendering quality of the target view image.

[0071] The structure-aware adaptive Gaussian splashing method in this embodiment includes the following steps:

[0072] Step S1: Input multi-view images of the scene and corresponding camera parameters, and initialize the 3D Gaussian primitive set and initial foreground mask based on the sparse reconstruction results. Let the multi-view images of the input scene be... ,in, This represents the number of input viewpoints. Based on the input multi-view images and corresponding camera parameters, sparse reconstruction of the scene is performed to obtain the sparse point cloud result of the scene; then, a set of 3D Gaussian primitives is initialized based on the sparse point cloud result, denoted as . ,in, Indicates the first The central position of a Gaussian primitive, Indicates the first The covariance matrix of a Gaussian primitive. Indicates the first The transparency of a Gaussian primitive. Indicates the first The color attribute of a Gaussian primitive This represents the number of Gaussian primitives. Simultaneously, the sparse reconstruction result is projected onto the reference image plane to obtain an initial foreground mask, providing basic constraint information for subsequent construction of a structure-aware mask.

[0073] Step S2: Perform structure-aware enhancement on the initial foreground mask to obtain a structure-aware mask, thereby enhancing the geometric representation of thin-structure regions and occlusion boundary regions. The specific steps are as follows:

[0074] Step S2.1: Apply the initial foreground mask. Perform morphological dilation and erosion operations to obtain a preliminary corrected mask. Specifically, it is expressed as:

[0075]

[0076] in, Indicates the initial foreground mask. This represents the initial correction mask after morphological processing. Represents the structural element corresponding to morphological operations. This indicates the expansion operation. This represents the erosion operation.

[0077] Step S2.2: For the pixel region in the preliminary correction mask, calculate the surface normal vector consistency of adjacent pixels, and filter out regions that do not satisfy geometric continuity based on the normal vector consistency.

[0078] Step S2.3: The consistency of the surface normal vectors of adjacent pixels satisfies the following constraints:

[0079]

[0080] in, and Representing pixels and pixels The corresponding surface normal vector, This represents the normal consistency threshold.

[0081] Step S2.4: Based on the morphological processing results and the normal vector consistency filtering results, obtain the structure-aware mask. It is used to enhance the geometric representation of thin structural regions and occluded boundary regions.

[0082] Step S3: Based on the structure-aware mask, a cascaded multi-resolution optimization strategy is adopted to progressively optimize the position parameters, covariance parameters, transparency parameters, and color parameters of the three-dimensional Gaussian primitive from coarse to fine.

[0083] After optimization in step S3, an updated set of 3D Gaussian primitives is obtained, which includes position parameters, covariance parameters, transparency parameters, and color parameters updated through multi-resolution training. These parameters serve as inputs for anisotropic adaptive adjustment in step S4. The specific steps are as follows:

[0084] Step S3.1: Set up a low-resolution optimization stage. At a lower resolution, perform coarse-scale initialization of the covariance parameters of the 3D Gaussian primitives, and jointly optimize the position parameters, covariance parameters, transparency parameters, and color parameters of the Gaussian primitives to restore the global geometry of the scene.

[0085] Step S3.2: Set up the medium resolution optimization stage. According to the resolution change, the covariance parameter of the 3D Gaussian primitive is scaled and updated, and the position parameter, covariance parameter, transparency parameter and color parameter of the Gaussian primitive are jointly optimized to enhance the medium-scale structure expression of the scene.

[0086] Step S3.3: Set up a high-resolution optimization stage to further refine and optimize the 3D Gaussian primitives at a higher resolution to improve the expressive power of boundary regions and high-frequency detail regions.

[0087] Step S3.4: Through progressive training in low-resolution, medium-resolution, and high-resolution optimization stages, cascaded multi-resolution optimization from global structure restoration to local detail enhancement is achieved.

[0088] Step S4: Based on the structural differences between the rendered image and the corresponding reference viewpoint real image input in Step S1, adaptively adjust the principal direction and anisotropic shape of the 3D Gaussian primitive. The specific steps are as follows:

[0089] Step S4.1: Determine the influence region of each 3D Gaussian primitive in the image space. Calculate the image gradient and construct the gradient structure tensor. Specifically, it is expressed as:

[0090] ;

[0091] in, Indicates the first The region of influence of a Gaussian primitive in the image space. Indicates the pixel position in the affected area. Indicates pixel position Image gradient at that location, Indicates the first The gradient structure tensor corresponding to each Gaussian primitive.

[0092] Step S4.2: Based on the gradient structure tensor The principal eigenvector determines the first The principal direction of the Gaussian primitive is determined, and the major axis of the Gaussian primitive is aligned with the principal structure direction of the image.

[0093] Step S4.3: Based on the gradient structure tensor eigenvalue distribution adjustment The degree of anisotropy of a Gaussian primitive, wherein the degree of anisotropy satisfies:

[0094] ;

[0095] in, Indicates the first The degree of anisotropy of a Gaussian primitive, Indicates the adjustment factor. Represents the gradient structure tensor The largest eigenvalue, Represents the gradient structure tensor The average eigenvalues.

[0096] Step S5: Integrate Bayesian uncertainty, color information entropy, gradient strength, and view coverage to evaluate the importance of the 3D Gaussian primitives and perform adaptive pruning. The specific steps are as follows:

[0097] Step S5.1: Calculate the Bayesian uncertainty for each three-dimensional Gaussian primitive. Color information entropy Gradient strength and view coverage .

[0098] Step S5.2: Based on Bayesian uncertainty... Color information entropy Gradient strength and view coverage Construct the first Importance rating of Gaussian primitives Specifically, it is expressed as:

[0099] ;

[0100] in, Indicates the first The importance score of each Gaussian primitive , , and These represent the weight coefficients corresponding to Bayesian uncertainty, color information entropy, gradient strength, and view coverage, respectively.

[0101] Step S5.3: Score based on the importance of each three-dimensional Gaussian primitive. All Gaussian primitives are sorted, and adaptive pruning is performed on low-importance Gaussian primitives to reduce the number of redundant Gaussian primitives while preserving scene structure details and rendering quality.

[0102] Step S6: Output the optimized 3D Gaussian primitive set, and perform target viewpoint image rendering based on the 3D Gaussian primitive set. The optimized 3D Gaussian primitive set is represented as... The target view image is a result of differentiable rendering based on an optimized set of three-dimensional Gaussian primitives, used to characterize the number of final scene rendering outputs of the method.

[0103] Example 1:

[0104] In this embodiment, the input image data sample is as follows: Figure 2 As shown; Figure 3 This is a schematic diagram of the construction mask in the embodiment; Figure 4 As can be seen from the final output of this embodiment, the target image output by the present invention has good structural continuity and detail expression ability in thin structure regions, edge regions and texture detail regions.

[0105] As can be seen from the above embodiments, the present invention first inputs multi-view image data and camera parameters, and initializes a three-dimensional Gaussian primitive; then it constructs a structure-aware mask (such as...). Figure 3 Secondly, a cascaded multi-resolution optimization strategy is adopted to progressively optimize the 3D Gaussian primitives from coarse to fine. Next, the principal orientation and anisotropic shape of the Gaussian primitives are adaptively adjusted based on rendering feedback. Then, a multi-index evaluation strategy is constructed to assess the importance of the Gaussian primitives and perform adaptive pruning. Finally, the optimized 3D Gaussian primitive set is output, and the target image is rendered (e.g., ...). Figure 4(As shown). This invention fully utilizes the synergistic effects of structure-aware enhancement, cascaded multi-resolution optimization, adaptive adjustment of rendering feedback, and multi-dimensional constraint pruning to effectively improve the structural representation capability and rendering quality of target viewpoint images in complex scenes.

[0106] Based on the final experimental results ( Figure 4 ),for Figure 2 Using the same sample data, if the existing technical solution proposed at ECCV 2024 ("Mini-Splatting: Representing Scenes with a Constrained Number of Gaussians") is applied, the output target image will be as follows: Figure 5 As shown.

[0107] contrast Figure 4 and Figure 5 The results show that the present invention has better structural integrity and rendering clarity in thin structure regions, occlusion boundary regions and high frequency detail regions, and can effectively improve the problems of edge blurring, lack of detail and unstable structural expression in existing methods.

[0108] In summary, this invention directly optimizes the structural representation of 3D Gaussian primitives and the rendering quality of the target viewpoint in complex scenes. The optimization process involves structure-aware mask constraints, multi-resolution progressive optimization, shape adjustment driven by rendering feedback, and multi-dimensional importance pruning (the pruning in this invention serves to suppress redundant Gaussian representations while preserving rendering quality and structural details). The final output is an optimized set of 3D Gaussian primitives and their target viewpoint rendered image. Specifically, this invention first constructs a structure-aware mask in image space through morphological processing and normal consistency filtering, moving structural constraints to the optimization entry point. Second, it achieves global structure restoration and local detail enhancement by adjusting Gaussian parameters in stages. It constructs a gradient structure tensor based on the structural differences between the rendered image and the real image, and dynamically adjusts the principal direction and anisotropy level accordingly, essentially a rendering feedback-driven adaptive shape update mechanism. Finally, it employs a set of multi-dimensional importance evaluation metrics: Bayesian uncertainty, color information entropy, gradient strength, and viewpoint coverage. This invention effectively improves the structural representation and detail restoration in complex scenes, rendering high-quality scene images.

Claims

1. A structure-aware adaptive Gaussian splashing method, characterized in that, Includes the following steps: Step S1: Input multi-view images of the scene And the corresponding camera parameters, and initialize the set of 3D Gaussian primitives based on the sparse reconstruction results. and initial foreground mask ; in, Indicates the number of input viewpoints. Indicates the first The central position of a Gaussian primitive, Indicates the first The covariance matrix of a Gaussian primitive. Indicates the first The transparency of a Gaussian primitive. Indicates the first The color attribute of a Gaussian primitive Indicates the number of Gaussian primitives; Step S2: Perform structure-aware enhancement on the initial foreground mask to obtain the structure-aware mask. To enhance the geometric representation of thin-structured regions and occluded boundary regions; Step S3: Based on structure-aware masking, a cascaded multi-resolution optimization strategy is used to progressively optimize the position parameters, covariance parameters, transparency parameters, and color parameters of the 3D Gaussian primitives from coarse to fine, and output the set of 3D Gaussian primitives after multi-resolution optimization. Step S4: Based on the rendered image and the original multi-view image from step S1 The structural differences between them allow for adaptive adjustment of the principal direction and anisotropic shape of the three-dimensional Gaussian primitives; Step S5: Integrate Bayesian uncertainty, color information entropy, gradient strength, and view coverage to evaluate the importance of the 3D Gaussian primitives and perform adaptive pruning. Step S6: Output the optimized set of 3D Gaussian primitives. And based on the set of three-dimensional Gaussian primitives Perform target viewpoint image rendering.

2. The structure-aware adaptive Gaussian splashing method according to claim 1, characterized in that, The detailed steps of step S2 are as follows: Step S2.1: Apply the initial foreground mask. Perform morphological dilation and erosion operations to obtain a preliminary corrected mask. : ； in, Indicates the initial foreground mask. This represents the initial correction mask after morphological processing. Represents the structural element corresponding to morphological operations. This represents the expansion operation. This represents the erosion operation; Step S2.2: For the initial modified mask In the pixel region, the surface normal vector is estimated based on the local geometric relationship of the corresponding pixel, and the normal vector consistency between adjacent pixels is calculated; when the normal vector consistency between adjacent pixels does not meet the preset threshold condition, the corresponding pixel region is determined as a geometrically discontinuous region and removed from the preliminary correction mask; Meanwhile, the consistency of surface normal vectors of adjacent pixels satisfies the following constraints: ； in, and Representing pixels and pixels The corresponding surface normal vector, Indicates the normal consistency threshold; Step S2.3: Based on the preliminary correction mask obtained in step S2.1 The structure-aware mask is obtained by matching the normal vector consistency screening results obtained in step S2.

2. ;in, This represents the final generated structure-aware mask, used to enhance the geometric representation of thin-structure regions and occluded boundary regions.

3. The structure-aware adaptive Gaussian splashing method according to claim 1, characterized in that, The detailed steps of step S3 are as follows: Step S3.1: Set up a low-resolution optimization stage. At a lower resolution after downsampling relative to the target rendering resolution, coarse-scale initialization of the covariance parameters of the 3D Gaussian primitives is performed. Based on the differentiable rendering error, joint gradient optimization is performed on the position parameters, covariance parameters, transparency parameters and color parameters of the Gaussian primitives to restore the global geometry of the scene. Step S3.2: Set up the medium-resolution optimization stage. Based on the resolution change, scale and update the covariance parameters of the 3D Gaussian primitives, and continue to jointly optimize the position parameters, covariance parameters, transparency parameters, and color parameters of the Gaussian primitives to enhance the medium-scale structural representation of the scene; the scaling update of the covariance parameters satisfies: ;in, This represents the covariance parameter in the low-resolution stage. This represents the covariance parameter at the medium resolution stage. and These represent the scale coefficients corresponding to the low-resolution and medium-resolution stages, respectively. Step S3.3: Set up a high-resolution optimization stage. At a higher resolution close to the target rendering resolution, further refine and optimize the 3D Gaussian primitives to improve the expressive power of boundary areas and high-frequency detail areas. Step S3.4: Through progressive training in low-resolution, medium-resolution, and high-resolution optimization stages, cascaded multi-resolution optimization from global structure restoration to local detail enhancement is achieved.

4. The structure-aware adaptive Gaussian splashing method according to claim 1, characterized in that, The detailed steps of step S4 are as follows: Step S4.1, the first Projecting a three-dimensional Gaussian primitive onto the image plane of the current viewpoint yields its region of influence in the image space. The image gradient is calculated within the affected region, and a gradient structure tensor is constructed. : ; in, Indicates the first The region of influence of a Gaussian primitive in the image space. Indicates the pixel position in the affected area. Indicates pixel position Image gradient at that location, Indicates the first The gradient structure tensor corresponding to each Gaussian primitive; Step S4.2: Based on the gradient structure tensor The principal eigenvector determines the first The main direction of the Gaussian primitive is determined, and the major axis of the Gaussian primitive is aligned with the main structure direction of the image. Step S4.3: Based on the gradient structure tensor eigenvalue distribution adjustment The degree of anisotropy of a Gaussian primitive satisfies: ； in, Indicates the first The degree of anisotropy of a Gaussian primitive, Indicates the adjustment factor. Represents the gradient structure tensor The largest eigenvalue, Represents the gradient structure tensor The average eigenvalues; Step S4.4: Update the shape parameters of the 3D Gaussian primitive based on the adjusted principal direction and anisotropy degree, specifically: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] The covariance matrix of a Gaussian primitive is decomposed into a rotation matrix and a scale matrix. The rotation matrix is ​​updated according to the principal direction and the scale matrix is ​​updated according to the degree of anisotropy to reconstruct the updated covariance matrix, thereby improving the fitting ability of edge regions and detail regions.

5. The structure-aware adaptive Gaussian splashing method according to claim 1, characterized in that, The detailed steps of step S5 are as follows: Step S5.1: Calculate the Bayesian uncertainty for each three-dimensional Gaussian primitive. Color information entropy Gradient strength and view coverage The Bayesian uncertainty Represented as: ； in, Indicates the first The Gaussian primitive in the first Prediction results under random sampling or random deactivation This represents the mean of the corresponding prediction results. Indicates the number of samples; The color information entropy Represented as: ； in, Indicates the first The color distribution of the Gaussian primitives falls into the first... The probability of each bin. Indicates the number of color bins; The gradient strength Represented as: ； in, Indicates the first A Gaussian primitive affects the number of pixels within a region; The view coverage Represented as: ； in, Indicates the first The Gaussian primitive in the first An indicator function for visibility from a given viewpoint; it takes the value 1 if visible and 0 otherwise. Indicates the number of input viewpoints; Step S5.2: Based on Bayesian uncertainty... Color information entropy Gradient strength and view coverage Construct the first Importance rating of Gaussian primitives : ； in, Indicates the first The importance score of each Gaussian primitive , , and These represent the weighting coefficients corresponding to Bayesian uncertainty, color information entropy, gradient strength, and view coverage, respectively. Step S5.3: Score based on the importance of each three-dimensional Gaussian primitive. All Gaussian primitives are sorted, and low-importance Gaussian primitives are selected for adaptive pruning according to a preset pruning ratio to reduce the number of redundant Gaussian primitives while maintaining scene structure details and rendering quality.

6. The structure-aware adaptive Gaussian splashing method according to claim 1, characterized in that, The output of step S6 includes an optimized set of 3D Gaussian primitives and a target view image rendered based on the set of 3D Gaussian primitives; wherein, the optimized set of 3D Gaussian primitives is represented as... The target view image is a result of differentiable rendering based on an optimized set of three-dimensional Gaussian primitives, used to characterize the final scene rendering output of the method.

Images