A 3DGS new view rendering quality improvement method based on Gaussian visibility

Abstract

The application relates to the technical field of three-dimensional reconstruction. By providing a 3D GS new view rendering quality improvement method based on Gaussian visibility, the method comprises the following steps: performing sparse point cloud reconstruction processing on input multi-view images to generate initial 3D Gaussian distribution; performing visibility judgment processing on each Gaussian to filter Gaussians invisible relative to the view, and obtaining a visibility judgment processing result; performing integral calculation processing on the coverage range and contribution intensity of each Gaussian in the view according to the visibility judgment processing result to generate Gaussian visibility weight; performing hierarchical processing on the Gaussians according to the Gaussian visibility weight, dynamically adjusting a gradient accumulation strategy, and obtaining a hierarchical processing result; integrating the hierarchical processing result, optimizing a Gaussian densification mechanism, and generating refined 3D Gaussian distribution; and performing new view rendering processing and outputting an image without artifacts, so as to solve the problems of insufficient consideration of occlusion relationship, insufficient large-size Gaussian splitting capability and mismatch between scene representation and high-frequency texture.

Description

Technical Field

[0001] This invention relates to the field of 3D reconstruction technology, and in particular to a new 3DGS view rendering quality improvement method based on Gaussian visibility. Background Technology

[0002] With the rapid development of technologies such as virtual reality (VR), augmented reality (AR), and autonomous driving, novel view synthesis (NVS) has become one of the core research directions in the interdisciplinary field of computer vision and graphics. Its goal is to generate high-quality scene rendering results from arbitrary viewpoints using input images with limited viewpoints, providing key technical support for applications such as immersive interaction, digital twins, and film and television special effects.

[0003] However, the typical new view compositing technique 3D Gaussian Splatter (3DGS) does not fully consider the occlusion relationship between Gaussians in the visibility judgment mechanism, and the gradient accumulation strategy adopts the "bidding" mechanism, which results in insufficient average gradient level of large-size Gaussians, inhibiting their splitting ability and exacerbating the mismatch between scene representation and high-frequency textures. Summary of the Invention

[0004] Therefore, it is necessary to provide a method for improving the rendering quality of new 3DGS views based on Gaussian visibility to address the above-mentioned technical problems, such as insufficient consideration of occlusion relationships, insufficient large-size Gaussian splitting capability, and mismatch between scene representation and high-frequency textures. This would reduce needle-like artifacts and detail blurring in the rendering results, and improve the accuracy and visual effect of new view compositing.

[0005] Firstly, this application provides a method for improving the rendering quality of a new 3DGS view based on Gaussian visibility, the method comprising:

[0006] Sparse point cloud reconstruction processing is performed on the input multi-view images to generate an initial 3D Gaussian distribution;

[0007] Based on the projection area of ​​the existing 3D Gaussian distribution and the occlusion relationship between Gaussians in the current iteration, a visibility judgment process is performed on each Gaussian, and Gaussians that are not visible relative to the view are filtered out to obtain the visibility judgment result.

[0008] Based on the visibility judgment processing results, the coverage and contribution intensity of each visible Gaussian in the view are integrally calculated to generate Gaussian visibility weights.

[0009] Based on Gaussian visibility weights, Gaussian layers are processed in layers, and the gradient accumulation strategy is dynamically adjusted to obtain the layered processing results.

[0010] By integrating the results of hierarchical processing and optimizing the Gaussian densification mechanism, a refined 3D Gaussian distribution is generated.

[0011] Based on a refined 3D Gaussian distribution, a new view rendering process is performed to output an image without artifacts.

[0012] Furthermore, based on Gaussian visibility weights, Gaussian layers are processed hierarchically, and the gradient accumulation strategy is dynamically adjusted to obtain the hierarchical processing results, including:

[0013] Based on the Gaussian visibility weights, the Gaussians are layered to generate a set of highly visible Gaussians and a set of low-visibility Gaussians.

[0014] For the Gaussians in the set of highly visible Gaussians, process them according to the original gradient accumulation strategy to generate highly visible gradient results;

[0015] The Gaussians in the set of low-visibility Gaussians are processed based on the dynamic decay coefficient to generate a corrected low-visibility gradient result.

[0016] Integrate the highly visible gradient results and the corrected low-visibility gradient results to adjust and optimize the gradient accumulation strategy.

[0017] Furthermore, the Gaussians in the highly visible Gaussian set are processed according to the original gradient accumulation strategy to generate highly visible gradient results, including:

[0018] Extracting the positional gradients of Gaussians from the set of highly visible Gaussians during backpropagation of stochastic gradient descent;

[0019] The position gradient is subjected to gradient filtering to remove noise interference, resulting in the filtered position gradient.

[0020] Based on the filtered position gradients, gradient accumulation is performed on the Gaussians in the set of highly visible Gaussians to generate highly visible gradient results.

[0021] Furthermore, the Gaussians in the low-visibility Gaussian set are processed based on a dynamic decay coefficient to generate corrected low-visibility gradient results, including:

[0022] A dynamic decay coefficient is generated based on dynamic Gaussian visibility weights;

[0023] The dynamic attenuation coefficient is judged to be reasonable, and outlier interference is eliminated to obtain the denoised dynamic attenuation coefficient.

[0024] For Gaussians in the low-visibility Gaussian set, gradient weighting is performed based on the dynamic decay coefficient to generate corrected low-visibility gradient results.

[0025] Based on the relationship between the validated gradient and the spatial coverage of the low-visibility Gaussian set, a corrected low-visibility gradient result is generated.

[0026] Furthermore, the results of the hierarchical processing are integrated, the Gaussian compaction mechanism is optimized, and a refined 3D Gaussian distribution is generated, including:

[0027] Gradient integration processing is performed on the highly visible gradient results and the corrected low-visible gradient results in the hierarchical processing results to generate integrated gradient results;

[0028] Based on the integrated gradient results, the Gaussians in the scene are split or cloned to generate new Gaussians to fill the areas in the scene that are not yet fully reconstructed.

[0029] Based on the new Gaussian generated by splitting or cloning, update the position, covariance, and color attributes of the Gaussian to generate the refined 3D Gaussian distribution.

[0030] Furthermore, based on the integrated gradient results, the Gaussians in the scene are split or cloned to generate new Gaussian-filled regions in the scene that are not yet fully reconstructed, including:

[0031] Based on the integrated gradient results, Gaussians with sizes exceeding the average scale of the scene are split to ensure that excessively large Gaussians in the scene can be correctly split to characterize high-frequency texture regions in the scene.

[0032] Based on the integrated gradient results, Gaussians with sizes smaller than the average scene scale are cloned to ensure that there are a sufficient number of Gaussians to represent the scene.

[0033] Furthermore, sparse point cloud reconstruction processing is performed on the input multi-view images to generate an initial 3D Gaussian distribution, including:

[0034] Feature extraction and matching processing are performed on the input multi-view images to generate cross-view matching point pairs;

[0035] Sparse point cloud reconstruction is performed based on matching point pairs to generate an initial sparse point cloud.

[0036] Covariance estimation is performed on each point in the initial sparse point cloud to generate the covariance matrix of each point.

[0037] An initial 3D Gaussian distribution is constructed based on the covariance matrix and preset Gaussian attribute parameters.

[0038] Secondly, this application also provides a new 3DGS view rendering quality improvement system based on Gaussian visibility, the system comprising:

[0039] The sparse point cloud reconstruction module is used to perform sparse point cloud reconstruction processing on the input multi-view images to generate an initial 3D Gaussian distribution.

[0040] The visibility filtering module is used to perform visibility judgment processing on each Gaussian based on the projection area of ​​the existing Gaussian distribution in the current iteration and the occlusion relationship between Gaussians, and to filter out Gaussians that are not visible relative to the view, so as to obtain the visibility judgment processing result.

[0041] The weight calculation module is used to perform integral calculation on the coverage and contribution intensity of each visible Gaussian in the view based on the visibility judgment processing results, and generate Gaussian visibility weights.

[0042] The hierarchical gradient module is used to perform hierarchical processing on Gaussian based on Gaussian visibility weights, dynamically adjust the gradient accumulation strategy, and obtain the hierarchical processing result.

[0043] The densification optimization module is used to integrate the results of hierarchical processing, optimize the Gaussian densification mechanism, and generate a refined 3D Gaussian distribution.

[0044] The rendering output module is used to perform new view rendering based on a refined 3D Gaussian distribution, and output an image without artifacts.

[0045] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the methods in the first aspect of this application.

[0046] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the methods in the first aspect of this application.

[0047] The technical solution provided in this application includes the following technical effects: It provides a method for improving the rendering quality of new 3DGS views based on Gaussian visibility, comprising: performing sparse point cloud reconstruction processing on the input multi-view image to generate an initial 3D Gaussian distribution; performing visibility judgment processing on each Gaussian based on the projection area of ​​the existing Gaussian distribution in the current iteration and the occlusion relationship between Gaussians, filtering out Gaussians that are not visible relative to the view, and obtaining the visibility judgment processing result; performing integral calculation processing on the coverage and contribution intensity of each visible Gaussian in the view according to the visibility judgment processing result, generating Gaussian visibility weights; performing layered processing on the Gaussians based on the Gaussian visibility weights, dynamically adjusting the gradient accumulation strategy, and obtaining layered processing results; combining the layered processing results, optimizing the Gaussian densification mechanism to generate a refined 3D Gaussian distribution; and performing new view rendering processing based on the refined 3D Gaussian distribution to output an image without artifacts, thereby solving the problems of insufficient consideration of occlusion relationships, insufficient large-size Gaussian splitting ability, and mismatch between scene representation and high-frequency textures, thus reducing needle-like artifacts and detail blurring in the rendering result, and improving the accuracy and visual effect of new view synthesis. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is a flowchart of a method for improving the rendering quality of a new 3DGS view based on Gaussian visibility, as described in one embodiment of the present invention.

[0050] Figure 2 This is a structural diagram of a 3DGS new view rendering quality improvement system based on Gaussian visibility, according to one embodiment of the present invention. Detailed Implementation

[0051] To make the above-mentioned objects, features, and advantages of this application more apparent and understandable, the specific implementation methods of this application will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0052] In this application, 3DGS refers to 3D Gaussian Splatting (3DGS).

[0053] like Figure 1 As shown, this application provides a method for improving the rendering quality of a new 3DGS view based on Gaussian visibility. The method includes:

[0054] S101: Perform sparse point cloud reconstruction processing on the input multi-view image to generate an initial 3D Gaussian distribution.

[0055] Specifically, a multi-view feature matching algorithm based on deep feature descriptors (such as SIFT or SuperPoint) is adopted. Through cross-view feature point extraction and robust matching (such as the RANSAC algorithm iteratively filtering out mismatched point pairs), a high-confidence sparse matching point set is generated. Then, based on the Structure from Motion (SFM) framework, the three-dimensional spatial coordinates of the matching points are calculated using the multi-view geometric triangulation principle. The camera pose parameters and the geometric position of the sparse point cloud are jointly optimized through Bundle Adjustment to construct an initial sparse point cloud with geometric consistency optimization. Next, for each sparse point, the reconstruction confidence is fused based on the local geometric distribution of its neighboring points and the KNN nearest neighbor algorithm to generate a covariance matrix representing the Gaussian spatial distribution. Then, the covariance matrix is ​​combined with preset Gaussian attribute parameters (including the view-related color base based on spherical harmonics, initial opacity, and illumination response coefficient) to construct a 3D Gaussian distribution covering the key geometric structure of the scene, providing physically reasonable initialization conditions for subsequent gradient optimization and densification processing.

[0056] S102: Based on the projection area of ​​the existing 3D Gaussian distribution and the occlusion relationship between Gaussians in the current iteration, perform visibility judgment processing on each Gaussian, filter out Gaussians that are not visible relative to the current view, and obtain the visibility judgment processing result.

[0057] Specifically, each 3D Gaussian distribution is projected onto the 2D image plane of the target view to form a 2D Gaussian distribution. This step is achieved through a projection transformation from the world coordinate system to the pixel coordinate system, ensuring that the projection area of ​​the Gaussian is accurately mapped to the image plane. Based on the projection area and depth information of the Gaussian, it is determined whether the Gaussian is visible. Specifically, this involves checking whether the projection of the Gaussian in the current view falls within the view boundary, and combining this with depth sorting to determine if the Gaussian is occluded by other Gaussians in front. The Gaussians are sorted using depth information to ensure that Gaussians located at the end of the depth sequence and occluded by those in front are considered invisible during rendering. This step is achieved through visibility ordering, ensuring that only visible Gaussians are considered during rendering. Through the above judgment and sorting process, Gaussians that are not visible relative to the view are filtered out, generating a visibility judgment result. Invisible Gaussians are excluded from subsequent rendering and optimization processes, thereby improving rendering efficiency and quality. Through these steps, invisible Gaussians can be effectively filtered out, ensuring that subsequent rendering and optimization processes are based only on visible Gaussians, thus improving the quality and efficiency of new view rendering.

[0058] S103: Based on the visibility judgment processing results, perform integral calculation on the coverage and contribution intensity of each Gaussian in the view to generate Gaussian visibility weights.

[0059] Specifically, based on the Gaussian projection area, its coverage area in the view is calculated. This step determines the effective coverage area of ​​the Gaussian in the view by analyzing the intersection of the Gaussian's projection area and the view area. Combining the Gaussian's coverage area and the 2D Gaussian distribution of its projection, its contribution to the view, i.e., the visibility weight, is calculated. This step analyzes the opacity variation of the Gaussian in the coverage area, comprehensively considering the size of its projection area and the contribution strength, and uses a double integral mathematical method to ensure that the weight can accurately reflect the visibility and contribution of the Gaussian in the view. Through the above steps, Gaussian visibility weights can be generated, providing a basis for subsequent rendering and optimization.

[0060] S104: Based on Gaussian visibility weights, Gaussian layers are processed and the gradient accumulation strategy is dynamically adjusted to obtain the layered processing results.

[0061] Specifically, based on Gaussian visibility weights, Gaussians are categorized into "highly visible" and "lowly visible" classes. Highly visible Gaussians are processed using the original gradient accumulation strategy, while low-visibility Gaussians undergo gradient weighting with a dynamic decay coefficient. For highly visible Gaussians, the original gradient accumulation strategy is applied; for low-visibility Gaussians, the gradient accumulation is adjusted using a dynamic decay coefficient to balance the gradient contributions of Gaussians with different visibility levels. Combining the gradient results of highly and low-visibility Gaussians generates a layered gradient accumulation strategy, providing a basis for subsequent Gaussian densification optimization. Through these steps, the gradient accumulation strategy can be dynamically adjusted to optimize the Gaussian densification process, improving the quality and efficiency of new view rendering.

[0062] S105: Integrate the results of the hierarchical processing, optimize the Gaussian compaction mechanism, and generate a refined 3D Gaussian distribution.

[0063] Specifically, based on the highly visible gradient results and the corrected low-visibility gradient results from the layered processing, gradient integration is performed to generate an integrated gradient result. This step ensures the integrity and rationality of gradient information by comprehensively considering the gradient contributions of Gaussians with different visibility. According to the integrated gradient result, Gaussians larger than the scene's average scale are split, and Gaussians smaller than the scene's average scale are cloned. Splitting and cloning generate more new Gaussians to fill areas in the scene that are not yet fully reconstructed. This step ensures that excessively large Gaussians can be correctly split, while optimizing stable small-sized Gaussians remains largely unchanged, ensuring that the split Gaussians better match the scene's detailed textures. Based on the split and cloned Gaussians, the position, covariance, and color attributes of the Gaussians are updated to generate a refined 3D Gaussian distribution. This step optimizes the Gaussian parameters to ensure a more accurate representation of the scene. Through these steps, the Gaussian densification mechanism can be optimized to generate a refined 3D Gaussian distribution, thereby improving the quality and efficiency of new view rendering.

[0064] S106: Based on a refined 3D Gaussian distribution, perform new view rendering processing to output an image without artifacts.

[0065] Specifically, the parameters of the refined 3D Gaussian distribution (such as position, covariance matrix, color, and transparency) are optimized to ensure accurate representation of details and high-frequency textures in the scene. The optimized 3D Gaussian distribution is projected onto the 2D image plane of the target view to form a 2D Gaussian distribution. This step is achieved through a projection transformation from the world coordinate system to the pixel coordinate system, ensuring that the projected area of ​​the Gaussian is accurately mapped to the image plane. The Gaussians are ordered using depth information to ensure that Gaussians located at the back of the depth sequence are considered invisible during rendering when occluded by Gaussians in front. This step is achieved through visibility ordering, ensuring that only visible Gaussians are considered during rendering. Alpha blending rendering is performed based on the depth-ordered Gaussian distribution. The final image is formed by overlaying multiple semi-transparent Gaussian distributions. This process is not only efficient but also maintains high quality in real-time rendering. By analyzing the visibility weights and coverage of the Gaussians, potential needle-like artifacts and detail blurring are identified and eliminated. This step is achieved by filtering outlier Gaussians and optimizing the splitting and cloning strategy of the Gaussians. Post-processing of the rendered image, such as removing noise or unwanted parts from the background, further optimizes the image's appearance. Through these steps, new view rendering can be performed based on a refined 3D Gaussian distribution, outputting high-quality images without artifacts, thereby improving the accuracy and visual effect of new view compositing.

[0066] One embodiment of this application provides a method for improving the rendering quality of new 3DGS views based on Gaussian visibility, comprising: performing sparse point cloud reconstruction processing on the input multi-view image to generate an initial 3D Gaussian distribution; performing visibility judgment processing on each Gaussian based on the projection area of ​​the existing 3D Gaussian distribution in the current iteration and the occlusion relationship between Gaussians, filtering out Gaussians that are not visible relative to the view, and obtaining a visibility judgment processing result; calculating the coverage area and contribution intensity of each Gaussian in the view according to the visibility judgment processing result, and generating Gaussian visibility weights; performing layer processing on the Gaussians based on the Gaussian visibility weights, dynamically adjusting the gradient accumulation strategy, and obtaining a layer processing result; integrating the layer processing result, optimizing the Gaussian densification mechanism, and generating a refined 3D Gaussian distribution; performing new view rendering processing based on the refined 3D Gaussian distribution, and outputting an image without artifacts, in order to solve the problems of insufficient consideration of occlusion relationships, insufficient large-size Gaussian splitting ability, and mismatch between scene representation and high-frequency textures, thereby reducing needle-like artifacts and detail blurring in the rendering result, and improving the accuracy and visual effect of new view synthesis.

[0067] Furthermore, based on Gaussian visibility weights, Gaussian layers are processed hierarchically, and the gradient accumulation strategy is dynamically adjusted to obtain the hierarchical processing results, including:

[0068] Based on the visibility weight of Gaussians, the Gaussians are classified to generate a set of highly visible Gaussians and a set of low-visibility Gaussians.

[0069] For Gaussians in the set of highly visible Gaussians, process them according to the original gradient accumulation strategy to generate highly visible gradient results;

[0070] For Gaussians in the set of low-visibility Gaussians, the dynamic decay coefficient is used to process them and generate corrected low-visibility gradient results.

[0071] Integrate highly visible gradient results and corrected low-visible gradient results to generate hierarchical processing results adjusted by the gradient accumulation strategy.

[0072] Specifically, Gaussian visibility weights are generated based on the coverage and contribution strength of Gaussians in the view. This step is achieved by analyzing the visibility judgment results and the projection area of ​​the Gaussians. Based on the Gaussian visibility weights, Gaussians are divided into two categories: "highly visible" and "lowly visible." Highly visible Gaussians are processed using the original gradient accumulation strategy, while low-visibility Gaussians are processed with a dynamic decay coefficient for gradient weighting. For highly visible Gaussians, the original gradient accumulation strategy is applied to generate highly visible gradient results. This step ensures that the gradient accumulation of Gaussians accurately reflects their contribution to the view. For low-visibility Gaussians, the dynamic decay coefficient is applied to generate corrected low-visibility gradient results. This step balances the gradient contributions of Gaussians with different visibility levels by adjusting the gradient accumulation. Combining the highly visible gradient results and the corrected low-visibility gradient results, a layered processing result after adjusting the gradient accumulation strategy is generated. This step ensures that the gradient accumulation strategy can dynamically adapt to changes in the visibility of Gaussians, thereby optimizing the Gaussian densification process.

[0073] Furthermore, the Gaussians in the highly visible Gaussian set are processed according to the original gradient accumulation strategy to generate highly visible gradient results, including:

[0074] Based on the Gaussian properties in the highly visible Gaussian set, the original gradient accumulation parameters are extracted;

[0075] Based on the original gradient accumulation parameters, gradient accumulation is performed on the Gaussians in the set of highly visible Gaussians to generate uncorrected gradient results.

[0076] The uncorrected gradient results were subjected to gradient filtering to remove noise interference, resulting in filtered gradient results.

[0077] Based on the spatial distribution relationship between the filtered gradient results and the highly visible Gaussian set, highly visible gradient results are generated.

[0078] Specifically, based on the spatial distribution attributes of highly visible Gaussians (including position gradient, covariance matrix gradient, and color gradient), the original gradient accumulation parameters (such as the mean gradient magnitude and orientation consistency coefficient) are extracted in different training iteration cycles. Then, according to the 3DGS standard gradient accumulation strategy, the gradients contributed by multiple views in the current iteration cycle are weighted and summed to generate uncorrected initial gradient results. Next, a joint screening mechanism based on statistical distribution threshold and gradient orientation consistency verification is adopted to remove abnormal gradient components caused by partial view occlusion or insufficient sampling, and retain effective gradients that conform to the geometric evolution trend of the scene. Then, combined with the spatial density distribution map of highly visible Gaussians, the selected gradients are subjected to local energy normalization and orientation smoothing interpolation to ensure that the Gaussians after gradient adjustment maintain spatial continuity in splitting direction and position offset, thereby generating highly visible gradient results that are adapted to the high-frequency texture of the scene, providing a stable driving signal for subsequent densification splitting.

[0079] Furthermore, the Gaussians in the low-visibility Gaussian set are processed based on a dynamic decay coefficient to generate corrected low-visibility gradient results, including:

[0080] A dynamic attenuation coefficient is generated based on the dynamic visibility weight distribution of the low-degree visibility Gaussian set.

[0081] For Gaussians in the low-visibility Gaussian set, gradient weighting is performed based on the dynamic decay coefficient to generate a preliminary corrected gradient;

[0082] The initial corrected gradient is subjected to gradient stability verification processing to remove abnormal gradient interference and obtain the verified gradient.

[0083] Based on the relationship between the validated gradient and the spatial coverage of the low-visibility Gaussian set, a corrected low-visibility gradient result is generated.

[0084] Specifically, based on the visibility weight distribution dynamically updated by the low-visibility Gaussian during training, a dynamic decay coefficient adaptive to the current scene optimization stage is generated by statistically analyzing the range fluctuation characteristics of its weight values. Then, the dynamic decay coefficient is element-wise weighted and fused with the original gradient of the low-visibility Gaussian to generate a preliminary corrected gradient. Next, the stability of the preliminary corrected gradient is verified based on statistical distribution thresholds and gradient direction consistency, eliminating abnormal gradients that deviate from the group distribution or have significantly conflicting directions, while retaining effective gradient components that conform to scene geometric constraints. Finally, combined with the spatial coverage density distribution of the low-visibility Gaussian, the verified gradient is subjected to region smoothing and energy normalization to ensure that the corrected gradient result meets the requirements of subsequent densification optimization in terms of spatial continuity and physical rationality, thereby generating the final corrected low-visibility gradient result.

[0085] Furthermore, by integrating the results of the hierarchical processing, the Gaussian compaction mechanism is optimized to generate a refined 3D Gaussian distribution, including:

[0086] Based on the highly visible gradient results and the corrected low-visible gradient results in the hierarchical processing results, gradient integration processing is performed to generate integrated gradient results;

[0087] Based on the integrated gradient results, Gaussians with sizes exceeding the average scale of the scene are split, and their splitting directions are calculated to generate split Gaussians.

[0088] The optimized and stable small-sized Gaussian is cloned and filled to generate a cloned Gaussian.

[0089] Based on the split and cloned Gaussians, the position, covariance, and color attributes of the Gaussians are updated to generate a refined 3D Gaussian distribution.

[0090] Specifically, the highly visible gradient results and the corrected low-visible gradient results from the layered processing are fused at multiple scales. A weighted superposition strategy based on gradient energy proportion is used to generate an integrated gradient result that balances global and local features. Then, based on the vector direction distribution and magnitude statistics of the integrated gradient result, Gaussians with sizes exceeding the average scale of the scene are decomposed into principal directions and their orthogonal splitting directions are calculated. Combined with the density distribution constraints of high-frequency texture regions, a set of split Gaussians distributed along the geometric extension direction is generated. At the same time, for optimized and stable small-sized Gaussians (such as Gaussians with gradient volatility below a set threshold and continuous iteration of spatial position without abrupt changes), density detection and adaptive cloning filling of neighboring sparse regions are performed to generate a set of cloned Gaussians that supplement the detailed representation. Then, based on the spatial distribution topology relationship between the split Gaussians and cloned Gaussians, the position offset of Gaussians, the anisotropy parameters of the covariance matrix, and the view-related color attributes driven by the spherical harmonic coefficients are updated synchronously through gradient backpropagation optimization and physical constraint interpolation. This forms a refined 3D Gaussian distribution with geometric details closely matching the high-frequency textures of the scene and spatial distribution density adapting to rendering requirements.

[0091] Furthermore, based on the integrated gradient results, the Gaussians in the scene are split or cloned to generate new Gaussian-filled regions in the scene that are not yet fully reconstructed, including:

[0092] Based on the integrated gradient results, Gaussians with sizes exceeding the average scale of the scene are split to ensure that excessively large Gaussians in the scene can be correctly split to characterize high-frequency texture regions in the scene.

[0093] Based on the integrated gradient results, Gaussians with sizes smaller than the average scene scale are cloned to ensure that there are a sufficient number of Gaussians to represent the scene.

[0094] Specifically, based on the vector field distribution of the integrated gradient results and the spatial density characteristics of the visibility weights, the gradient principal components of the target Gaussian are decomposed into eigenvalues ​​to extract the principal axis components of its geometric extension direction. Combined with the distribution consistency of the visibility weights in adjacent views, the initial splitting direction is calculated. Then, based on the density distribution map of the high-frequency texture region of the scene and the direction of the geometric edges, the initial splitting direction is locally aligned and oriented. At the same time, the number of splits and the spatial spacing of the sub-Gaussians are dynamically adjusted based on the texture density threshold to generate optimized splitting parameters (including the number of splits, position offset, and direction constraint factor). Then, the original Gaussian is attribute-reassigned according to the splitting parameters: the position is interpolated at equal intervals along the corrected splitting direction to generate sub-Gaussian nodes. The anisotropy parameter of the covariance matrix is ​​adjusted according to the angle between the splitting direction and the texture direction. Based on the spherical harmonic coefficients of the original Gaussian, bidirectional interpolation inheritance of view-related color attributes is performed to ensure that the split sub-Gaussians closely fit the high-frequency details of the scene in terms of geometric structure and appearance attributes, thereby generating a physically reasonable and detail-enhanced set of split Gaussians.

[0095] Furthermore, sparse point cloud reconstruction processing is performed on the input multi-view images to generate an initial 3D Gaussian distribution, including:

[0096] Feature extraction and matching processing are performed on the input multi-view images to generate cross-view matching point pairs;

[0097] Sparse point cloud reconstruction is performed based on matching point pairs to generate an initial sparse point cloud.

[0098] Covariance estimation is performed on each point in the initial sparse point cloud to generate the covariance matrix of each point.

[0099] An initial 3D Gaussian distribution is constructed based on the covariance matrix and preset Gaussian attribute parameters.

[0100] Specifically, a cross-view matching algorithm based on deep feature descriptors (such as SIFT or deep learning features) is employed to extract feature points and perform robust matching on the input multi-view images. Iterative filtering of mismatched point pairs using epipolar geometric constraints and the RANSAC algorithm generates a high-confidence set of cross-view matching point pairs. Subsequently, based on the Structure from Motion (SFM) framework, the three-dimensional spatial coordinates of the matching point pairs are calculated using the multi-view geometric triangulation principle, combined with bundle adjustment. The adjustment process optimizes the camera pose and point cloud position to generate an initial sparse point cloud with optimized geometric consistency. Then, for each 3D point in the sparse point cloud, the covariance matrix representing the spatial anisotropy is estimated based on the KNN nearest neighbor algorithm of its neighborhood point distribution. The uncertainty quantification parameters in the reconstruction process are then fused to construct an error-aware correction model for the covariance matrix. After that, the covariance matrix is ​​combined with preset Gaussian attribute parameters (including initial opacity and view-related color basis functions based on spherical harmonic coefficients) to generate an initial 3D Gaussian distribution covering the key geometric structures of the scene, providing physically reasonable initialization conditions for subsequent gradient optimization and densification processing.

[0101] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Except for step S101, which is executed only once during the initialization phase, the other steps are executed multiple times in a loop. Moreover, at least some of the steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily completed at the same time, but may be executed at different times.

[0102] In one embodiment, such as Figure 2 As shown, this application also provides a 3DGS new view rendering quality improvement system 200 based on Gaussian visibility, the system 200 comprising:

[0103] The sparse point cloud reconstruction module 201 is used to perform sparse point cloud reconstruction processing on the input multi-view image to generate an initial 3D Gaussian distribution.

[0104] The visibility filtering module 202 is used to perform visibility judgment processing on each Gaussian based on the projection area of ​​the existing 3D Gaussian distribution and the occlusion relationship between Gaussians in the current iteration, and to filter out Gaussians that are not visible relative to the view, so as to obtain the visibility judgment processing result.

[0105] The weight calculation module 203 is used to perform integral calculation on the coverage and contribution intensity of each Gaussian in the view based on the visibility judgment processing result, and generate Gaussian visibility weights.

[0106] The hierarchical gradient module 204 is used to perform hierarchical processing on Gaussian based on Gaussian visibility weights, dynamically adjust the gradient accumulation strategy, and obtain the hierarchical processing result.

[0107] The densification optimization module 205 is used to optimize the Gaussian densification mechanism by combining the results of the hierarchical processing, and generate a refined 3D Gaussian distribution.

[0108] The rendering output module 206 is used to perform new view rendering processing based on a refined 3D Gaussian distribution and output an image without artifacts.

[0109] Specifically, the sparse point cloud reconstruction module 201 initializes the scene, generates an initial sparse point cloud using multi-view geometric triangulation and bundle adjustment, estimates the covariance matrix of each point based on the KNN nearest neighbor algorithm, and constructs an initial Gaussian distribution by combining preset opacity and spherical harmonic color basis functions; the visibility filtering module 202 filters invisible Gaussian points based on depth sorting sequence and projection region judgment, using depth occlusion detection and pixel contribution value dual verification; the weight calculation module 203 generates a weight distribution that quantifies the visibility strength based on the proportion of Gaussian pixels covered in the view and the integral of the contribution value; and the hierarchical gradient... Module 204 divides the high / low visibility Gaussian set by dynamic thresholding, applies gradient correction based on weight decay coefficient to the low visibility set, and integrates noise filtering and spatial continuity constraints to generate hierarchical gradient results; the densification optimization module 205 integrates gradient data, drives large Gaussian splitting based on texture density map and gradient principal direction decomposition, supplements detailed regions with clone filling mechanism, and simultaneously optimizes position, covariance and color attributes through backpropagation; the rendering output module 206 uses a depth-sorted alpha blending rendering pipeline, combined with anisotropic Gaussian projection and anti-aliasing filtering to generate an artifact-free image.

[0110] The hierarchical gradient module 204 is also used for:

[0111] Based on the visibility weight of Gaussians, the Gaussians are classified to generate a set of highly visible Gaussians and a set of low-visibility Gaussians.

[0112] For Gaussians in the set of highly visible Gaussians, process them according to the original gradient accumulation strategy to generate highly visible gradient results;

[0113] For Gaussians in the set of low-visibility Gaussians, the dynamic decay coefficient is used to process them and generate corrected low-visibility gradient results.

[0114] Integrate highly visible gradient results and corrected low-visible gradient results to generate hierarchical processing results adjusted by the gradient accumulation strategy.

[0115] The hierarchical gradient module 204 is also used for:

[0116] Based on the Gaussian properties in the highly visible Gaussian set, the original gradient accumulation parameters are extracted;

[0117] Based on the original gradient accumulation parameters, gradient accumulation is performed on the Gaussians in the set of highly visible Gaussians to generate uncorrected gradient results.

[0118] The uncorrected gradient results were subjected to gradient filtering to remove noise interference, resulting in filtered gradient results.

[0119] Based on the spatial distribution relationship between the filtered gradient results and the highly visible Gaussian set, highly visible gradient results are generated.

[0120] The hierarchical gradient module 204 is also used for:

[0121] A dynamic attenuation coefficient is generated based on the dynamic visibility weight distribution of the low-degree visibility Gaussian set.

[0122] For Gaussians in the low-visibility Gaussian set, gradient weighting is performed based on the dynamic decay coefficient to generate a preliminary corrected gradient;

[0123] The initial corrected gradient is subjected to gradient stability verification processing to remove abnormal gradient interference and obtain the verified gradient.

[0124] Based on the relationship between the validated gradient and the spatial coverage of the low-visibility Gaussian set, a corrected low-visibility gradient result is generated.

[0125] The densification optimization module 205 is also used for:

[0126] A dynamic attenuation coefficient is generated based on the dynamic visibility weight distribution of the low-degree visibility Gaussian set.

[0127] For Gaussians in the low-visibility Gaussian set, gradient weighting is performed based on the dynamic decay coefficient to generate a preliminary corrected gradient;

[0128] The initial corrected gradient is subjected to gradient stability verification processing to remove abnormal gradient interference and obtain the verified gradient.

[0129] Based on the relationship between the validated gradient and the spatial coverage of the low-visibility Gaussian set, a corrected low-visibility gradient result is generated.

[0130] The densification optimization module 205 is also used for:

[0131] Based on the integrated gradient results, Gaussians with sizes exceeding the average scale of the scene are split to generate split Gaussians.

[0132] The optimized and stable small-sized Gaussian is cloned and filled to generate a cloned Gaussian.

[0133] Based on the split and cloned Gaussians, the position, covariance, and color attributes of the Gaussians are updated to generate a refined 3D Gaussian distribution.

[0134] The sparse point cloud reconstruction module 201 is also used for:

[0135] Feature extraction and matching processing are performed on the input multi-view images to generate cross-view matching point pairs;

[0136] Sparse point cloud reconstruction is performed based on matching point pairs to generate an initial sparse point cloud.

[0137] Covariance estimation is performed on each point in the initial sparse point cloud to generate the covariance matrix of each point.

[0138] An initial 3D Gaussian distribution is constructed based on the covariance matrix and preset Gaussian attribute parameters.

[0139] In one embodiment, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0140] In one embodiment, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described method embodiments.

[0141] In one embodiment, based on the original visibility judgment conditions of 3DGS, a strict constraint based on contribution value is introduced. When the contribution value of Gaussian to any pixel in the current view exceeds a threshold, it is determined to be visible; otherwise, it is considered invisible. By eliminating the invalid gradient accumulation of occluded Gaussian, the gradient level is avoided from being diluted, ensuring that the gradient calculation of large-size Gaussian is based only on the effective visible view, thus improving the splitting capability.

[0142] In one embodiment, the visibility differences of Gaussians are quantified based on the coverage area of ​​the projected region of the 2D Gaussians in the view and the intensity of their central contribution. A dynamic partitioning of "highly visible" and "lowly visible" Gaussian sets is performed based on visibility weights; the former uses the original gradient accumulation strategy, while the latter employs gradient correction based on weight decay. This differentiated gradient accumulation strategy reduces noise interference from low-visibility Gaussians while enhancing the effective gradient signal in high-frequency regions, optimizing the Gaussian splitting direction and density distribution. The formulas used are as follows:

[0143]

[0144]

[0145] in, Denotes the center of a 2D Gaussian. R represents the radius of the 2D Gaussian rectangle bounding box, G 2D Represent a 2D Gaussian. It is any pixel point on the two-dimensional image plane, ω represents the visibility weight, and τ weight Let represent a given threshold, N represent the depth sorting sequence, x represent the x-coordinate of pixel p, y represent the y-coordinate of pixel p, i and j are index variables, λ represent a user-defined parameter used to prevent large differences between the newly calculated gradient level and the original 3DGS gradient level, W is the camera extrinsic parameter matrix of the current viewpoint, and T... L This indicates that the Gaussian is considered as a "highly visible" iterative set, T H This indicates that the Gaussian is considered as an iterative set with "low visibility," g i This represents the position gradient value obtained by the Gaussian sphere in each iteration. This represents the gradient level of the Gaussian sphere.

[0146] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0147] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for improving the rendering quality of a new view of 3DGS based on Gaussian visibility, characterized in that, The method includes: Sparse point cloud reconstruction processing is performed on the input multi-view images to generate an initial 3D Gaussian distribution; Based on the existing 3D Gaussian distribution in the projection area of ​​the 2D view and the occlusion relationship between Gaussians in the current iteration, a visibility judgment process is performed on each Gaussian, and invisible Gaussians are filtered out to obtain the visibility judgment result. Based on the visibility judgment processing result, the coverage and contribution intensity of each visible Gaussian in the view are integrally calculated to generate Gaussian visibility weights. Based on the Gaussian visibility weights, the Gaussian layers are processed in layers, and the gradient accumulation strategy is dynamically adjusted to obtain the layered processing results. By integrating the results of the hierarchical processing, the Gaussian densification mechanism is optimized to generate a refined 3D Gaussian distribution; Based on the refined 3D Gaussian distribution, a new view rendering process is performed to output an image without artifacts. Specifically, based on the Gaussian visibility weights, Gaussians are subjected to hierarchical processing, and the gradient accumulation strategy is dynamically adjusted to obtain the hierarchical processing results, including: Based on the Gaussian visibility weights, the Gaussians are layered to generate a set of highly visible Gaussians and a set of low-visibility Gaussians. For the Gaussians in the set of highly visible Gaussians, process them according to the original gradient accumulation strategy to generate highly visible gradient results; The Gaussians in the set of low-visibility Gaussians are processed based on the dynamic decay coefficient to generate a corrected low-visibility gradient result. Integrate the highly visible gradient results and the corrected low-visibility gradient results, and adjust and optimize the gradient accumulation strategy; Based on the integrated gradient results, the Gaussians in the scene are split or cloned to generate new Gaussian-filled regions in the scene that are not yet fully reconstructed, including: Based on the integrated gradient results, Gaussians with sizes exceeding the average scale of the scene are split to ensure that excessively large Gaussians in the scene can be correctly split to characterize high-frequency texture regions in the scene. Based on the integrated gradient results, Gaussians with sizes smaller than the average scene scale are cloned to ensure that there are a sufficient number of Gaussians to represent the scene.

2. The method of claim 1, wherein the method is a method for improving the rendering quality of a new view of a 3D Gaussian visibility-based GS. The process of processing the Gaussians in the highly visible Gaussian set according to the original gradient accumulation strategy to generate highly visible gradient results includes: Extract the position gradient of the Gaussian in the set of highly visible Gaussians obtained by backpropagation of stochastic gradient descent; The position gradient is subjected to gradient filtering to remove noise interference, resulting in a filtered position gradient. Based on the filtered position gradient, gradient accumulation processing is performed on the Gaussians in the set of highly visible Gaussians to generate highly visible gradient results.

3. The method of claim 1, wherein the method further comprises: The process of processing the Gaussians in the set of low-visibility Gaussians based on a dynamic decay coefficient to generate a corrected low-visibility gradient result includes: Based on the Gaussian visibility weights, a dynamic decay coefficient is generated; The dynamic attenuation coefficient is judged to be reasonable, and outlier interference is eliminated to obtain the denoised dynamic attenuation coefficient. For the Gaussians in the set of low-visibility Gaussians, gradient weighting is performed according to the dynamic decay coefficient to generate a corrected low-visibility gradient result.

4. The method of claim 1, wherein the method further comprises: The process of integrating the hierarchical processing results and optimizing the Gaussian compaction mechanism to generate a refined 3D Gaussian distribution includes: The highly visible gradient results and the corrected low-visibility gradient results in the layered processing results are subjected to gradient integration processing to generate integrated gradient results; Based on the integrated gradient results, the Gaussians in the scene are split or cloned to generate new Gaussians to fill the areas in the scene that are not yet fully reconstructed. Based on the new Gaussian generated by the split or clone, the position, covariance, and color attributes of the Gaussian are updated to generate the refined 3D Gaussian distribution.

5. The method of claim 1, wherein the method further comprises: The step of performing sparse point cloud reconstruction processing on the input multi-view images to generate an initial 3D Gaussian distribution includes: Feature extraction and matching processing are performed on the input multi-view images to generate cross-view matching point pairs; Based on the matching point pairs, sparse point cloud reconstruction processing is performed to generate an initial sparse point cloud. Covariance estimation is performed on each point in the initial sparse point cloud to generate the covariance matrix of each point. The initial 3D Gaussian distribution is constructed based on the covariance matrix and the preset Gaussian attribute parameters.

6. A 3DGS new view rendering quality improvement system based on Gaussian visibility, characterized in that, The system includes: The sparse point cloud reconstruction module is used to perform sparse point cloud reconstruction processing on the input multi-view images to generate an initial 3D Gaussian distribution. The visibility filtering module is used to perform visibility judgment processing on each Gaussian based on the projection area of ​​the existing 3D Gaussian distribution and the occlusion relationship between Gaussians in the current iteration, and to filter out Gaussians that are not visible relative to the current view, so as to obtain the visibility judgment processing result. The weight calculation module is used to perform integral calculation on the coverage and contribution intensity of each visible Gaussian in the view based on the visibility judgment processing result, and generate Gaussian visibility weights. The hierarchical gradient module is used to perform hierarchical processing on the Gaussian based on the Gaussian visibility weights, dynamically adjust the gradient accumulation strategy, and obtain the hierarchical processing result; wherein, performing hierarchical processing on the Gaussian based on the Gaussian visibility weights and dynamically adjusting the gradient accumulation strategy to obtain the hierarchical processing result includes: Based on the Gaussian visibility weights, the Gaussians are layered to generate a set of highly visible Gaussians and a set of low-visibility Gaussians. For the Gaussians in the set of highly visible Gaussians, process them according to the original gradient accumulation strategy to generate highly visible gradient results; The Gaussians in the set of low-visibility Gaussians are processed based on the dynamic decay coefficient to generate a corrected low-visibility gradient result. Integrate the highly visible gradient results and the corrected low-visibility gradient results, and adjust and optimize the gradient accumulation strategy; The densification optimization module is used to integrate the results of the layered processing, optimize the Gaussian densification mechanism, and generate a refined 3D Gaussian distribution. Based on the integrated gradient results, the Gaussians in the scene are split or cloned to generate new Gaussian-filled areas in the scene that are not yet sufficiently reconstructed, including: Based on the integrated gradient results, Gaussians with sizes exceeding the average scale of the scene are split to ensure that excessively large Gaussians in the scene can be correctly split to characterize high-frequency texture regions in the scene. Based on the integrated gradient results, Gaussians with sizes smaller than the average scene scale are cloned to ensure that there are a sufficient number of Gaussians to represent the scene. The rendering output module is used to perform new view rendering processing based on the refined 3D Gaussian distribution and output an image without artifacts. 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. When the processor executes the computer program, it implements the steps of the method for improving the rendering quality of a new 3DGS view based on Gaussian visibility as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for improving the rendering quality of a new 3DGS view based on Gaussian visibility as described in any one of claims 1 to 5.

Images