3DGS new view rendering quality improvement method based on Gaussian visibility

Abstract

The invention relates to the technical field of three-dimensional reconstruction. The 3D GS new view rendering quality improving method based on Gaussian visibility comprises the steps that sparse point cloud reconstruction processing is carried out on an input multi-view image, and initial 3D Gaussian distribution is generated; carrying out visibility judgment processing on each Gaussian, filtering the Gaussian which is invisible relative to the visual angle, and obtaining a visibility judgment processing result; according to a visibility judgment processing result, integral calculation processing is carried out on the coverage range and the contribution intensity of each Gaussian in the view, and a Gaussian visibility weight is generated; according to the Gaussian visibility weight, performing hierarchical processing on Gaussian, and dynamically adjusting a gradient accumulation strategy to obtain a hierarchical processing result; layering processing results are integrated, a Gaussian densification mechanism is optimized, and refined 3D Gaussian distribution is generated; and carrying out new view rendering processing, and outputting an image without artifacts, so as to solve the problems that the occlusion relationship is not fully considered, the large-size Gaussian splitting capability is insufficient, and scene representation is mismatched with high-frequency textures.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a method for improving 3DGS new view rendering quality based on Gaussian visibility. Background Art

[0002] With the rapid development of technologies such as virtual reality (VR), augmented reality (AR), and autonomous driving, Novel View Synthesis (NVS) has become a core research direction at the intersection of computer vision and graphics. Its goal is to generate high-quality scene renderings from input images with a limited viewpoint, providing key technical support for applications such as immersive interaction, digital twins, and film and television special effects.

[0003] However, the typical new view synthesis technology 3D Gaussian splattering (3DGS) does not fully consider the occlusion relationship between Gaussians in its visibility judgment mechanism, and the gradient accumulation strategy adopts a "bisect" mechanism, which leads to insufficient average gradient level of large-scale Gaussians, inhibiting their splitting ability and exacerbating the mismatch between scene representation and high-frequency textures. Summary of the Invention

[0004] Based on this, it is necessary to provide a 3DGS new view rendering quality improvement method based on Gaussian visibility to address the above technical problems, so as to solve the problems of insufficient consideration of occlusion relationships, insufficient large-scale Gaussian splitting capabilities, and mismatch between scene representation and high-frequency textures, thereby reducing needle artifacts and blurred details in the rendering results, and improving the accuracy and visual effects of new view synthesis.

[0005] In a first aspect, the present application provides a method for improving 3DGS new view rendering quality based on Gaussian visibility, the method comprising:

[0006] Perform sparse point cloud reconstruction on the input multi-view images to generate an initial 3D Gaussian distribution;

[0007] Based on the projection area of the existing 3D Gaussian distribution in the current iteration and the occlusion relationship between the Gaussians, visibility judgment processing is performed on each Gaussian, and Gaussians that are invisible to the view are filtered out to obtain the visibility judgment processing result;

[0008] Based on the visibility judgment processing results, the coverage and contribution strength of each visible Gaussian in the view are integrated and calculated to generate the Gaussian visibility weight;

[0009] Based on the Gaussian visibility weight, the Gaussian is layered and the gradient accumulation strategy is dynamically adjusted to obtain the layered processing results.

[0010] Integrate the layered processing results, optimize the Gaussian densification mechanism, and generate a refined 3D Gaussian distribution;

[0011] Based on the refined 3D Gaussian distribution, new view rendering is performed to output artifact-free images.

[0012] Furthermore, based on the Gaussian visibility weight, the Gaussian is layered and the gradient accumulation strategy is dynamically adjusted to obtain the layered processing results, including:

[0013] Based on the Gaussian visibility weights, the Gaussians are layered to generate a highly visible Gaussian set and a low visible Gaussian set;

[0014] Processing the Gaussians in the highly visible Gaussian set according to the original gradient accumulation strategy to generate a highly visible gradient result;

[0015] Processing the Gaussians in the low-visibility Gaussian set based on a dynamic attenuation coefficient to generate a modified low-visibility gradient result;

[0016] The highly visible gradient result and the corrected low visible gradient result are integrated to adjust and optimize the gradient accumulation strategy.

[0017] Furthermore, the Gaussians in the highly visible Gaussian set are processed according to the original gradient accumulation strategy to generate highly visible gradient results, including:

[0018] Extract the position gradients of Gaussians from the set of highly visible Gaussians obtained during the back-propagation of stochastic gradient descent;

[0019] Perform gradient screening on the position gradient to filter out noise interference and obtain the screened position gradient;

[0020] According to the filtered position gradient, the Gaussians in the highly visible Gaussian set are subjected to gradient accumulation processing to generate a highly visible gradient result.

[0021] Furthermore, the Gaussians in the low-visibility Gaussian set are processed based on the dynamic attenuation coefficient to generate a corrected low-visibility gradient result, including:

[0022] Generate dynamic attenuation coefficient based on dynamic Gaussian visibility weight;

[0023] Make a rational judgment on the dynamic attenuation coefficient, eliminate the interference of abnormal values, and obtain the denoised dynamic attenuation coefficient;

[0024] The Gaussians in the low-visibility Gaussian set are subjected to gradient weighting processing based on the dynamic attenuation coefficient to generate a modified low-visibility gradient result.

[0025] Based on the relationship between the verified gradient and the spatial coverage of the low-visibility Gaussian set, a corrected low-visibility gradient result is generated.

[0026] Furthermore, the hierarchical processing results are integrated to optimize the Gaussian densification mechanism and generate a refined 3D Gaussian distribution, including:

[0027] Performing gradient integration processing on the highly visible gradient results and the corrected low visible gradient results in the layered processing results to generate an integrated gradient result;

[0028] Based on the integrated gradient results, the Gaussians in the scene are split or cloned to generate new Gaussians to fill the areas in the scene that are not yet fully reconstructed;

[0029] Based on the new Gaussian generated by splitting or cloning, the position, covariance and color attributes of the Gaussian are updated to generate the refined 3D Gaussian distribution.

[0030] Furthermore, based on the integrated gradient results, the Gaussians in the scene are split or cloned to generate new Gaussians to fill the areas in the scene that are not yet fully reconstructed, including:

[0031] Based on the integrated gradient results, the Gaussian whose size exceeds the average scale of the scene is split to ensure that the excessively large Gaussian in the scene can be correctly split to represent the high-frequency texture area in the scene;

[0032] According to the integrated gradient results, the Gaussians whose sizes are smaller than the average scale of the scene are cloned to ensure that there are a sufficient number of Gaussians to represent the scene.

[0033] Furthermore, the input multi-view images are subjected to sparse point cloud reconstruction to generate an initial 3D Gaussian distribution, including:

[0034] Perform feature extraction and matching processing on the input multi-view images to generate cross-view matching point pairs;

[0035] Perform sparse point cloud reconstruction based on matching point pairs to generate an initial sparse point cloud;

[0036] Perform covariance estimation on each point in the initial sparse point cloud to generate the covariance matrix of each point;

[0037] Construct an initial 3D Gaussian distribution based on the covariance matrix and preset Gaussian attribute parameters.

[0038] In a second aspect, the present application further provides a 3DGS new view rendering quality improvement system based on Gaussian visibility, the system comprising:

[0039] The sparse point cloud reconstruction module is used to perform sparse point cloud reconstruction on the input multi-view images and generate the initial 3D Gaussian distribution;

[0040] The visibility filtering module is used to perform visibility judgment processing on each Gaussian based on the projection area of the existing Gaussian distribution in the current iteration and the occlusion relationship between the Gaussians, filter out the Gaussians that are invisible to the view, and obtain the visibility judgment processing result;

[0041] A weight calculation module is used to perform integral calculation on the coverage and contribution strength of each visible Gaussian in the view based on the visibility judgment processing result to generate a Gaussian visibility weight;

[0042] The layered gradient module is used to perform layered processing on Gaussians based on Gaussian visibility weights, dynamically adjust the gradient accumulation strategy, and obtain layered processing results;

[0043] Densification optimization module, which is used to integrate the layered processing results, optimize the Gaussian densification mechanism, and generate a refined 3D Gaussian distribution;

[0044] The rendering output module is used to perform new view rendering processing based on the refined 3D Gaussian distribution and output an artifact-free image.

[0045] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of any method in the first aspect of the present application are implemented.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any method in the first aspect of the present application when the computer program is executed by a processor.

[0047] The technical solution provided by this application includes the following technical effects: a method for improving the rendering quality of a 3DGS new view based on Gaussian visibility is provided, comprising: performing sparse point cloud reconstruction processing on an input multi-view image to generate an initial 3D Gaussian distribution; performing visibility judgment processing on each Gaussian based on the projection area of the existing Gaussian distribution in the current iteration and the occlusion relationship between the Gaussians, filtering out Gaussians that are invisible relative to the view, and obtaining a visibility judgment processing result; based on the visibility judgment processing result, integrating the coverage range and contribution intensity of each visible Gaussian in the view to generate a Gaussian visibility weight; based on the Gaussian visibility weight, performing layered processing on the Gaussians, dynamically adjusting the gradient accumulation strategy, and obtaining a layered processing result; combining the layered processing result with the Gaussian densification mechanism to generate a refined 3D Gaussian distribution; performing new view rendering processing based on the refined 3D Gaussian distribution and outputting an artifact-free image, thereby addressing the problems of insufficient consideration of occlusion relationships, insufficient splitting capability of large-scale Gaussians, and mismatch between scene representation and high-frequency textures, thereby reducing needle artifacts and blurred details in the rendering results and improving the accuracy and visual effect of new view synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 Flowchart of a method for improving 3DGS new view rendering quality based on Gaussian visibility in one embodiment of the present invention;

[0050] Figure 2 This is a structural diagram of a 3DGS new view rendering quality improvement system based on Gaussian visibility in one embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the above-mentioned purposes, features and advantages of the present application more clearly understood, the specific implementation methods of the present application are described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to fully understand the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0052] The 3DGS mentioned in this application represents 3D Gaussian Splatting (3DGS).

[0053] like Figure 1 As shown, the present application provides a method for improving 3DGS new view rendering quality based on Gaussian visibility, the method comprising:

[0054] S101: Perform sparse point cloud reconstruction on the input multi-view images to generate an initial 3D Gaussian distribution.

[0055] Specifically, a multi-view feature matching algorithm based on deep feature descriptors (such as SIFT or SuperPoint) is adopted to generate a high-confidence sparse matching point set through cross-view feature point extraction and robust matching (such as the RANSAC algorithm iteratively screening out mismatched point pairs). Then, based on the Structure from Motion (SFM) framework, the multi-view geometric triangulation principle is used to calculate the 3D spatial coordinates of the matching points, and the camera pose parameters and the sparse point cloud geometric positions are jointly optimized through bundle adjustment to construct an initial sparse point cloud with geometric consistency optimization. Then, for each sparse point, the local geometric distribution of its neighborhood points and the KNN nearest neighbor algorithm are combined to fuse the reconstruction confidence and generate a covariance matrix representing the Gaussian spatial distribution. Then, the covariance matrix is combined with preset Gaussian attribute parameters (including the view-dependent color basis based on spherical harmonics, initial opacity and illumination response coefficient) to construct a 3D Gaussian distribution covering the key geometric structures of the scene, providing physically reasonable initialization conditions for subsequent gradient optimization and densification processing.

[0056] S102: Based on the projection area of the existing 3D Gaussian distribution in the current iteration and the occlusion relationship between the Gaussians, visibility judgment processing is performed on each Gaussian, and Gaussians that are invisible relative to the current view are filtered to obtain a visibility judgment processing result.

[0057] Specifically, each 3D Gaussian distribution is projected onto the 2D image plane of the target view to form a 2D Gaussian distribution. This step is achieved by transforming the projection from the world coordinate system to the pixel coordinate system, ensuring that the Gaussian's projected area is accurately mapped to the image plane. Based on the Gaussian's projected area and depth information, the visibility of the Gaussian is determined. This is done by checking whether the Gaussian's projection in the current view falls within the view boundaries and, combined with depth sorting, determining whether the Gaussian is occluded by other Gaussians in front of it. The Gaussians are sorted using depth information to ensure that Gaussians at the end of the depth sequence that are occluded by Gaussians in front of them are considered invisible during rendering. This step is achieved through visibility ordering, ensuring that only visible Gaussians are considered during rendering. Through this determination and sorting process, Gaussians that are invisible relative to the view are filtered out, generating a visibility determination result. Invisible Gaussians are excluded from subsequent rendering and optimization processes, thereby improving rendering efficiency and quality. Through these steps, invisible Gaussians are effectively filtered out, ensuring that subsequent rendering and optimization processes are performed only based on visible Gaussians, thereby improving the quality and efficiency of rendering the new view.

[0058] S103: Based on the visibility judgment processing result, perform integral calculation on the coverage and contribution strength of each Gaussian in the view to generate a Gaussian visibility weight.

[0059] Specifically, based on the projection area of the Gaussian, its coverage area in the view is calculated. This step determines the effective coverage area of the Gaussian in the view by analyzing the intersection of the Gaussian's projection area and the view area. Combining the Gaussian's coverage area and the 2D Gaussian distribution of its projection, calculate its contribution intensity to the view, that is, the visibility weight. This step is achieved by analyzing the opacity changes of the Gaussian in the coverage area, comprehensively considering the size of its projection area and the contribution intensity, and through the mathematical method of double integral to ensure that the weight can more accurately reflect the visibility and contribution of the Gaussian in the view. Through the above steps, the Gaussian visibility weight can be generated to provide a basis for subsequent rendering and optimization.

[0060] S104: Based on the Gaussian visibility weight, perform layered processing on the Gaussian, dynamically adjust the gradient accumulation strategy, and obtain a layered processing result.

[0061] Specifically, based on the Gaussian visibility weight, Gaussians are divided into two categories: "highly visible" and "lowly visible". Highly visible Gaussians use the original gradient accumulation strategy, while low-visibility Gaussians introduce a dynamic attenuation coefficient for gradient weighting. For highly visible Gaussians, the original gradient accumulation strategy is used; for low-visibility Gaussians, the gradient accumulation is adjusted by the dynamic attenuation coefficient to balance the gradient contributions of Gaussians with different visibilities. Combining the gradient results of highly visible and low-visibility Gaussians, a gradient accumulation strategy is generated after layered processing, providing a basis for subsequent Gaussian densification optimization. Through the above steps, the gradient accumulation strategy can be dynamically adjusted to optimize the Gaussian densification process, thereby improving the quality and efficiency of new view rendering.

[0062] S105: Integrate the layered processing results, optimize the Gaussian densification mechanism, and generate a refined 3D Gaussian distribution.

[0063] Specifically, based on the highly visible gradient results and the corrected low-visibility gradient results in the layered processing results, gradient integration processing is performed to generate an integrated gradient result. This step ensures the integrity and rationality of the gradient information by comprehensively considering the gradient contributions of Gaussians with different visibility. According to the integrated gradient results, Gaussians whose sizes exceed the average scale of the scene are split, and Gaussians whose sizes are smaller than the average scale of the scene are cloned. More new Gaussians are generated by splitting and cloning to fill in areas in the scene that are not yet fully reconstructed. This step ensures that oversized Gaussians can be correctly split, and the optimized and stable small-sized Gaussians remain basically unchanged, ensuring that the split Gaussians can better match the detailed texture of the scene. Based on the split Gaussians and the cloned Gaussians, the position, covariance and color attributes of the Gaussians are updated to generate a refined 3D Gaussian distribution. This step ensures that the parameters of the Gaussians are optimized to more accurately represent the scene. Through the above steps, the Gaussian densification mechanism can be optimized to generate a refined 3D Gaussian distribution, thereby improving the quality and efficiency of new view rendering.

[0064] S106: Based on the refined 3D Gaussian distribution, a new view rendering process is performed to output an artifact-free image.

[0065] Specifically, the parameters of the refined 3D Gaussian distribution (such as position, covariance matrix, color, and transparency) are optimized to ensure that it accurately represents the details and high-frequency textures in the scene. The optimized 3D Gaussian distribution is projected onto the 2D image plane of the target view to form a 2D Gaussian distribution. This step is achieved by transforming the projection from the world coordinate system to the pixel coordinate system, ensuring that the projected area of the Gaussian is accurately mapped to the image plane. Depth information is used to sort the Gaussians, ensuring that Gaussians at the end of the depth sequence are considered invisible during rendering if they are occluded by Gaussians in front. This step is achieved through visibility ordering, ensuring that only visible Gaussians are considered during rendering. Alpha blending is performed based on the depth-sorted Gaussian distribution. The final image is formed by superimposing multiple semi-transparent Gaussian distributions. This process is efficient and maintains high quality in real-time rendering. By analyzing the visibility weights and coverage of the Gaussians, potential pinhole artifacts and blurred details are identified and eliminated. This step is achieved by filtering outliers and optimizing the Gaussian splitting and cloning strategy. Post-processing is performed on the rendered image, such as removing background noise or unwanted areas to further optimize the image's appearance. Through these steps, a new view can be rendered based on the refined 3D Gaussian distribution, outputting artifact-free, high-quality images. This improves the accuracy and visual quality of new view synthesis.

[0066] An embodiment of the present application provides a method for improving the rendering quality of a 3DGS new view based on Gaussian visibility, comprising: performing sparse point cloud reconstruction on an input multi-view image to generate an initial 3D Gaussian distribution; performing visibility judgment on each Gaussian based on the projection area of the existing 3D Gaussian distribution in the current iteration and the occlusion relationship between the Gaussians, filtering out Gaussians that are invisible relative to the view, and obtaining a visibility judgment result; calculating the coverage and contribution strength of each Gaussian in the view based on the visibility judgment result to generate a Gaussian visibility weight; performing layered processing on the Gaussians based on the Gaussian visibility weight, dynamically adjusting the gradient accumulation strategy, and obtaining a layered processing result; integrating the layered processing results, optimizing the Gaussian densification mechanism, and generating a refined 3D Gaussian distribution; performing new view rendering based on the refined 3D Gaussian distribution to output an artifact-free image, thereby addressing issues such as insufficient consideration of occlusion relationships, insufficient splitting capability of large-scale Gaussians, and mismatch between scene representation and high-frequency textures. This reduces pinhole artifacts and blurred details in the rendering result, thereby improving the accuracy and visual quality of new view synthesis.

[0067] Furthermore, based on the Gaussian visibility weight, the Gaussian is layered and the gradient accumulation strategy is dynamically adjusted to obtain the layered processing results, including:

[0068] Based on the Gaussian visibility weight, the Gaussians are classified to generate a set of highly visible Gaussians and a set of low-visibility Gaussians;

[0069] The Gaussians in the highly visible Gaussian set are processed according to the original gradient accumulation strategy to generate highly visible gradient results;

[0070] The Gaussians in the low-visibility Gaussian set are processed based on the dynamic attenuation coefficient to generate a corrected low-visibility gradient result;

[0071] The highly visible gradient results and the corrected low visible gradient results are integrated to generate the hierarchical processing results after the gradient accumulation strategy is adjusted.

[0072] Specifically, a Gaussian visibility weight is generated based on the coverage and contribution strength of the Gaussian in the view. This step is achieved by analyzing the visibility judgment results and the projection area of the Gaussian. Based on the Gaussian visibility weight, the Gaussian is divided into two categories: "highly visible" and "lowly visible". Highly visible Gaussians use the original gradient accumulation strategy, while low-visibility Gaussians introduce a dynamic attenuation coefficient for gradient weighting. For highly visible Gaussians, the original gradient accumulation strategy is used to generate a highly visible gradient result. This step ensures that the gradient accumulation of the Gaussian can accurately reflect its contribution to the view. For low-visibility Gaussians, the dynamic attenuation coefficient is used to process and generate a corrected low-visibility gradient result. This step balances the gradient contribution of Gaussians with different visibilities by adjusting the gradient accumulation. Combining the highly visible gradient result and the corrected low-visibility gradient result, a layered processing result after the gradient accumulation strategy is adjusted is generated. This step ensures that the gradient accumulation strategy can dynamically adapt to the visibility changes of the Gaussian, thereby optimizing the Gaussian densification process.

[0073] Furthermore, the Gaussians in the highly visible Gaussian set are processed according to the original gradient accumulation strategy to generate highly visible gradient results, including:

[0074] Extract the original gradient accumulation parameters based on the Gaussian properties in the highly visible Gaussian set;

[0075] According to the original gradient accumulation parameters, the Gaussians in the highly visible Gaussian set are subjected to gradient accumulation processing to generate an uncorrected gradient result;

[0076] Perform gradient screening on the uncorrected gradient result to filter out noise interference and obtain the screened gradient result;

[0077] Based on the spatial distribution relationship between the filtered gradient results and the highly visible Gaussian set, a highly visible gradient result is generated.

[0078] Specifically, based on the spatial distribution properties of highly visible Gaussians (including position gradient, covariance matrix gradient and color gradient), their original gradient accumulation parameters (such as gradient modulus mean and directional consistency coefficient) in different training iteration cycles are extracted; then, according to the 3DGS standard gradient accumulation strategy, the gradients contributed by multiple views in the current iteration cycle are weighted summed to generate an uncorrected initial gradient result; then, a joint screening mechanism based on statistical distribution threshold and gradient direction consistency verification is adopted to eliminate abnormal gradient components caused by partial view occlusion or insufficient sampling, and retain valid gradients that conform to the geometric evolution trend of the scene; then, combined with the spatial density distribution map of highly visible Gaussians, the screened gradients are subjected to local energy normalization and directional smoothing interpolation to ensure that the gradient-adjusted Gaussians maintain spatial continuity in the splitting direction and position offset, thereby generating highly visible gradient results that are adapted to the high-frequency texture of the scene, providing a stable driving signal for subsequent densification splitting.

[0079] Furthermore, the Gaussians in the low-visibility Gaussian set are processed based on the dynamic attenuation coefficient to generate a corrected low-visibility gradient result, including:

[0080] Generate dynamic attenuation coefficient based on dynamic visibility weight distribution of low-visibility Gaussian set;

[0081] For the Gaussians in the low-visibility Gaussian set, gradient weighting is performed according to the dynamic attenuation coefficient to generate a preliminary correction gradient;

[0082] Perform gradient stability verification on the preliminary corrected gradient to remove abnormal gradient interference and obtain the verified gradient;

[0083] Based on the relationship between the verified gradient and the spatial coverage of the low-visibility Gaussian set, a corrected low-visibility gradient result is generated.

[0084] Specifically, according to the visibility weight distribution dynamically updated during the training process of the low-visibility Gaussian, the interval fluctuation characteristics of its weight value are statistically analyzed to generate a dynamic attenuation coefficient adaptive to the current scene optimization stage; then, the dynamic attenuation coefficient is weightedly fused with the original gradient of the low-visibility Gaussian element by element to generate a preliminary corrected gradient; then, the preliminary corrected gradient is subjected to stability verification based on the statistical distribution threshold and gradient direction consistency, and abnormal gradients that deviate from the group distribution or have significant direction conflicts are eliminated, retaining the effective gradient components that meet the scene geometric constraints; then, combined with the spatial coverage density distribution of the low-visibility Gaussian, the verified gradient is regionally smoothed and energy normalized to ensure that the corrected gradient result meets the requirements of subsequent densification optimization in terms of spatial continuity and physical rationality, thereby generating the final corrected low-visibility gradient result.

[0085] Furthermore, the hierarchical processing results are integrated to optimize the Gaussian densification mechanism and generate a refined 3D Gaussian distribution, including:

[0086] Based on the highly visible gradient results and the corrected low visible gradient results in the layered processing results, gradient integration processing is performed to generate an integrated gradient result;

[0087] According to the integrated gradient results, the Gaussian whose size exceeds the average scale of the scene is split, and its splitting direction is calculated to generate the split Gaussian;

[0088] Perform cloning and filling processing on the optimized and stable small-size Gaussian to generate a cloned Gaussian;

[0089] Based on the split Gaussian and the cloned Gaussian, the position, covariance and color attributes of the Gaussian are updated to generate a refined 3D Gaussian distribution.

[0090] Specifically, the highly visible gradient results output by the layered processing and the corrected low-visible gradient results are fused at multi-scale, and an integrated gradient result with balanced global and local features is generated through a weighted superposition strategy based on the gradient energy ratio; then, based on the vector direction distribution and modulus statistics of the integrated gradient result, the Gaussians whose sizes exceed the average scale of the scene are decomposed in the main direction and the orthogonal splitting directions are calculated, and combined with the density distribution constraints of the high-frequency texture area, a set of split Gaussians distributed along the extension direction of the geometric structure is generated; at the same time, the optimized and stable small-size Gaussians (such as Gaussians whose gradient volatility is lower than the set threshold and whose spatial position has not suddenly changed after continuous iteration) are subjected to density detection and adaptive cloning filling in the adjacent sparse areas to generate a set of cloned Gaussians for supplementary detail representation; then, based on the spatial distribution topological relationship between the split Gaussians and the cloned Gaussians, the position offset of the Gaussians, the anisotropy parameters of the covariance matrix and the perspective-related color properties driven by the spherical harmonic coefficients are synchronously updated through gradient backpropagation optimization and physical constraint interpolation, forming a refined 3D Gaussian distribution whose geometric details closely fit the high-frequency texture of the scene and whose spatial distribution density adapts to rendering requirements.

[0091] Furthermore, based on the integrated gradient results, the Gaussians in the scene are split or cloned to generate new Gaussians to fill the areas in the scene that are not yet fully reconstructed, including:

[0092] Based on the integrated gradient results, the Gaussian whose size exceeds the average scale of the scene is split to ensure that the excessively large Gaussian in the scene can be correctly split to represent the high-frequency texture area in the scene;

[0093] According to the integrated gradient results, the Gaussians whose sizes are smaller than the average scale of the scene are cloned to ensure that there are a sufficient number of Gaussians to represent the scene.

[0094] Specifically, based on the spatial density characteristics of the vector field distribution of the integrated gradient results and the visibility weight, the gradient principal components of the target Gaussian are eigenvalue decomposed to extract the principal axis component of its geometric extension direction. The initial splitting direction is calculated in combination with the distribution consistency of the visibility weight in adjacent views. Then, the initial splitting direction is locally curvature aligned and directional corrected according to the density distribution map of the high-frequency texture area of the scene and the geometric edge direction. At the same time, the number of splits and the spatial spacing distribution of the sub-Gaussians are dynamically adjusted based on the texture density threshold to generate optimized splitting parameters (including the number of splits, position offset and direction constraint factor). Then, the original Gaussian is attribute redistributed according to the splitting parameters: the positions are equally spaced along the corrected splitting direction to generate sub-Gaussian nodes, the anisotropy parameters of the covariance matrix are adjusted according to the angle between the splitting direction and the texture direction, and the perspective-dependent color attributes are bidirectionally interpolated and inherited based on the spherical harmonic coefficients of the original Gaussian to ensure that the split sub-Gaussians closely fit the high-frequency details of the scene in terms of geometric structure and appearance attributes, thereby generating a physically reasonable and detail-enhanced split Gaussian set.

[0095] Furthermore, the input multi-view images are subjected to sparse point cloud reconstruction to generate an initial 3D Gaussian distribution, including:

[0096] Perform feature extraction and matching processing on the input multi-view images to generate cross-view matching point pairs;

[0097] Perform sparse point cloud reconstruction based on matching point pairs to generate an initial sparse point cloud;

[0098] Perform covariance estimation on each point in the initial sparse point cloud to generate the covariance matrix of each point;

[0099] Construct an initial 3D Gaussian distribution based on the covariance matrix and preset Gaussian attribute parameters.

[0100] Specifically, a cross-view matching algorithm based on deep feature descriptors (such as SIFT or deep learning features) is used to extract feature points and perform robust matching on the input multi-view images. Epipolar geometric constraints and the RANSAC algorithm are used to iteratively filter out mismatched point pairs to generate a high-confidence cross-view matching point set. Then, based on the Structure from Motion (SFM) framework, the three-dimensional spatial coordinates of the matching point pairs are calculated by multi-view geometric triangulation principle, and the bundle adjustment is combined with the bundle adjustment to obtain the best matching point set. Adjustment) optimizes the camera pose and point cloud position to generate an initial sparse point cloud with optimized geometric consistency. Then, for each 3D point in the sparse point cloud, the covariance matrix characterizing the anisotropy of the spatial distribution is estimated based on the KNN nearest neighbor algorithm of the distribution of its neighborhood points. The uncertainty quantification parameters in the reconstruction process are integrated to construct an error-aware correction model for the covariance matrix. Afterwards, the covariance matrix is combined with preset Gaussian attribute parameters (including initial opacity and perspective-dependent color basis functions based on spherical harmonic coefficients) to generate an initial 3D Gaussian distribution covering the key geometric structures of the scene, providing physically reasonable initialization conditions for subsequent gradient optimization and densification processing.

[0101] It should be understood that although the steps in the flowcharts of the various embodiments described above are shown sequentially as indicated by the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Except for step S101, which is only executed once during the initialization phase, the remaining steps are executed multiple times in a loop. Moreover, at least some of the steps in the flowcharts of the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but may be executed at different times.

[0102] In one embodiment, if Figure 2 As shown, the present application also provides a 3DGS new view rendering quality improvement system 200 based on Gaussian visibility, the system 200 comprising:

[0103] The sparse point cloud reconstruction module 201 is used to perform sparse point cloud reconstruction processing on the input multi-view images to generate an initial 3D Gaussian distribution;

[0104] The visibility filtering module 202 is configured to perform visibility determination on each Gaussian based on the projection area of the existing 3D Gaussian distribution in the current iteration and the occlusion relationship between the Gaussians, filter out Gaussians that are not visible relative to the view, and obtain a visibility determination result.

[0105] The weight calculation module 203 is used to perform an integral calculation on the coverage and contribution strength of each Gaussian in the view according to the visibility judgment processing result to generate a Gaussian visibility weight;

[0106] A layered gradient module 204 is used to perform layered processing on the Gaussian based on the Gaussian visibility weight, dynamically adjust the gradient accumulation strategy, and obtain a layered processing result;

[0107] Densification optimization module 205, for optimizing the Gaussian densification mechanism based on the layered processing results to generate a refined 3D Gaussian distribution;

[0108] The rendering output module 206 is configured to perform new view rendering processing based on the refined 3D Gaussian distribution and output an artifact-free image.

[0109] Specifically, the sparse point cloud reconstruction module 201 realizes scene initialization, uses multi-view geometric triangulation and bundle adjustment to generate the initial sparse point cloud, and estimates the covariance matrix of each point based on the KNN nearest neighbor algorithm, and constructs the initial Gaussian distribution in combination with the preset opacity and spherical harmonic color basis function; the visibility filtering module 202 is based on the depth sorting sequence and projection area judgment, and filters the invisible Gaussian through the double verification of depth occlusion detection and pixel contribution value; the weight calculation module 203 generates a weight distribution that quantifies the visibility strength according to the coverage pixel ratio and contribution value integral of the Gaussian in the view; the layered gradient Module 204 divides the high / low visible Gaussian sets by dynamic thresholding, applies gradient correction based on the weighted attenuation coefficient to the low visible set, and integrates noise filtering with spatial continuity constraints to generate a layered gradient result; the densification optimization module 205 integrates gradient data, drives large Gaussian splitting based on texture density maps and gradient main direction decomposition, combines clone filling mechanism to supplement detail areas, and simultaneously optimizes position, covariance and color attributes through back propagation; the rendering output module 206 uses a depth-sorted alpha blending rendering pipeline, combined with anisotropic Gaussian projection and anti-aliasing filtering to generate artifact-free images.

[0110] The layered gradient module 204 is further configured to:

[0111] Based on the Gaussian visibility weight, the Gaussians are classified to generate a set of highly visible Gaussians and a set of low-visibility Gaussians;

[0112] The Gaussians in the highly visible Gaussian set are processed according to the original gradient accumulation strategy to generate highly visible gradient results;

[0113] The Gaussians in the low-visibility Gaussian set are processed based on the dynamic attenuation coefficient to generate a corrected low-visibility gradient result;

[0114] The highly visible gradient results and the corrected low visible gradient results are integrated to generate the hierarchical processing results after the gradient accumulation strategy is adjusted.

[0115] The layered gradient module 204 is further configured to:

[0116] Extract the original gradient accumulation parameters based on the Gaussian properties in the highly visible Gaussian set;

[0117] According to the original gradient accumulation parameters, the Gaussians in the highly visible Gaussian set are subjected to gradient accumulation processing to generate an uncorrected gradient result;

[0118] Perform gradient screening on the uncorrected gradient result to filter out noise interference and obtain the screened gradient result;

[0119] Based on the spatial distribution relationship between the filtered gradient results and the highly visible Gaussian set, a highly visible gradient result is generated.

[0120] The layered gradient module 204 is further configured to:

[0121] Generate dynamic attenuation coefficient based on dynamic visibility weight distribution of low-visibility Gaussian set;

[0122] For the Gaussians in the low-visibility Gaussian set, gradient weighting is performed according to the dynamic attenuation coefficient to generate a preliminary correction gradient;

[0123] Perform gradient stability verification on the preliminary corrected gradient to remove abnormal gradient interference and obtain the verified gradient;

[0124] Based on the relationship between the verified gradient and the spatial coverage of the low-visibility Gaussian set, a corrected low-visibility gradient result is generated.

[0125] The densification optimization module 205 is also used to:

[0126] Generate dynamic attenuation coefficient based on dynamic visibility weight distribution of low-visibility Gaussian set;

[0127] For the Gaussians in the low-visibility Gaussian set, gradient weighting is performed according to the dynamic attenuation coefficient to generate a preliminary correction gradient;

[0128] Perform gradient stability verification on the preliminary corrected gradient to remove abnormal gradient interference and obtain the verified gradient;

[0129] Based on the relationship between the verified gradient and the spatial coverage of the low-visibility Gaussian set, a corrected low-visibility gradient result is generated.

[0130] The densification optimization module 205 is also used to:

[0131] According to the integrated gradient results, the Gaussian whose size exceeds the average scale of the scene is split to generate the split Gaussian;

[0132] Perform cloning and filling processing on the optimized and stable small-size Gaussian to generate a cloned Gaussian;

[0133] Based on the split Gaussian and the cloned Gaussian, the position, covariance and color attributes of the Gaussian are updated to generate a refined 3D Gaussian distribution.

[0134] The sparse point cloud reconstruction module 201 is further used for:

[0135] Perform feature extraction and matching processing on the input multi-view images to generate cross-view matching point pairs;

[0136] Perform sparse point cloud reconstruction based on matching point pairs to generate an initial sparse point cloud;

[0137] Perform covariance estimation on each point in the initial sparse point cloud to generate the covariance matrix of each point;

[0138] Construct an initial 3D Gaussian distribution based on the covariance matrix and preset Gaussian attribute parameters.

[0139] In one embodiment, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0140] In one embodiment, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned method embodiments when the computer program is executed by a processor.

[0141] In one embodiment, strict contribution-based constraints are introduced to the existing 3DGS visibility criteria. When the contribution of a Gaussian to any pixel in the current view exceeds a threshold, the pixel is considered visible; otherwise, it is considered invisible. This eliminates the invalid gradient accumulation of occluded Gaussians, preventing gradient dilution and ensuring that the gradient calculation of large-scale Gaussians is based only on valid visible views, thus improving segmentation capabilities.

[0142] In one embodiment, the visibility difference of the Gaussian is quantified based on the projection area coverage and center contribution strength of the 2D Gaussian in the view. The "highly visible" and "lowly visible" Gaussian sets are dynamically divided based on the visibility weight. The former uses the original gradient accumulation strategy, while the latter uses a gradient correction based on weight attenuation. The differentiated gradient accumulation strategy reduces the noise interference of the low-visibility Gaussian, while enhancing the effective gradient signal in the high-frequency area, and optimizing the Gaussian splitting direction and density distribution. The formula used is as follows:

[0143]

[0144]

[0145] in, represents the center of the 2D Gaussian, R represents the radius of the 2D Gaussian rectangular bounding box, G 2D represents a 2D Gaussian, is any pixel point on the two-dimensional image plane, ω represents the visibility weight, τ weight represents a given threshold, N represents the depth sorting sequence, x represents the horizontal coordinate of pixel p, y represents the vertical coordinate of pixel p, i and j are index variables, λ represents a custom parameter used to prevent the newly calculated gradient level from having a large difference with the gradient level of the original 3DGS, W is the camera extrinsic parameter matrix of the current view, T L represents the set of iterations at which the Gaussian is considered “highly visible”, T H represents the set of iterations for which the Gaussian is considered “lowly visible”, g i Represents the position gradient value obtained by the Gaussian ball in each iteration, Indicates the gradient level of the Gaussian sphere.

[0146] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the components described as separate parts may or may not be physically separated, and the parts displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0147] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. A 3DGS new view rendering quality improvement method based on Gaussian visibility, characterized in that: The method comprises: Perform sparse point cloud reconstruction on the input multi-view images to generate an initial 3D Gaussian distribution; Based on the occlusion relationship between the projection area of the existing 3D Gaussian distribution in the 2D view and the Gaussian in the current iteration, visibility judgment processing is performed on each Gaussian, and invisible Gaussians are filtered out to obtain the visibility judgment processing result; According to the visibility judgment processing result, performing an integral calculation on the coverage and contribution strength of each visible Gaussian in the view to generate a Gaussian visibility weight; Based on the Gaussian visibility weight, the Gaussian is layered and the gradient accumulation strategy is dynamically adjusted to obtain a layered processing result; Integrating the layered processing results, optimizing the Gaussian densification mechanism, and generating a refined 3D Gaussian distribution; Based on the refined 3D Gaussian distribution, a new view rendering process is performed to output an artifact-free image.

2. The method for improving 3DGS new view rendering quality based on Gaussian visibility according to claim 1, characterized in that: Based on the Gaussian visibility weight, the Gaussian is layered and the gradient accumulation strategy is dynamically adjusted to obtain the layered processing results, including: Based on the Gaussian visibility weights, the Gaussians are layered to generate a highly visible Gaussian set and a low visible Gaussian set; Processing the Gaussians in the highly visible Gaussian set according to the original gradient accumulation strategy to generate a highly visible gradient result; Processing the Gaussians in the low-visibility Gaussian set based on a dynamic attenuation coefficient to generate a modified low-visibility gradient result; The highly visible gradient result and the corrected low visible gradient result are integrated to adjust and optimize the gradient accumulation strategy.

3. The method for improving 3DGS new view rendering quality based on Gaussian visibility according to claim 2, characterized in that: The processing of the Gaussians in the highly visible Gaussian set according to the original gradient accumulation strategy to generate a highly visible gradient result includes: Extracting the position gradient of Gaussians in the highly visible Gaussian set obtained during stochastic gradient descent back propagation; Performing gradient screening on the position gradient to filter out noise interference and obtain a screened position gradient; According to the filtered position gradient, a gradient accumulation process is performed on the Gaussians in the highly visible Gaussian set to generate a highly visible gradient result.

4. The method for improving 3DGS new view rendering quality based on Gaussian visibility according to claim 2, characterized in that: The processing of the Gaussians in the low-visibility Gaussian set based on the dynamic attenuation coefficient to generate a corrected low-visibility gradient result includes: generating a dynamic attenuation coefficient based on the dynamic Gaussian visibility weight; Performing a rationality judgment on the dynamic attenuation coefficient, eliminating abnormal value interference, and obtaining a denoised dynamic attenuation coefficient; The Gaussians in the low-visibility Gaussian set are subjected to gradient weighting processing according to the dynamic attenuation coefficient to generate a modified low-visibility gradient result.

5. The method for improving 3DGS new view rendering quality based on Gaussian visibility according to claim 1, characterized in that: The step of integrating the layered processing results, optimizing the Gaussian densification mechanism, and generating a refined 3D Gaussian distribution includes: Performing gradient integration processing on the highly visible gradient result and the corrected low visible gradient result in the layered processing result to generate an integrated gradient result; Splitting or cloning the Gaussians in the scene based on the integrated gradient results to generate new Gaussians to fill in the insufficiently reconstructed areas of the scene; Based on the new Gaussian generated by the splitting or cloning, the position, covariance and color attributes of the Gaussian are updated to generate the refined 3D Gaussian distribution.

6. The method for improving 3DGS new view rendering quality based on Gaussian visibility according to claim 5, characterized in that: The step of splitting or cloning the Gaussians in the scene based on the integrated gradient result to generate new Gaussians to fill in the insufficiently reconstructed areas in the scene includes: Based on the integrated gradient result, splitting is performed on Gaussians whose sizes exceed the average scale of the scene, so as to ensure that the excessively large Gaussians in the scene can be correctly split to represent the high-frequency texture areas in the scene; Based on the integrated gradient result, Gaussians with sizes smaller than the average scale of the scene are cloned to ensure that there are a sufficient number of Gaussians to represent the scene.

7. The method for improving 3DGS new view rendering quality based on Gaussian visibility according to claim 1, characterized in that: The step of performing sparse point cloud reconstruction on the input multi-view images to generate an initial 3D Gaussian distribution includes: Perform feature extraction and matching on the input multi-view images to generate cross-view matching point pairs; Performing sparse point cloud reconstruction processing based on the matching point pairs to generate an initial sparse point cloud; Performing covariance estimation processing on each point in the initial sparse point cloud to generate a covariance matrix for each point; The initial 3D Gaussian distribution is constructed based on the covariance matrix and preset Gaussian attribute parameters.

8. A 3DGS new view rendering quality improvement system based on Gaussian visibility, characterized in that: The system comprises: The sparse point cloud reconstruction module is used to perform sparse point cloud reconstruction on the input multi-view images and generate the initial 3D Gaussian distribution; The visibility filtering module is used to perform visibility judgment on each Gaussian based on the projection area of the existing 3D Gaussian distribution in the current iteration and the occlusion relationship between the Gaussians, and filter out Gaussians that are invisible relative to the current view to obtain the visibility judgment result; a weight calculation module, configured to perform integral calculation on the coverage and contribution strength of each visible Gaussian in the view according to the visibility judgment processing result, and generate a Gaussian visibility weight; A hierarchical gradient module is used to perform hierarchical processing on the Gaussian based on the Gaussian visibility weight, dynamically adjust the gradient accumulation strategy, and obtain a hierarchical processing result; A densification optimization module is used to integrate the layered processing results, optimize the Gaussian densification mechanism, and generate a refined 3D Gaussian distribution; A rendering output module is used to perform new view rendering processing based on the refined 3D Gaussian distribution and output an artifact-free image.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of a method for improving 3DGS new view rendering quality based on Gaussian visibility according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a method for improving 3DGS new view rendering quality based on Gaussian visibility according to any one of claims 1 to 7 are implemented.

Images