A saliency guided 3d gaussian reconstruction method

Abstract

The application relates to the technical field of three-dimensional images, in particular to a saliency-guided three-dimensional Gaussian reconstruction method, which comprises the following steps: performing search-based matching on multi-view images to construct a globally connected image matching graph; performing first-order motion restoration structure optimization based on the image matching graph to obtain initial camera parameters and sparse three-dimensional structures; initializing a three-dimensional Gaussian scene representation based on the initial camera parameters and the sparse three-dimensional structures; calculating Gaussian saliency scores and Gaussian redundancy scores based on multi-view rendering errors, and performing Gaussian generation, Gaussian relocation and Gaussian capacity redistribution; and simultaneously optimizing camera poses based on photometric loss and geometric re-projection loss to obtain a trained target three-dimensional Gaussian scene representation and optimized camera parameters. The method can reduce the dependence on pre-processing, cooperatively optimize poses and scene representations, and dynamically allocate Gaussian resources to improve overall reconstruction quality and efficiency.

Description

Technical Field

[0001] The embodiments of this application relate to the field of three-dimensional image technology, and in particular to a saliency-guided three-dimensional Gaussian reconstruction method. Background Technology

[0002] With the development of applications such as novel perspective synthesis, digital twins, and robot perception, rapidly reconstructing high-quality 3D scenes from multiple 2D images has become an important research direction in the field of computer vision. Existing 3D Gaussian splashing technology, due to its fast rendering speed and strong expressive power, has become one of the mainstream technical approaches for high-fidelity scene representation.

[0003] However, existing 3D Gaussian reconstruction workflows generally employ a two-stage paradigm: first, using Structure for Motion Reconstruction (SfM) to recover the camera pose and sparse point cloud from the input image; second, using this as initialization conditions to train a 3D Gaussian scene representation. However, this approach has several problems: First, traditional SfM relies on pairwise matching of large-scale images and incremental bundle adjustment. As the number of images increases, the preprocessing computation cost rises significantly, resulting in excessive time consumption. Second, existing workflows typically pass the camera pose output by SfM as a fixed input to the 3D Gaussian training stage, making it difficult to effectively correct front-end pose errors in subsequent training, leading to problems such as texture blurring and edge misalignment in the reconstruction results. Third, existing 3D Gaussian density control strategies often rely on gradient magnitude or thresholds for densification and pruning, easily over-allocating representational power to well-fitted regions while neglecting high-frequency detail regions that have long been underfitted, resulting in wasted representational resources and decreased reconstruction quality. Summary of the Invention

[0004] In view of this, embodiments of this application propose a saliency-guided 3D Gaussian reconstruction method, which aims to overcome the shortcomings of existing 3D Gaussian reconstruction methods, such as high preprocessing time, difficulty in closed-loop correction of pose errors, and low efficiency of Gaussian resource allocation. In this way, while reducing preprocessing dependence, it can collaboratively optimize pose and scene representation, and dynamically allocate Gaussian resources to improve the overall reconstruction quality and efficiency.

[0005] To achieve the above objectives, embodiments of this application propose a saliency-guided 3D Gaussian reconstruction method, the method comprising the following steps: Obtain multi-view images of the scene to be reconstructed; Perform retrieval-based matching on multi-view images to construct a globally connected image matching graph; First-order motion recovery structure optimization is performed based on image matching graphs to obtain initial camera parameters and sparse 3D structure; Initialize the 3D Gaussian scene representation based on the initial camera parameters and sparse 3D structure; During the 3D Gaussian training process, Gaussian saliency score and Gaussian redundancy score are calculated based on multi-view rendering error, and Gaussian generation, Gaussian relocation and Gaussian capacity redistribution are performed based on Gaussian saliency score and Gaussian redundancy score. While optimizing the Gaussian parameters, the camera pose is jointly optimized based on photometric loss and geometric reprojection loss, resulting in the trained target 3D Gaussian scene representation and optimized camera parameters.

[0006] To achieve the above objectives, embodiments of this application also propose a saliency-guided 3D Gaussian reconstruction system, the system comprising: The image input module is used to acquire multi-view images of the scene to be reconstructed; The matching graph construction module is used to perform retrieval-based matching on multi-view images and construct a globally connected image matching graph. The first-order motion reconstruction structure optimization module is used to perform first-order motion reconstruction structure optimization based on the image matching map to obtain the initial camera parameters and sparse 3D structure. The Gaussian initialization module is used to initialize a 3D Gaussian scene representation based on initial camera parameters and a sparse 3D structure. The saliency evaluation module is used to calculate the Gaussian saliency score and Gaussian redundancy score based on the multi-view rendering error during the 3D Gaussian training process. The Gaussian allocation module is used to perform Gaussian generation, Gaussian relocation, and Gaussian capacity redistribution based on Gaussian saliency scores and Gaussian redundancy scores. The joint optimization module is used to jointly optimize the camera pose based on photometric loss and geometric reprojection loss while optimizing Gaussian parameters. The output module is used to output the trained 3D Gaussian scene representation of the target and the optimized camera parameters.

[0007] The embodiments of this application propose a saliency-guided 3D Gaussian reconstruction method. First, multi-view images of the scene to be reconstructed are acquired. Retrieval-based matching is performed on the multi-view images to construct a globally connected image matching graph. First-order motion recovery structure optimization is performed based on the image matching graph to obtain initial camera parameters and a sparse 3D structure. A 3D Gaussian scene representation is initialized based on the initial camera parameters and the sparse 3D structure. During 3D Gaussian training, Gaussian saliency scores and Gaussian redundancy scores are calculated based on the multi-view rendering errors, and Gaussian generation, Gaussian relocalization, and Gaussian capacity reallocation are performed based on these scores. While optimizing the Gaussian parameters, the camera pose is jointly optimized based on photometric loss and geometric reprojection loss to obtain the trained target 3D Gaussian scene representation and optimized camera parameters. This scheme overcomes the shortcomings of existing 3D Gaussian reconstruction methods, such as high preprocessing time, difficulty in closed-loop correction of pose errors, and low Gaussian resource allocation efficiency. Furthermore, while reducing preprocessing dependence, it can collaboratively optimize pose and scene representation and dynamically allocate Gaussian resources to improve overall reconstruction quality and efficiency.

[0008] Optionally, a retrieval-based matching process is performed on the multi-view images to construct a globally connected image matching graph. This includes: extracting local features from each image and generating an image-level global descriptor based on a Gaussian mixture model; using an approximate nearest neighbor retrieval method to retrieve similar images for each image based on the image-level global descriptor, thus constructing a sparse candidate matching graph; constructing a maximum spanning tree based on the similarity weights of edges in the candidate matching graph to supplement cross-cluster connection edges; performing robust geometric verification based on polar geometric constraints on candidate image pairs to eliminate incorrect matches; and merging the retrieved edges that pass the geometric verification with the edges supplemented by the maximum spanning tree to obtain the final image matching graph.

[0009] Optionally, first-order motion recovery structure optimization is performed based on the image matching map to obtain initial camera parameters and sparse 3D structure, including: estimating camera intrinsic parameters and distortion parameters based on the image matching map; solving global rotation based on the relative rotation constraints between pairs of images, and solving global translation based on the consistency of translation direction; constructing a quadratic form expression corresponding to the epigeometric constraints, performing structure-independent first-order epigeometric adjustments on the camera parameters, so that the optimization cost is weakly dependent on the number of 3D points; recovering sparse 3D points based on the optimized camera parameters, and constructing cross-view feature trajectories; outputting initial camera parameters, camera intrinsic parameters, sparse 3D points, and corresponding feature trajectory information.

[0010] Optionally, during 3D Gaussian training, Gaussian saliency scores and Gaussian redundancy scores are calculated based on multi-view rendering errors, and Gaussian generation, Gaussian relocalization, and Gaussian capacity redistribution are performed based on these scores. This includes: periodically sampling multiple viewpoints during training to render the current 3D Gaussian scene representation; calculating the pixel-level error between the rendered image and the real image under each sampled viewpoint; performing quantile normalization on the pixel-level errors under each sampled viewpoint to obtain a robust error response map, thereby enhancing consistency across viewpoints and training phases; aggregating high-error responses based on the two-dimensional projection footprints of each Gaussian under the sampled viewpoints to obtain Gaussian saliency scores; aggregating low-error responses based on the two-dimensional projection footprints of each Gaussian under the sampled viewpoints to obtain Gaussian redundancy scores; and performing Gaussian generation, Gaussian relocalization, and Gaussian capacity redistribution based on the aggregation results of the Gaussian saliency scores and Gaussian redundancy scores.

[0011] Optionally, based on the aggregation results of Gaussian significance score and Gaussian redundancy score, Gaussian generation, Gaussian relocation, and Gaussian capacity redistribution are performed, including: constructing a significance-guided sampling distribution based on the Gaussian significance score, and sampling a parent Gaussian from the significance-guided sampling distribution to generate a new Gaussian; and / or, selecting a low-value Gaussian based on the Gaussian redundancy score and transparency, and relocating it to a high-significance region; wherein, the low-value Gaussian is a Gaussian with a redundancy score higher than a preset value.

[0012] Optionally, the pixel-level errors under each sampling viewpoint are normalized by quantiles to obtain a robust error response map, specifically including: ; in, This represents the robust error response diagram after quantile normalization. Indicates the first Pixel-level error map from each viewpoint and These represent the lower quantile and the upper quantile, respectively.

[0013] Gaussian significance score The calculation formulas include: ; in, Indicates the first Gauss in the th... Two-dimensional projected footprint area from each viewpoint Represents pixels High error weighting; The formula for calculating the Gaussian redundancy fraction includes: ; in, Represents pixels Low error weights.

[0014] Optionally, while optimizing the Gaussian parameters, the camera pose is jointly optimized based on photometric loss and geometric reprojection loss to obtain the trained target 3D Gaussian scene representation and optimized camera parameters. This includes: establishing a photometric loss function, which is used to constrain the difference between the rendered image and the real image; establishing a geometric reprojection loss function based on the sparse feature trajectories corresponding to the sparse 3D structure, which is used to constrain the camera pose; weighted summing of the photometric loss function, geometric reprojection loss function, and Gaussian regularization term to obtain the total loss function; optimizing the camera pose based on the total loss function, and simultaneously updating the Gaussian parameters and camera pose correction during training to output the optimized target 3D Gaussian scene representation, camera pose parameters, and new viewpoint rendering results.

[0015] Optionally, the method provided in the embodiments of this application further includes: eliminating normative degrees of freedom by fixing at least one reference camera.

[0016] Optionally, photometric loss function The expression is: ; in, For real images, To render the image, The balance coefficients for structural similarity terms;

[0017] The expression for the geometric reprojection loss function is: ; in, Represents two-dimensional observation geometry; Indicates the first Each camera pose; For triangulated 3D points; For the corresponding two-dimensional observation; For projection functions; The expression for the total loss function is: ; in, For the geometric reprojection loss weights, These are regularization terms related to Gaussian scaling, transparency, and stability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0019] Figure 1 This is a flowchart of a saliency-guided 3D Gaussian reconstruction method provided in one embodiment of this application; Figure 2 This is a flowchart of a global matching graph construction provided in one embodiment of this application; Figure 3 This is a flowchart of a first-order motion recovery structure provided in one embodiment of this application; Figure 4 This is a flowchart of a saliency-guided Gaussian assignment provided in one embodiment of this application; Figure 5 This is a flowchart of a joint optimization of data flow provided in one embodiment of this application; Figure 6 This is a comparison diagram of effects provided in one embodiment of this application; Figure 7 This is another effect comparison diagram provided in one embodiment of this application; Figure 8 This is a schematic diagram of a saliency-guided three-dimensional Gaussian reconstruction system provided in another embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0021] With the development of applications such as new perspective synthesis, digital twins, robot perception, cultural heritage modeling, and immersive content production, how to quickly reconstruct high-quality 3D scenes from multiple 2D images has become an important research direction in the field of computer vision. Among existing 3D reconstruction methods, 3D Gaussian splashing technology has gradually become an important technical route for high-fidelity scene representation due to its fast rendering speed and strong expressive power.

[0022] Existing 3D Gaussian reconstruction workflows typically include the following two stages: First, a Structure-from-Motion (SfM) method is used to recover the camera pose and sparse point cloud from the input image; second, the obtained camera pose and sparse point cloud are used as initialization conditions to train a 3D Gaussian scene representation. This two-stage workflow has shown some effectiveness in practical applications, but several problems remain.

[0023] On the one hand, traditional motion recovery structures typically rely on large-scale pairwise image matching, incremental image registration, and second-order bundle adjustment. As the number of input images increases, the computational cost of image pairing and pose optimization rises significantly, resulting in long preprocessing times, which in many scenarios even exceed those of 3D Gaussian training.

[0024] On the other hand, most existing processes directly pass the camera pose output from the motion reconstruction structure as a fixed input to the 3D Gaussian training stage. If there is an error in the front-end estimation, it is difficult to effectively correct in the subsequent training stage, which can easily lead to problems such as texture blurring, edge misalignment, local geometric deformation, or inconsistency across views in the reconstruction results.

[0025] Meanwhile, some existing weak SfM schemes rely on ordered video input, feedforward network initialization, or pose correction driven by pure photometric errors. These methods still have limited adaptability to scenes with sparse textures, uneven viewpoint distribution, and complex shooting trajectories, and suffer from insufficient stability. Therefore, there is an urgent need to provide a new 3D reconstruction method that can quickly and reliably complete image matching and pose initialization while ensuring unordered image input. Furthermore, during training, it should dynamically redistribute Gaussian representation capabilities based on multi-view error information, and jointly optimize camera pose and scene appearance to improve end-to-end reconstruction efficiency, pose stability, and final reconstruction quality.

[0026] Furthermore, most existing 3D Gaussian density control strategies rely on gradient magnitude, threshold splitting, cloning, or transparency pruning for densification and reduction. When the number of Gaussians is limited, these methods tend to over-allocate representational power to well-fitted regions, neglecting high-frequency detail regions that have long been underfitted, resulting in wasted representational resources and decreased reconstruction quality. Some methods also cause a rapid expansion of the number of Gaussians, which is detrimental to controlling GPU memory and rendering complexity.

[0027] In view of this, embodiments of this application propose a saliency-guided 3D Gaussian reconstruction method, which aims to overcome the shortcomings of existing 3D Gaussian reconstruction methods, such as high preprocessing time, difficulty in closed-loop correction of pose errors, and low efficiency of Gaussian resource allocation. In this way, while reducing preprocessing dependence, it can collaboratively optimize pose and scene representation, and dynamically allocate Gaussian resources to improve the overall reconstruction quality and efficiency.

[0028] Specifically, addressing the technical challenges of existing technologies, such as high preprocessing time, difficulty in closed-loop correction of pose errors, low efficiency of Gaussian resource allocation, and the difficulty in balancing end-to-end reconstruction quality and efficiency, this embodiment proposes a saliency-guided 3D Gaussian reconstruction method. This method constructs a unified framework for joint optimization of a first-order motion recovery structure and 3D Gaussian representation, and designs a dynamic Gaussian allocation mechanism for salient and underfitting regions. This reduces preprocessing dependence while achieving collaborative optimization of pose and scene representation, thereby improving applicability to disordered multi-view images and enhancing overall reconstruction quality and efficiency. Under finite Gaussian budget constraints, this algorithm can still prioritize enhancing the representation of underfitting regions and achieve superior 3D reconstruction results.

[0029] One embodiment of this application proposes a saliency-guided 3D Gaussian reconstruction method applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments will use a server as an example for illustration. The implementation details of the saliency-guided 3D Gaussian reconstruction method proposed in this embodiment are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this solution.

[0030] The specific process of the saliency-guided 3D Gaussian reconstruction method proposed in this embodiment can be as follows: Figure 1 As shown, it includes: Step 101: Obtain multi-view images of the scene to be reconstructed.

[0031] For example, an unordered collection of multi-view images of the same scene to be reconstructed can be obtained.

[0032] Specifically, multiple images of the same scene taken from different perspectives are acquired. These images can be input in an unordered manner, meaning a strict shooting order or trajectory is not required. The images can be acquired by handheld terminals, drones, industrial cameras, vehicle-mounted equipment, or other imaging devices. The number of images can be dozens, hundreds, or more. Preferably, the input images should cover multiple directions and scales of the target scene to ensure the integrity of the reconstruction.

[0033] Optionally, to facilitate subsequent processing, this step also includes preprocessing the acquired images. For example, all images may undergo resolution unification, distortion preprocessing, and format standardization to obtain a set of images to be paired. Simultaneously, an image index table is created, assigning a unique image number to each image for subsequent construction of the matching map and camera number.

[0034] It is understood that this embodiment is applicable to unordered multi-view image input, does not rely on strict timing acquisition conditions, and has strong engineering applicability.

[0035] Step 102: Perform retrieval-based matching on the multi-view images to construct a globally connected image matching graph.

[0036] Understandably, this step aims to efficiently and reliably identify the correspondences between images, laying the foundation for subsequent 3D reconstruction.

[0037] like Figure 2 As shown, in one possible embodiment, step 102 includes: Step 1021: Extract local features from each image and generate image-level global descriptors based on Gaussian mixture models.

[0038] For example, firstly, local features are extracted for each input image. Then, Scale-Invariant Feature Transform (SIFT) features are used as local feature descriptors. SIFT features have good scale and rotation invariance and are suitable for multi-view image matching.

[0039] Understandably, in order to quickly retrieve similar images and avoid time-consuming pairwise matching, this step uses a Gaussian Mixture Model (GMM) to encode local features and generate image-level global descriptors.

[0040] For example, an image-level global descriptor is the Fisher Vector descriptor. The Fisher Vector can aggregate a set of local features into a fixed-length vector, effectively capturing the overall visual distribution of an image.

[0041] Step 1022: Using an approximate nearest neighbor retrieval method, similar images are retrieved for each image based on the image-level global descriptor, and a sparse candidate matching graph is constructed.

[0042] For example, an Approximate Nearest Neighbor (ANN) search method can be used to find the most similar image for each image based on the generated image-level global descriptor. The images form a sparse candidate matching graph.

[0043] in, The value can be set according to the image size, for example, between 5 and 30. The value is set to 20. This significantly reduces the number of image pairs requiring precise geometric verification, thus improving processing efficiency.

[0044] Step 1023: Construct a maximum spanning tree based on the similarity weights of edges in the candidate matching graph to supplement cross-cluster connection edges.

[0045] For example, because ANN retrieval tends to find local nearest neighbors, it may cause the image set to be segmented into several disconnected subgraphs (e.g., images taken from different sides of a scene). To ensure the global connectivity of the entire image set, this step constructs a Maximum Spanning Tree (MST) on the weighted graph composed of candidate matching graphs. By supplementing the connection edges across different visual clusters with the MST, it is ensured that any two images can be connected through a series of matching edges, thereby avoiding reconstruction failure.

[0046] Step 1024: Perform robust geometric verification based on polar geometric constraints on the candidate image pairs to eliminate incorrect matches.

[0047] For example, candidate image pairs include image pairs retrieved by ANN and image pairs supplemented by MST.

[0048] Robust geometric verification based on polar geometric constraints is performed on candidate image pairs to eliminate erroneous matches caused by factors such as texture duplication and illumination variations. In a preferred embodiment, the Random Sample Consensus (RANSAC) algorithm is used to estimate the fundamental matrix or essential matrix and filter out outlier matches that do not conform to the polar geometric constraints.

[0049] Step 1025: The retrieved edges that have passed geometric verification are merged with the supplementary edges of the maximum spanning tree to obtain the final image matching graph.

[0050] For example, the retrieved edges that have passed geometric verification are merged with the maximum spanning tree supplementary edges to form a final image matching graph that combines sparsity, high reliability, and global connectivity. This matching graph will be used for subsequent pose recovery.

[0051] It is understandable that this embodiment avoids the high computational cost of traditional full-pair matching by using retrieval-based image matching and maximum spanning tree edge completion, thereby improving the processing efficiency of the image matching stage.

[0052] Step 103: Perform first-order motion recovery structure optimization based on the image matching map to obtain the initial camera parameters and sparse 3D structure.

[0053] For example, this step aims to quickly and robustly recover the camera's intrinsic and extrinsic parameters and the sparse 3D point cloud of the scene from the matched image.

[0054] like Figure 3 As shown, in one possible embodiment, step 103 includes: Step 1031: Estimate camera intrinsic parameters and distortion parameters based on image matching maps.

[0055] For example, the camera's intrinsic parameters and distortion parameters can be estimated based on the image matching map obtained in the above embodiments, utilizing the correspondence between images. For instance, initialization can be performed using a self-calibration method or by assuming known factory parameters for the camera model.

[0056] Step 1032: Solve the global rotation based on the relative rotation constraints between pairs of images, and solve the global translation based on the consistency of the translation direction.

[0057] For example, the global rotation pose of all cameras is solved based on the relative rotation constraints obtained through geometric verification between pairs of images. This method avoids the inefficiency and error accumulation problems of registering images one by one in traditional incremental SfM. Then, based on the known global rotation, the global translation vector of all cameras is solved according to the translation direction consistency constraint.

[0058] Step 1033: Construct a quadratic form expression corresponding to the polar geometry constraints, and perform a structure-independent first-order polar geometry adjustment on the camera parameters so that the optimization cost is weakly dependent on the number of 3D points.

[0059] For example, after initially obtaining the camera parameters, a lightweight global optimization is performed. Unlike traditional bundle adjustment (BA), which requires optimizing a large number of 3D point coordinates simultaneously, this step constructs a quadratic form expression corresponding to the polar geometry constraints, and performs a structure-independent first-order polar geometry adjustment on the camera parameters.

[0060] Understandably, by using the above method, pre-computed matrices can be used to represent constraints between image pairs during the optimization process. This makes the computational cost of each optimization round weakly dependent on the number of 3D points, thereby significantly improving preprocessing efficiency and reducing processing time. This adjustment process can correct camera parameters without explicitly co-optimizing a large number of 3D point coordinates.

[0061] Step 1034: Recover sparse 3D points based on the optimized camera parameters and construct cross-view feature trajectories.

[0062] For example, based on the optimized camera parameters, reliable sparse 3D points are recovered using a triangulation method, and a 2D observation trajectory, i.e. a feature trajectory, is constructed for each 3D point on multiple images.

[0063] Step 1035: Output the initial camera parameters, camera intrinsic parameters, sparse 3D points, and corresponding feature trajectory information.

[0064] For example, the initial camera pose, camera intrinsic parameters, sparse 3D points, and corresponding feature trajectory information can be output. This information will serve as the initialization conditions for subsequent steps.

[0065] It is understandable that this embodiment reduces the reliance on large-scale second-order beam adjustment by optimizing the first-order motion recovery structure, thereby shortening the preprocessing time while ensuring the initialization quality.

[0066] Step 104: Initialize the 3D Gaussian scene representation based on the initial camera parameters and sparse 3D structure.

[0067] Understandably, this step transforms the output of SfM into a differentiable, renderable 3D Gaussian scene representation. Specifically, it includes: Sparse 3D points are used as the initial center positions of a 3D Gaussian ellipsoid. Scale, rotation, color, and opacity parameters are set for each 3D Gaussian ellipsoid. For example, the scale parameter controls the ellipsoid size; the rotation parameter controls the ellipsoid orientation; the color parameter can be represented using spherical harmonics to support view-dependent colors; and the opacity parameter controls the opacity.

[0068] Based on camera pose, a differentiable rendering interface is established. Specifically, the minute correction of the camera pose is set as a learnable variable so that it can be updated synchronously with the Gaussian parameters during subsequent training, thereby achieving closed-loop correction of front-end pose errors.

[0069] To prevent the Gaussian number from expanding indefinitely and causing excessive memory and computational overhead, an upper limit or target budget for the Gaussian number is set to constrain the overall representation size during training.

[0070] Step 105: During the 3D Gaussian training process, Gaussian saliency score and Gaussian redundancy score are calculated based on the multi-view rendering error, and Gaussian generation, Gaussian relocation and Gaussian capacity redistribution are performed based on the Gaussian saliency score and Gaussian redundancy score.

[0071] It is understood that this embodiment aims to achieve dynamic and efficient allocation of finite Gaussian budgets.

[0072] like Figure 4 As shown, in one possible embodiment, step 105 includes: Step 1051: During the training process, multiple perspectives are periodically sampled to render the current 3D Gaussian scene representation.

[0073] For example, during the training of a 3D Gaussian model, samples are periodically taken from the training set. The system uses a single perspective and performs differentiable rendering on the current 3D Gaussian scene representation to obtain the rendered image.

[0074] Step 1052: Calculate the pixel-level error between the rendered image and the real image at each sampling viewpoint.

[0075] For example, the pixel-level error between the rendered image and the corresponding real image under each sampling viewpoint is calculated to obtain an error map.

[0076] Step 1053: Perform quantile normalization on the pixel-level errors under each sampling viewpoint to obtain a robust error response map, so as to enhance consistency across viewpoints and across training phases.

[0077] For example, to enhance the comparability and consistency of error metrics across different perspectives and training stages, pixel-level errors are normalized using quantiles to obtain a robust error response map. For instance, error values ​​can be mapped to a range of 0 to 1, where larger error values ​​correspond to higher response values.

[0078] For example, quantile normalization is performed on the pixel-level errors under each sampling viewpoint to obtain a robust error response map, specifically including: ; in, Indicates the first Error response value of each pixel after quantile normalization; Indicates the first Pixel-level error map from each viewpoint and These represent the lower quantile and the upper quantile, respectively. This represents a clipping function that restricts the calculation result to between 0 and 1. It outputs 0 when the numerator is less than 0 and 1 when it is greater than the denominator.

[0079] Step 1054: Based on the two-dimensional projection footprints of each Gaussian under the sampling viewpoint, the high-error responses are aggregated to obtain the Gaussian significance score.

[0080] For example, aggregation can be done by summation or averaging.

[0081] For example, for each 3D Gaussian, its 2D projection footprint (i.e., the region on the image projected by the Gaussian ellipsoid) is determined at each sampling viewpoint. Within this projection footprint, high error responses (e.g., errors higher than a first error response threshold) are aggregated to obtain the significance score of the Gaussian. A higher significance score indicates that the region containing the Gaussian is poorly fitted at the current viewpoint and is an underfitting region requiring special attention.

[0082] Gaussian significance score The calculation formulas include: ; in, Indicates the first Gauss in the th... Two-dimensional projected footprint area from each viewpoint Represents pixels High error weighting.

[0083] Step 1055: Based on the two-dimensional projection footprints of each Gaussian under the sampling viewpoint, the low-error responses are aggregated to obtain the Gaussian redundancy score.

[0084] For example, within the two-dimensional projected footprint of each Gaussian, low-error responses (e.g., errors below a second error response threshold) are aggregated to obtain the redundancy score of that Gaussian. A higher redundancy score indicates that the region containing that Gaussian has been well-fitted, and its representational power may be wasted.

[0085] The formula for calculating the Gaussian redundancy fraction includes: ; in, Represents pixels Low error weights.

[0086] Optionally, in order to reduce the undue statistical advantage of large-scale Gaussians due to their large projected area, the above aggregation results can be normalized by dividing the corresponding two-dimensional projected footprint area of ​​the Gaussian.

[0087] Step 1056: Based on the aggregation results of Gaussian significance score and Gaussian redundancy score, perform Gaussian generation, Gaussian relocation, and Gaussian capacity redistribution.

[0088] In one possible embodiment, based on the aggregation results of Gaussian saliency scores and Gaussian redundancy scores, Gaussian generation, Gaussian relocation, and Gaussian capacity reallocation are performed, including: Construct a significance-guided sampling distribution based on Gaussian significance scores, and sample a parent Gaussian from the significance-guided sampling distribution to generate a new Gaussian; and / or, select a low-value Gaussian based on Gaussian redundancy scores and transparency, and relocate it to a high-significance region.

[0089] Among them, low-value Gaussians are Gaussians with redundancy scores higher than the preset value.

[0090] For example, during Gaussian generation, a saliency-guided sampling distribution can be constructed based on the saliency scores of all Gaussians. Gaussians with higher saliency scores are more likely to be selected as parent Gaussians. Several parent Gaussians are sampled from this distribution, and new child Gaussians are generated in their vicinity, thereby injecting representational power into the underfitting region.

[0091] The significance-guided sampling distribution can be represented as: ; ; in, The significance threshold, The transparency blending factor. For the first A Gaussian transparency parameter.

[0092] For example, during Gaussian relocalization, Gaussians with very low transparency (nearly transparent, contributing little) or redundancy scores higher than a preset value are considered low-value Gaussians. These low-value Gaussians are removed from their current positions and repositioned into underfitting regions based on a saliency-guided sampling distribution. This is equivalent to moving representation resources from well-fitted regions to regions requiring more detail.

[0093] For example, during capacity redistribution, the aforementioned Gaussian generation and Gaussian relocation operations can achieve dynamic redistribution of Gaussian capacity. In this way, under a limited Gaussian budget, the system's representational power can adaptively tilt towards high-frequency detail regions that have been underfitting for a long time, thereby improving the utilization of representational resources and avoiding resource waste.

[0094] It is understood that this embodiment uses saliency score and redundancy score to jointly drive Gaussian generation and Gaussian relocation, so that the limited Gaussian budget is preferentially allocated to the persistent underfit region, thereby improving the utilization of representation resources.

[0095] It should be noted that the saliency-guided allocation in this embodiment is a heuristic allocation mechanism that operates on Gaussian capacity management. It does not change the differentiable optimization framework of the underlying parameter update, but can significantly improve the representation efficiency under finite budget conditions.

[0096] Step 106: While optimizing the Gaussian parameters, the camera pose is jointly optimized based on photometric loss and geometric reprojection loss to obtain the trained target 3D Gaussian scene representation and the optimized camera parameters.

[0097] Understandably, this step aims to achieve end-to-end joint optimization of scene representation and camera pose.

[0098] like Figure 5 As shown, in one possible embodiment, step 106 includes: Step 1061: Establish the photometric loss function; based on the sparse feature trajectories corresponding to the sparse 3D structure, establish the geometric reprojection loss function.

[0099] Among them, the photometric loss function is used to constrain the difference between the rendered image and the real image; the geometric reprojection loss function is used to constrain the camera pose.

[0100] Specifically, the photometric loss function is used to measure the appearance consistency between a rendered image and a realistic image. For example, the photometric loss function... The expression is: ; in, For real images, To render the image, The balance coefficients for structural similarity terms; The structural similarity index measures the similarity between two images in terms of brightness, contrast, and structure.

[0101] The expression for the geometric reprojection loss function is: ; in, Represents two-dimensional observation geometry; Indicates the first Each camera pose; For triangulated 3D points; For the corresponding two-dimensional observation; This is the projection function.

[0102] It is understood that this embodiment improves the stability of pose joint optimization and reduces the risk of degradation in pure photometric optimization by introducing geometric reprojection constraints based on sparse feature trajectories.

[0103] Step 1062: The photometric loss function, geometric reprojection loss function, and Gaussian regularization term are weighted and summed to obtain the total loss function.

[0104] For example, the expression for the total loss function is: ; in, For the geometric reprojection loss weights, These are regularization terms related to Gaussian scaling, transparency, and stability, used to prevent Gaussian degradation (such as scale explosion or transparency approaching zero).

[0105] Step 1063: Optimize the camera pose based on the total loss function, and update the Gaussian parameters and camera pose correction during training to output the optimized target 3D Gaussian scene representation, camera pose parameters and new view rendering results.

[0106] For example, during training, all Gaussian parameters and camera pose corrections are updated simultaneously using the backpropagation algorithm. In this way, pose errors introduced by front-end SfM can be effectively corrected during back-end training, achieving closed-loop optimization.

[0107] In this embodiment, at least one reference camera can be fixed during training to eliminate canonical degrees of freedom. Specifically, in order to eliminate global rotational and translational canonical degrees of freedom, at least one reference camera (e.g., the pose of the first camera) can be fixed during optimization, or the entire scene can be bound to a fixed reference coordinate system.

[0108] After training, the optimized 3D Gaussian scene representation of the target, the optimized camera pose parameters, and the rendering results from the new perspective are output.

[0109] It is understandable that this embodiment achieves closed-loop correction of front-end initialization errors by jointly optimizing camera pose during the 3D Gaussian training phase, thereby mitigating the error propagation problem caused by pose freezing.

[0110] This application proposes a saliency-guided 3D Gaussian reconstruction method, which acquires multi-view images of the scene to be reconstructed; performs retrieval-based matching on the multi-view images to construct a globally connected image matching graph; performs first-order motion recovery structure optimization based on the image matching graph to obtain initial camera parameters and a sparse 3D structure; initializes a 3D Gaussian scene representation based on the initial camera parameters and the sparse 3D structure; during 3D Gaussian training, calculates Gaussian saliency scores and Gaussian redundancy scores based on multi-view rendering errors, and performs Gaussian generation, Gaussian relocalization, and Gaussian capacity reallocation based on the Gaussian saliency scores and Gaussian redundancy scores; while optimizing Gaussian parameters, jointly optimizes camera pose based on photometric loss and geometric reprojection loss to obtain the trained target 3D Gaussian scene representation and optimized camera parameters; this scheme can overcome the shortcomings of existing 3D Gaussian reconstruction methods, such as high preprocessing time, difficulty in closed-loop correction of pose errors, and low Gaussian resource allocation efficiency, thereby reducing preprocessing dependence while synergistically optimizing pose and scene representation, and dynamically allocating Gaussian resources to improve overall reconstruction quality and efficiency.

[0111] based on Figure 6 and Figure 7 The comparison diagrams shown demonstrate that the embodiments of this application have the following advantages compared to the prior art: First, this application utilizes a saliency-guided Gaussian dynamic allocation mechanism to effectively filter out low-frequency noise and floating objects generated during training. During optimization, the system focuses Gaussian points on the true surface geometry, rather than on incorrectly fitted background interference or random errors in the optimization process, thereby significantly improving the smoothness and accuracy of the scene's geometric surface.

[0112] Second, traditional photometric loss optimization is prone to overfitting in areas with weak textures or partial occlusion (e.g., complex foliage), leading to floating objects. This solution, thanks to a joint optimization strategy and a Gaussian redundancy score removal mechanism, can effectively identify and remove redundant Gaussian points that contribute very little to the rendering or interfere with the geometry, ensuring the purity and physical realism of the background.

[0113] Third, the saliency-guided 3D Gaussian reconstruction method proposed in this application can prioritize the allocation of the limited Gaussian budget to high-frequency detail regions in the image (e.g., object edges, textured surfaces) during the training process. This allows the model to maintain high rendering sharpness while keeping the overall noise level low, resolving the contradiction between noise reduction and edge preservation that is often mutually exclusive in traditional methods.

[0114] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0115] Another embodiment of this application proposes a saliency-guided 3D Gaussian reconstruction system. The details of this saliency-guided 3D Gaussian reconstruction system are described below. The following implementation details are provided for ease of understanding and are not essential for implementing this example. Figure 8 This is a schematic diagram of the structure of a saliency-guided 3D Gaussian reconstruction system proposed in this embodiment, including: Image input module 210 is used to acquire multi-view images of the scene to be reconstructed; The matching graph construction module 220 is used to perform retrieval matching on multi-view images and construct a globally connected image matching graph. The first-order motion reconstruction structure optimization module 230 is used to perform first-order motion reconstruction structure optimization based on the image matching map to obtain initial camera parameters and sparse 3D structure. Gaussian initialization module 240 is used to initialize a 3D Gaussian scene representation based on initial camera parameters and sparse 3D structure; The saliency evaluation module 250 is used to calculate the Gaussian saliency score and Gaussian redundancy score based on the multi-view rendering error during the 3D Gaussian training process. Gaussian allocation module 260 is used to perform Gaussian generation, Gaussian relocation, and Gaussian capacity redistribution based on Gaussian saliency score and Gaussian redundancy score; The joint optimization module 270 is used to jointly optimize the camera pose based on photometric loss and geometric reprojection loss while optimizing Gaussian parameters. Output module 280 is used to output the trained target 3D Gaussian scene representation and optimized camera parameters.

[0116] It is not difficult to see that this embodiment is a system embodiment corresponding to the above method embodiments, and this embodiment can be implemented in conjunction with the above method embodiments. The relevant technical details and technical effects mentioned in the above method embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above method embodiments.

[0117] It is worth mentioning that all modules and units involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed in this application; however, this does not mean that other units do not exist in this embodiment.

[0118] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A saliency-guided three-dimensional Gaussian reconstruction method, characterized in that, The method includes: Obtain multi-view images of the scene to be reconstructed; Perform retrieval-based matching on multi-view images to construct a globally connected image matching graph; First-order motion recovery structure optimization is performed based on image matching graphs to obtain initial camera parameters and sparse 3D structure; Initialize the 3D Gaussian scene representation based on the initial camera parameters and sparse 3D structure; During the 3D Gaussian training process, Gaussian saliency score and Gaussian redundancy score are calculated based on multi-view rendering error, and Gaussian generation, Gaussian relocation and Gaussian capacity redistribution are performed based on Gaussian saliency score and Gaussian redundancy score. While optimizing the Gaussian parameters, the camera pose is jointly optimized based on photometric loss and geometric reprojection loss, resulting in the trained target 3D Gaussian scene representation and optimized camera parameters.

2. The method according to claim 1, characterized in that, The step of performing retrieval-based matching on multi-view images to construct a globally connected image matching graph includes: Local features are extracted from each image, and image-level global descriptors are generated based on a Gaussian mixture model. An approximate nearest neighbor retrieval method is adopted, which retrieves similar images for each image based on image-level global descriptors and constructs a sparse candidate matching graph; A maximum spanning tree is constructed based on the similarity weights of edges in the candidate matching graph to supplement cross-cluster connection edges; Perform robust geometric verification based on polar geometry constraints on candidate image pairs to eliminate incorrect matches; The retrieved edges that pass geometric verification are merged with the supplementary edges of the maximum spanning tree to obtain the final image matching graph.

3. The method according to claim 1, characterized in that, The first-order motion recovery structure optimization based on the image matching map is performed to obtain the initial camera parameters and sparse 3D structure, including: Estimation of camera intrinsic parameters and distortion parameters based on image matching maps; The global rotation is solved based on the relative rotation constraints between pairs of images, and the global translation is solved based on the consistency of the translation direction. Construct a quadratic form expression corresponding to the polar geometry constraints, and perform a structure-independent first-order polar geometry adjustment on the camera parameters so that the optimization cost is weakly dependent on the number of 3D points; Sparse 3D points are recovered based on optimized camera parameters, and cross-view feature trajectories are constructed. Output the initial camera parameters, camera intrinsic parameters, sparse 3D points, and corresponding feature trajectory information.

4. The method according to claim 1, characterized in that, During the 3D Gaussian training process, Gaussian saliency scores and Gaussian redundancy scores are calculated based on multi-view rendering errors. Based on these scores, Gaussian generation, Gaussian relocation, and Gaussian capacity reallocation are performed, including: During training, multiple viewpoints are periodically sampled to render the current 3D Gaussian scene representation; Calculate the pixel-level error between the rendered image and the real image at each sampling viewpoint; The pixel-level errors under each sampling viewpoint are normalized by quantiles to obtain a robust error response map, thereby enhancing consistency across viewpoints and across training phases. Based on the two-dimensional projection footprints of each Gaussian under the sampling perspective, the high-error responses are aggregated to obtain the Gaussian significance score; Based on the two-dimensional projection footprints of each Gaussian under the sampling viewpoint, the low-error responses are aggregated to obtain the Gaussian redundancy score. Based on the aggregation results of Gaussian significance score and Gaussian redundancy score, Gaussian generation, Gaussian relocation, and Gaussian capacity redistribution are performed.

5. The method according to claim 4, characterized in that, The aggregation results based on Gaussian saliency scores and Gaussian redundancy scores are used to perform Gaussian generation, Gaussian relocation, and Gaussian capacity redistribution, including: Construct a significance-guided sampling distribution based on the Gaussian significance score, and sample a parent Gaussian from the significance-guided sampling distribution to generate a new Gaussian; and / or, Low-value Gaussians are selected based on Gaussian redundancy score and transparency, and then relocated to a highly significant region; where low-value Gaussians are those with redundancy scores higher than a preset value.

6. The method according to claim 4, characterized in that, The step of performing quantile normalization on the pixel-level errors under each sampling viewpoint to obtain a robust error response map specifically includes: ； in, This represents the robust error response diagram after quantile normalization. Indicates the first Pixel-level error map from each viewpoint and These represent the lower quantile and the upper quantile, respectively. Gaussian significance score The calculation formulas include: ； in, Indicates the first Gauss in the th... Two-dimensional projected footprint area from each viewpoint Represents pixels High error weighting; The formula for calculating the Gaussian redundancy fraction includes: ； in, Represents pixels Low error weights.

7. The method according to claim 1, characterized in that, While optimizing the Gaussian parameters, the camera pose is jointly optimized based on photometric loss and geometric reprojection loss to obtain the trained target 3D Gaussian scene representation and optimized camera parameters, including: Establish a photometric loss function; whereby the photometric loss function is used to constrain the difference between the rendered image and the real image; Based on the sparse feature trajectories corresponding to the sparse 3D structure, a geometric reprojection loss function is established; the geometric reprojection loss function is used to constrain the camera pose. The total loss function is obtained by weighted summation of the photometric loss function, the geometric reprojection loss function, and the Gaussian regularization term. The camera pose is optimized based on the total loss function, and the Gaussian parameters and camera pose correction are updated simultaneously during training to output the optimized target 3D Gaussian scene representation, camera pose parameters, and new viewpoint rendering results.

8. The method according to claim 7, characterized in that, During the training process, the method further includes: The canonical degrees of freedom are eliminated by fixing at least one reference camera.

9. The method according to claim 7, characterized in that, The photometric loss function The expression is: ； in, For real images, To render the image, The balance coefficients for structural similarity terms; The expression for the geometric reprojection loss function is: ； in, Represents two-dimensional observation geometry; Indicates the first Each camera pose; For triangulated 3D points; For the corresponding two-dimensional observation; For projection functions; The expression for the total loss function is: ； in, For the geometric reprojection loss weights, These are regularization terms related to Gaussian scaling, transparency, and stability.

10. A saliency-guided three-dimensional Gaussian reconstruction system, characterized in that, The system includes: The image input module is used to acquire multi-view images of the scene to be reconstructed; The matching graph construction module is used to perform retrieval-based matching on multi-view images and construct a globally connected image matching graph. The first-order motion reconstruction structure optimization module is used to perform first-order motion reconstruction structure optimization based on the image matching map to obtain the initial camera parameters and sparse 3D structure. The Gaussian initialization module is used to initialize a 3D Gaussian scene representation based on initial camera parameters and a sparse 3D structure. The saliency evaluation module is used to calculate the Gaussian saliency score and Gaussian redundancy score based on the multi-view rendering error during the 3D Gaussian training process. The Gaussian allocation module is used to perform Gaussian generation, Gaussian relocation, and Gaussian capacity redistribution based on Gaussian saliency scores and Gaussian redundancy scores. The joint optimization module is used to jointly optimize the camera pose based on photometric loss and geometric reprojection loss while optimizing Gaussian parameters. The output module is used to output the trained 3D Gaussian scene representation of the target and the optimized camera parameters.

Images