Three-dimensional gaussian modeling method with real physical scale and grid-driven editing

Abstract

The application discloses a three-dimensional Gaussian modeling method with real physical scale and grid-driven editing, and belongs to the technical field of three-dimensional reconstruction and computer vision. In view of the problem that the existing three-dimensional Gaussian splashing lacks structured geometric constraints and real physical scale, the method extracts an initial triangular mesh from a three-dimensional Gaussian field, constructs a vertex cost function based on camera distribution prior to perform adaptive mesh simplification; based on the simplified mesh, Gaussian primitives are generated by adaptive sampling, and normal consistency, surface projection, step opacity penalty and composite scale constraint are added as geometric regularization terms in the luminosity loss to perform optimization; by constructing a local tangent space for each patch and reparameterizing the Gaussian primitive as a local attribute, object extraction along the mesh boundary and cross-scene editing are realized; by seven degrees of freedom similarity transformation, the model is registered to a real physical coordinate system. The method can be applied to industrial detection, digital twin and high-precision three-dimensional content creation scenes.

Description

Technical Field

[0001] This invention relates to the fields of 3D reconstruction and computer vision technology. More specifically, this invention relates to a 3D Gaussian modeling method with realistic physical scale and mesh-driven editing. Background Technology

[0002] High-quality acquisition and detailed representation of complex 3D scenes have always been core research topics in photogrammetry and computer vision. In recent years, implicit continuous scene representation methods, represented by Neural Radiance Field (NeRF), have shown great potential in handling complex lighting effects and geometric details, greatly promoting the development of Novel View Synthesis (NVS) technology. However, NeRF-like methods generally face computational bottlenecks such as long training time and high inference overhead, limiting their real-time performance. Subsequently, 3D Gaussian Splatting (3DGS) proposed by Kerbl et al. introduced discrete anisotropic Gaussian primitives and efficient differentiable rasterization technology, achieving high-quality real-time rendering while maintaining high-fidelity rendering, and quickly becoming the mainstream paradigm for 3D reconstruction and representation.

[0003] While 3DGS delivers exceptional visual performance, it is fundamentally an unconstrained photometric optimization process, lacking explicit geometric modeling of the continuous surfaces beneath the scene. This leads to significant limitations in practical applications for native 3DGS. First, due to the lack of geometric priors, the optimization process often generates numerous redundant artifacts in space, some even completely detached from the surface. These unstructured discrete primitives disrupt the geometric continuity of the scene, resulting in disproportionate objects and spatial misalignment during cross-scene object migration or composite editing due to the lack of a true physical scale. Second, because 3DGS typically relies on Structure from Motion (SfM) algorithms to obtain camera pose, the reconstructed scene is essentially in a relative coordinate system lacking true physical units. This scale ambiguity severely restricts the in-depth application of visual models in scenarios requiring absolute size measurement, such as autonomous driving and industrial inspection.

[0004] To address these challenges, researchers have begun exploring the introduction of geometric constraints or the extraction of explicit meshes within the 3DGS framework. For example, SuGaR and 2DGS have investigated methods such as aligning Gaussian primitives to surfaces or employing 2D dimensionality reduction mechanisms to enhance surface consistency. However, these methods, which rely on implicit regularization or spatial compression, often result in the loss of high-frequency geometric details while improving geometric quality, making it difficult to maintain high-fidelity rendering quality. Furthermore, directly using high-density raw meshes to drive large-scale Gaussian primitives not only significantly increases computational overhead but also easily leads to noticeable visual artifacts or geometric tearing under non-rigid deformation due to insufficient coordination between primitives. How to achieve the leap from simple "visual reproduction" to "metric reconstruction" with absolute physical scale, while supporting robust interactive editing, remains a critical problem that urgently needs to be solved. Summary of the Invention

[0005] One object of the present invention is to provide a 3D Gaussian modeling method with realistic physical scale and mesh-driven editing, so as to at least solve the above-mentioned problems.

[0006] To achieve the objectives and other advantages of this invention, a 3D Gaussian modeling method with realistic physical scale and mesh-driven editing is provided, comprising the following steps: S1. Acquire multi-view images of the target scene, perform 3D Gaussian splash modeling on the target scene to obtain an initial 3D Gaussian field, and extract an initial triangular mesh from the initial 3D Gaussian field; calculate the observation weights of the mesh vertices based on the camera distribution prior, construct the vertex cost function in combination with the quadratic error metric, and perform adaptive simplification on the initial triangular mesh to obtain a simplified triangular mesh. S2. Based on a simplified triangular mesh, adaptive sampling is performed according to the multidimensional geometric features of each facet to generate Gaussian elements that initially fit the mesh surface. During the photometric iteration calculation of the Gaussian elements, normal consistency loss, surface projection loss, stepped opacity penalty loss, and composite scale constraint loss are added as geometric regularization constraints on the basis of the photometric rendering loss of 3D Gaussian splashing. These constitute the total loss function, which restricts the Gaussian elements from detaching from the mesh surface, thus obtaining a scene representation in which the mesh is bound to Gaussian. S3. Based on the binding relationship between the mesh and Gaussian, a local tangent space coordinate system is constructed for each facet in the simplified triangular mesh. The position and rotation parameters of the Gaussian elements are re-parameterized as local attributes under the local coordinate system of the bound facet, so that the Gaussian elements can move synchronously with the spatial transformation of the facet, realizing object extraction and cross-scene combination editing along the geometric boundary of the simplified triangular mesh. S4. Obtain point cloud data of the target scene and uniformly sample the simplified triangular mesh surface to obtain a geometric point set. Register the geometric point set and point cloud data by solving a seven-degree-of-freedom similarity transformation that includes scale factor, rotation matrix and translation vector. Align the scene representation of the mesh and Gaussian binding from the relative coordinate system to the real physical coordinate system, so that the model has a real physical scale and provides a unified physical scale benchmark for cross-scene editing.

[0007] Preferably, in S2, during the photometric iteration calculation of Gaussian primitives, when a Gaussian primitive needs to generate a sub-primitive to meet the compaction condition, the spatial position of the sub-primitive is calculated by centroid interpolation of the three vertices of the bound facet of the parent primitive; the sub-primitive automatically inherits the facet index corresponding to the parent primitive, ensuring that all newly added Gaussian primitives are always constrained within the surface range of the bound facet.

[0008] Preferably, in S1, the vertex cost function C ( v )for: , in, Q ( v ) is the quadratic error measure of vertex v, Φ( v , c i ) represents the position of vertex v in the camera's view. c i The visibility weight is α, which is the balance weight coefficient. N cam For the total number of training cameras, w i Contribute weighting coefficients to the viewpoint of the i-th training camera.

[0009] Preferably, in S2, during adaptive sampling, the sampling weight of the patch is... w j Determined in the following ways: , in, H area ( f j ) is a sheet f j area, H curv ( f j ) is a sheet f j The local curvature, H cam ( f j ) is a sheet fj Camera observation degree, λ area , λ curv , λ cam These are the weighting coefficients. f j ∈F, where F is the set of all faces in the simplified triangular mesh; The number of Gaussian elements within the patch is allocated according to the sampling weight, and Gaussian elements are generated inside the patch using the centroid coordinates. The initial rotation direction of the generated Gaussian elements is aligned with the patch normal, and the initial scale is set to a flat shape with tangential extension and normal compression.

[0010] Preferably, in S2, the normal consistency loss is used to minimize the cosine distance between the shortest axis of the Gaussian element and the normal vector of the bound surface, and the normal consistency loss function L... normal for: , in, N g This represents the total number of effective Gaussian elements participating in the geometric constraint calculation. n gi Let be the unit normal vector corresponding to the shortest axis of the i-th Gaussian element; n fi The unit normal vector of the facet bound to the i-th Gaussian element.

[0011] Surface projection loss is used to penalize the vertical displacement from the center of the Gaussian element to the plane containing the geometric center of the bound patch, thus limiting the Gaussian element from detaching from the mesh surface along the normal. The surface projection loss function L proj for: in, N g This represents the total number of effective Gaussian elements participating in the geometric constraint calculation. m i Let the coordinates be the center position coordinates of the i-th Gaussian element; C fi The coordinates of the geometric center of the patch bound to the i-th Gaussian element; n fi The unit normal vector of the facet bound to the i-th Gaussian element.

[0012] Preferably, in S2, the stepped opacity penalty applies a graded gradient penalty based on the Euclidean distance from the center of the Gaussian cell to the surface of the bound patch, and applies a double penalty to outlier Gaussian cells that exceed the threshold to eliminate floating artifacts. The stepped opacity penalty loss function L opacity for: in, tfar This is the near-field distance threshold; α i Let be the opacity value of the i-th Gaussian element; β 1 is the attenuation coefficient of the first-level penalty, used to control the rate at which the near-field region opacity penalty increases with distance; t vfar The far-field distance threshold; β 2 is the attenuation coefficient of the second-level penalty, used to control the rate at which the opacity penalty in the far-field outlier region increases with distance; γ is the far-field doubling penalty factor, used to amplify the penalty intensity of outlier Gaussian units that are severely deviated from the surface. d i Let be the Euclidean distance from the center of the i-th Gaussian element to the surface of the bound patch.

[0013] Preferably, in S2, the composite scale constraint is used to limit the anisotropy ratio of the Gaussian element normal and tangential scales, while simultaneously controlling the anisotropy ratio with surface distance. d i The decaying envelope function constrains the extent of the Gaussian element space, avoiding mesh penetration and depth conflicts; the composite scale constrains the loss function L. scale for: , in, s x,i , s y,i , s z,i These represent the scales of the Gaussian element along the three principal axes; t ratio The anisotropy ratio threshold; t scale ( d i () represents the distance from the Gaussian element to the mesh surface. d i The decaying envelope function, N g The total number of valid Gaussian elements participating in the geometric constraint calculation; s i Let be the three-dimensional scale vector of the i-th Gaussian element.

[0014] Preferably, in S3, the local tangent space coordinate system is constructed as follows: for the coordinate system formed by the vertex { v 1 , v 2 , v 3 Any facet formed by} f j Define its local rotation basis R t= [r1, r2, r3], where: , in, n f The normal vector of the surface; Gaussian elements are reparameterized as local properties of the bound patch's local coordinate system. m l , q l , s l}; when the geometric center of the bound facet m t or rotating base R t When changes occur, the global spatial position of the Gaussian element m The global rotation R is updated as follows: , in, R l This is the local rotation matrix of the Gaussian element relative to the local tangent space coordinate system of the bound surface, used to describe the rotational offset of the Gaussian element itself relative to the tangent plane of the surface.

[0015] Preferably, in step S4, before registration, coarse registration is performed on the geometric point set and point cloud data to obtain corresponding point pairs. Initial seven-degree-of-freedom similarity transformation parameters are then solved using singular value decomposition. Using these initial parameters as initial values, a scale-aware iterative nearest-point algorithm is employed to iteratively solve for the optimal seven-degree-of-freedom similarity transformation by minimizing the weighted Huber loss function. , Where s is the global scale factor, R is the rotation matrix, and t is the translation vector. p i For points in the geometric point set, q i For the corresponding points in the point cloud data, r () is the Huber kernel function, used to reduce the impact of outliers on the registration results and improve the robustness of the algorithm to environmental noise; N p The total number of corresponding point pairs to be registered; w pi Let be the weight coefficient of the corresponding point pair in the i-th group.

[0016] Preferably, in S3, object extraction along the simplified triangular mesh geometric boundary is achieved through a semantic guidance method: first, the input multi-view image is semantically annotated at the pixel level through a pre-trained semantic segmentation network; then, the pixel-level semantic labels are fused and mapped onto the facets of the simplified triangular mesh through multi-view projection to obtain mesh facets with semantic labels; the corresponding facets are selected according to the semantic labels of the target object, and the bound Gaussian units are automatically stripped along the facet boundaries, so that no adhesion is generated during the extraction process.

[0017] The present invention has at least the following beneficial effects: This invention proposes a Metric-GS method that deeply integrates explicit meshes with 3DGS. It simplifies and constructs a lightweight geometric skeleton through an adaptive mesh guided by camera distribution priors, and precisely fits the mesh using Gaussian elements constrained by multi-dimensional geometric feature initialization and multi-criteria surface regularization. A binding mechanism based on local tangent space is established to solve the background adhesion problem in cross-scene editing, and scale-aware multimodal registration restores the scene's true physical metrics. Experiments on standard datasets such as DTU and Mip-NeRF 360 show that the proposed method achieves an average chamfer distance of 0.69 mm, controls the physical scale error within 1.2%, maintains rendering quality comparable to native 3DGS, reduces the peak signal-to-noise ratio by an average of less than 0.4 dB, and maintains the real-time rendering frame rate.

[0018] Other advantages, objectives and features of the present invention will become apparent in part from the following description, and in part from those skilled in the art through study and practice of the invention. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall framework of a 3D Gaussian modeling method with realistic physical scale and mesh-driven editing according to an embodiment of the present invention; Figure 2 This is a qualitative comparison diagram of multi-scene rendering effects according to an embodiment of the present invention; Figure 3 This is a visual comparison image of mesh extraction quality according to an embodiment of the present invention; Figure 4 This is a visualization of the mesh-driven modeling process according to an embodiment of the present invention; Figure 5 This is a demonstration diagram of the physical scale measurement function of one embodiment of the present invention; Figure 6 This is a high-fidelity object extraction and boundary preservation effect diagram according to an embodiment of the present invention; Figure 7 This is a qualitative effect diagram of cross-scene object combination editing according to an embodiment of the present invention; Figure 8 This is a comparison diagram of the mesh simplification strategy of one embodiment of the present invention; Figure 9 This is a qualitative comparison diagram of loss function ablation experiments according to an embodiment of the present invention. Detailed Implementation

[0020] The present invention will now be described in further detail with reference to the embodiments and accompanying drawings, so that those skilled in the art can implement it based on the description.

[0021] It should be understood that terms such as “having,” “comprising,” and “including” as used herein do not exclude the presence or addition of one or more other elements or combinations thereof.

[0022] It should be noted that, unless otherwise specified, the experimental methods described in the following implementation plan are all conventional methods, and the reagents and materials described are all commercially available unless otherwise specified.

[0023] Example 1: To address the limitations of native 3DGS in geometric perception and physical scale, this invention proposes a Metric-GS method, such as... Figure 1 As shown, this paper aims to construct a 3D representation system with realistic physical scale and support for mesh-driven editing. Its core design concept is to use the geometry of an explicit mesh to constrain the originally unconstrained Gaussian field, thereby establishing a precise spatial correspondence between the radiation field and the geometric surface. The system first extracts and simplifies a lightweight simplified triangular mesh from the 3D Gaussian neural field, using it as the spatial anchor point for Gaussian primitives. To ensure consistency between appearance and geometry, a binding mechanism based on local tangent space and multi-criteria regularization constraints are introduced. Finally, a scale-aware registration strategy eliminates scale ambiguity in viewpoint synthesis, enabling the model to possess absolute physical measurement capabilities. This design not only maintains high-fidelity rendering quality but also provides a physical basis for mesh-driven editing.

[0024] Specifically, this embodiment provides a 3D Gaussian modeling method with realistic physical scale and mesh-driven editing, characterized by the following steps: Step S1: Geometric Skeleton Construction Acquire multi-view images of the target scene. Specifically, use a camera or depth camera to capture images of the target scene from multiple different perspectives, obtaining at least dozens to hundreds of images covering various regions of the scene. Based on the acquired multi-view images, obtain the camera pose parameters (including rotation matrices and translation vectors) of each image using the Structure from Motion (SfM) algorithm, and generate a sparse point cloud.

[0025] Based on the camera pose parameters and sparse point cloud, a 3D Gaussian Splatting (3DGS) model is performed on the target scene to obtain an initial 3D Gaussian field. The specific method for 3DGS modeling can refer to the standard 3DGS workflow: initialize a set of anisotropic 3D Gaussian primitives, each primitive containing learnable parameters such as center position, covariance matrix, opacity, and spherical harmonic coefficients; render the image using a differentiable rasterizer; and iteratively train the model with photometric rendering loss as the optimization objective until convergence.

[0026] The initial triangular mesh is extracted from the trained initial 3D Gaussian field. This embodiment preferably employs the Differentiable Marching Tetrahedra (DMTet) algorithm. Specifically, given the vertex set in the tetrahedral mesh optimized by 3DGS and its corresponding signed distance function (SDF) value, the intersection points of the isosurface and the tetrahedral edges are determined by linear interpolation, and the intersection points are connected to construct an explicit triangular mesh. This process can be performed end-to-end, thereby obtaining a high-fidelity initial triangular mesh.

[0027] After obtaining the initial triangular mesh, the observation weights of the mesh vertices are calculated based on the camera distribution prior. A vertex cost function is then constructed using the Quadratic Error Metrics (QEM) to adaptively simplify the initial triangular mesh. Specifically, for any vertex v in the mesh, its cost function is defined as: , Where Q(v) is the traditional quadratic error measure of vertex v, representing the geometric error caused by folding vertex v; Φ( v , c i ) represents the position of vertex v in the camera's view. c i The visibility weight is calculated based on the angle between the vertex normal and the view direction, as well as whether the vertex is within the view frustum. α is the balance weight coefficient, with a value range of 0.3-0.7, preferably α=0.5. N cam The total number of training cameras participating in scene reconstruction; w i This parameter contributes a weighting coefficient to the viewpoint of the i-th training camera. It is used to adjust the contribution ratio of different cameras to the importance of vertex observation. The default value is 1, but it can also be dynamically assigned based on the distance between the camera and the scene center, and the camera's observation coverage. The higher the visibility weight, the more camera views the vertex is observed, and the greater its contribution to the scene's appearance.

[0028] In each edge folding simplification operation, the cost function values ​​of the two vertices at both ends of the edge to be folded are calculated, and the vertex pairs with lower costs are folded first. Through this mechanism, the algorithm can perform more aggressive simplification of the background region (region with low visibility weight) while maintaining the total patch budget (e.g., compressing the total number of patches to 5%-20% of the original number of patches), and at the same time ensuring that the geometric details of the core observation region (Region of Interest, ROI, region with high visibility weight) are preserved with high fidelity. After adaptive simplification, a lightweight simplified triangular mesh Ms=(V,F) is obtained, where V is the vertex set and F is the patch set.

[0029] Step S2: Mesh-guided Gaussian initialization and constraint optimization Based on the simplified triangular mesh Ms=(V,F) obtained in step S1, adaptive sampling is performed according to the multidimensional geometric features of each facet to generate the initial Gaussian elements that fit the mesh surface.

[0030] Specifically, for any facet in the mesh f j ∈F, its sampling weight w j By area H area ( f j ), local curvature H curv ( f j and camera observation degree H cam ( f j Jointly decided: , Where, λ area , λ curv , λ cam The weighting coefficients are preferred, with a ratio of 0.4:0.4:0.2.

[0031] Based on the above sampling weights w j In the interval [n min , n max (Preferred n) min =1, n max =20) Number of samples for each patch in the dynamic mapping within the area n j The Gaussian elements are precisely generated within the triangular facets using the centroid coordinates. The rotation of all generated Gaussian elements is initialized to align with the facet normals, and the scale exhibits a flattened shape with tangential extension and controlled normals, thereby constructing an initial Gaussian field that tightly fits the mesh surface.

[0032] In the photometric iteration calculation of Gaussian elements, to prevent Gaussian elements from detaching from the geometric surface, the standard photometric rendering loss L in 3D Gaussian splashing is reduced. render Based on this, a geometric regularization constraint term is added as the total loss function: , Where S is the set of geometric constraint terms, λ k This represents the balancing weights for each item.

[0033] The geometric regularization constraints include at least the following four types: (1) Normal uniformity loss L normal By minimizing the cosine distance between the shortest axis square of the Gaussian element and the normal vector of the bound surface: , in, N g The total number of effective Gaussian elements participating in the geometric constraint calculation. n gi Let be the unit normal vector corresponding to the shortest axis of the i-th Gaussian element; n fi The unit normal vector of the facet bound to the i-th Gaussian element.

[0034] (2) Surface projection loss L proj By penalizing the vertical displacement of the Gaussian metacenter from the plane containing the geometric center of the bound patch, we ensure that the Gaussian metacenter is located near the surface: , in, N g The total number of effective Gaussian elements participating in the geometric constraint calculation. m i Let the coordinates be the center position coordinates of the i-th Gaussian element; C fi The coordinates of the geometric center of the patch bound to the i-th Gaussian element; n fi The unit normal vector of the facet bound to the i-th Gaussian element.

[0035] (3) Stepped opacity penalty loss L opacity A graded gradient penalty is applied based on the Euclidean distance from the Gaussian metacenter to the bound mesh surface: , in, t far This is the near-field distance threshold, i.e., the initial threshold for the first-level penalty. Its value is consistent with the scene coordinate system units, and is preferably 0.01-0.05 times the scene bounding box scale.α i Let be the opacity of the i-th Gaussian element. β 1 represents the attenuation coefficient for the first-level penalty, a preset positive real number used to control the distance between the Gaussian element and the surface. t far Then, the rate at which the opacity penalty increases with distance; t vfar This is the far-field distance threshold, i.e., the starting threshold for the second-level penalty. Its value is consistent with the scene coordinate system unit and satisfies the following conditions: t vfar > t far Preferably, the scene bounding box size is 0.08-0.15 times the scene bounding box size; β 2 represents the attenuation coefficient for the second-level penalty, a preset positive real number used to control the distance between the Gaussian element and the surface. t vfar Then, the rate at which the superimposed penalty term increases with distance; γ is the doubling penalty factor for severely deviating noise, preferably γ=2.0.

[0036] Gaussian elements that are significantly deviated from the mesh surface are quantized as follows: if the Euclidean distance from the center of the Gaussian element to the surface of the bound patch is greater than... t vfar The Gaussian element is then defined as an outlier artifact (i.e., severely outlier noise) that deviates significantly from the mesh surface. This mechanism applies to distances exceeding [a certain value]. t far The Gaussian element is subject to a first-order opacity decay penalty, and for distances exceeding [a certain value], [the following applies]. t vfar The Gaussian element is additionally superimposed with a second-level doubled penalty to forcibly eliminate outlier artifacts floating in space; while retaining t far The geometric undulations of the near-surface region are used to ensure the fitting accuracy of surface details.

[0037] (4) Composite scale constraint loss L scale : , in, s x,i , s y,i , s z,i These represent the scales of the Gaussian element along the three principal axes; t ratio The anisotropy ratio threshold is preferably set in the range of 0.1-0.3. It is used to limit the ratio of the minimum anisotropy scale of the Gaussian elements. When the ratio exceeds the threshold, a penalty is triggered, thereby suppressing the spherical expansion of the Gaussian elements and forcing them to fit the mesh surface in a sheet-like manner. t scale ( d i () represents the distance from the Gaussian element to the mesh surface. d i The decaying envelope function is used to dynamically constrain the spatial extent of primitives and prevent mesh penetration and depth conflicts. s i Let be the three-dimensional scale vector of the i-th Gaussian element.

[0038] Furthermore, it also includes an in-plane compaction mechanism. When a Gaussian element satisfies a compaction condition (such as a position gradient greater than a preset threshold or a large contribution to the view space), the position of the newly generated Gaussian element is... m new Through the vertices of the bound face v k Perform centroid interpolation calculation: And satisfy ,in, w k The weight coefficients are the centroid coordinates of the three vertices of the triangular facet.

[0039] This mechanism ensures that all newly added Gaussian elements automatically inherit the patch index of their parent nodes, strictly constraining them within the mesh surface range, thereby maintaining a high degree of surface consistency between the Gaussian field and the simplified mesh during dynamic optimization.

[0040] After the above optimization, a scene representation bound to Gaussian grids is obtained, that is, each Gaussian element establishes a spatial correspondence with a certain facet in the simplified triangular grid.

[0041] Step S3: Displaying geometric associations in the local coordinate system Based on the binding relationship between the mesh and Gaussian established in step S2, a local tangent space coordinate system is constructed for each triangular facet in the simplified triangular mesh.

[0042] Specifically, for the vertex { v 1 , v 2 , v 3 Any facet formed by} f j Define its local rotation basis R t = [r1, r2, r3], where r1 is the unit vector of the edge, r2 is the normal vector, and r3 is obtained by the cross product of the two: , in, n f is the normal vector of the surface.

[0043] Within this framework, Gaussian primitives are reparameterized as local properties in the bound patch's local coordinate system. m l , q l , s l}. When the geometric center of the bound patch m t or rotating base R t When changes occur, the global spatial position of the Gaussian element m The global rotation R is updated as follows: , in, R l This is the local rotation matrix of the Gaussian primitive (GMT) relative to the local tangent space coordinate system of the bound facet, used to describe the rotational offset of the GMT itself relative to the tangent plane of the facet. Through this explicit binding mechanism, the GMT moves synchronously with the spatial transformation of the facet. Simultaneously, this explicit binding allows users to perform lossless object extraction along precise geometric boundaries. Combined with a physical scale recovery strategy, separated objects can be seamlessly implanted into the target scene at their true physical dimensions, thereby eliminating the "scale barrier" in viewpoint compositing and achieving scene composition that conforms to real-world scale.

[0044] Specifically, users can select the set of mesh patches corresponding to the target object. The system automatically determines all Gaussian elements bound to that set of patches and completely separates the target object from the scene along the explicit geometric boundaries of the mesh, resulting in an independent object model without background adhesion. When an independent object model extracted from one scene is implanted into the target scene, the model's real physical scale is automatically aligned with the target scene's physical coordinate system, achieving cross-scene object combination that conforms to real physical scale.

[0045] Step S4: Alignment with Real-World Scene and Scale Restoration Acquire point cloud data of the target scene. The point cloud data is preferably a LiDAR point cloud or a structured light scan point cloud, which has known real physical coordinates.

[0046] The simplified triangular mesh surface obtained in step S1 is uniformly sampled to obtain a geometric point set. In this embodiment, the Poisson disk sampling algorithm is preferably used to generate uniformly distributed sampling points on the mesh surface, avoiding registration deviation caused by uneven sampling.

[0047] The geometric point set and point cloud data are registered by solving a seven-degree-of-freedom (7-DoF) similarity transformation that includes a scale factor s, a rotation matrix R, and a translation vector t. Specifically, a two-stage strategy, from coarse to fine, is employed. The first stage (coarse registration) involves manually selecting at least three pairs of corresponding feature points and using Singular Value Decomposition (SVD) to solve for the initial transformation matrix, providing reliable initial values ​​for subsequent optimization.

[0048] The second stage (fine registration): employing the scale-aware iterative nearest neighbor (ICP) algorithm, the optimal seven-degree-of-freedom similarity transformation is iteratively solved by minimizing the weighted Huber loss function. , Where s is the global scale factor, R is the rotation matrix, and t is the translation vector. p i For points in the geometric point set, q i ρ is the corresponding point in the point cloud data; ρ() is the Huber kernel function, used to reduce the impact of outliers on the registration results and improve the robustness of the algorithm to environmental noise. N p The total number of corresponding point pairs to be registered; w pi Let be the weight coefficient of the i-th pair of corresponding points. The weight... w i Dynamic allocation based on the consistency of the normals of point pairs: when the angle between the normal directions of point pairs is small, a higher weight is assigned, and when the angle between the normal directions is large, a lower weight is assigned, in order to enhance the robustness to environmental noise points.

[0049] The optimal scale factor s, rotation matrix R, and translation vector t obtained from the solution are mapped back to the geometric model to correct the covariance scale between the mesh vertex positions and Gaussian elements. This aligns the scene representation bound to the mesh and Gaussian elements from the relative coordinate system to the real physical coordinate system. After this step, the model has a real physical scale, and the physical scale error can be controlled within 1.2%. It also provides a unified physical scale benchmark for cross-scene editing.

[0050] Application Example 1: 1. Experimental setup 1.1 Dataset and Evaluation Metrics To comprehensively verify the performance of Metric-GS in rendering fidelity and geometric accuracy, this invention evaluates the quality of novel perspective synthesis (NVS) on the Mip-NeRF360, Tanks & Temples, and DeepBlending datasets, using PSNR, SSIM, and LPIPS as evaluation metrics. The evaluation of 3D surface reconstruction accuracy is performed on the DTU dataset, using Chamfer Distance (CD) to quantitatively measure the deviation between the reconstructed model and the real physical surface. To quantify the impact of geometric constraints on the Gaussian field space structure in the ablation experiments, three local metrics are introduced: Masked Mean Angular Error (MSD), which calculates the angle between the effective primitive and the normal of the facet using opacity weighting to assess consistency; Mean Surface Distance (MSD), which measures the tightness of the fit by the average projection distance from the primitive center to the facet; and Outlier Rate (OR), which quantifies the proportion of primitives with projection distances exceeding a physical threshold.

[0051] 1.2 Unified Scale Evaluation Benchmark To address the scale ambiguity prevalent in visual reconstruction, this invention establishes a rigorous evaluation benchmark: all comparison methods do not use the officially provided ground truth poses, but instead uniformly reconstruct based on relative poses generated by COLMAP. After reconstruction, the 7-DoF similarity transformation algorithm proposed in Example 1, Real Scene Alignment and Scale Restoration, is uniformly applied, combined with global scale factors, rotation and translation parameters, and real LiDAR point clouds for spatial registration. This strategy ensures that all geometric comparisons are performed at the same absolute physical scale, thus objectively reflecting the accuracy level of each method in real scene reconstruction tasks.

[0052] 1.3 Experimental Parameter Settings The algorithm presented in this paper is implemented in PyTorch and runs on an NVIDIA GeForce RTX 3090. During the geometry simplification stage, the foreground and background retention rates are set to 0.2 and 0.05, respectively, achieving an overall compression ratio of approximately 11x. Gaussian initialization employs adaptive sampling based on geometric features, with weights determined by patch area, local curvature, and viewpoint visibility (in a ratio of 0.4:0.4:0.2). The number of training iterations is set to 30,000, and the core geometric constraint weights are set to... λnormal =0.1、λ proj =0.5.

[0053] 2. Comparative Analysis 2.1 New Perspective on Compositing and Rendering Quality Table 1 shows the performance comparison between Metric-GS and baseline methods on the novel perspective synthesis task. Experimental results show that among all methods aimed at extracting explicit geometric representations, our proposed method leads in key metrics such as PSNR, SSIM, and LPIPS. Compared to native 3DGS, Metric-GS successfully achieves a trade-off between visual fidelity and geometric accuracy by introducing structured geometric guidance. Table 1 Comparison of rendering performance of different methods on public datasets Note: Bold text indicates the optimal value, and underlined text indicates the suboptimal value.

[0054] Combined with actual rendering results, Metric-GS demonstrates superior perceptual quality when handling complex scenes. For example, in outdoor scenes, the method of this invention effectively suppresses overfitting noise common in photometric optimization by performing adaptive simplification on the background region and imposing strict surface alignment constraints. Although this restriction on degrees of freedom leads to a slight decrease in PSNR in some scenes, the method of this invention comprehensively surpasses benchmarks such as native 3DGS and 2DGS in metrics reflecting structural integrity, such as SSIM and LPIPS. This proves that the method of this invention can better maintain the geometric structural features of the scene while eliminating non-physical artifacts.

[0055] Figure 2 The qualitative comparison further confirms the quantitative analysis above. It is clearly observed that native 3DGS and 2DGS produce significant floating artifacts in areas such as object edges and under seats, severely interfering with the purity of the rendering. In contrast, Metric-GS, with its stepped opacity penalty mechanism, achieves precise pruning of deviating surface primitives. In representing fine interior structures (such as in the Bonsai scene), SuGaR often suffers from the loss of fine structures due to excessive smoothing regularization, while the proposed method successfully restores high-frequency geometric details through adaptive sampling based on geometric features.

[0056] 2.2 Three-dimensional surface reconstruction and geometric accuracy Before formally conducting geometric accuracy assessment, the reliability of the unified scale assessment benchmark established in section 1.2 was first verified. Considering the difficulty in directly obtaining hardware calibration poses in practical applications, this invention conducted cross-experiments to compare the physical scale recovered from relative poses using the algorithm of this invention with the results obtained directly using official ground truth poses. Experiments showed that the geometric indices obtained by the two methods exhibited a high degree of consistency with minimal differences. This not only strongly confirms that the 7-DoF registration algorithm of this invention can accurately eliminate scale ambiguity, but also proves that the process is equivalent to the results of using high-precision hardware calibration in practice, thus laying a rigorous foundation for subsequent fair measurements under a unified physical scale.

[0057] Based on the aforementioned reliable evaluation benchmarks, Table 2 presents a comparison of the global geometric accuracy of the method of this invention with that of various baseline methods in typical scenarios on the DTU dataset. Quantitative analysis results show that Metric-GS demonstrates superior overall measurement accuracy, with its average chamfer distance significantly outperforming methods such as SuGaR and 2DGS, which rely on strong implicit regularization. Although the method of this invention performs aggressive patch simplification on the background region during optimization, its final reconstruction accuracy remains at a leading level thanks to the adaptive resource tilting strategy in the foreground core region. This result intuitively reflects that Metric-GS can effectively overcome the spatial drift problem that traditional Gaussian representations are prone to when fitting complex surfaces through structured geometric guidance.

[0058] Table 2. Quantitative Evaluation Table of Geometric Reconstruction Accuracy of DTU Dataset Note: Bold text indicates the optimal value.

[0059] Figure 3 A qualitative comparison of the geometric mesh quality extracted by various methods is provided, further revealing the geometric features behind the quantitative indicators. It can be seen that while 2DGS and the initial MILo retain some scene details, they generate extremely large and redundant high-frequency noise in non-interest areas due to their lack of effective background noise reduction mechanisms. In contrast, SuGaR often results in the loss of delicate structures such as bicycle wheel hubs due to excessive smoothing constraints. Metric-GS, while significantly compressing the overall mesh size, successfully eliminates outlier artifacts and restores the geometric integrity of objects with high fidelity by leveraging dynamic ROI partitioning guided by camera distribution priors.

[0060] To more clearly demonstrate the complete process from geometric acquisition to visual representation, in Figure 4The rendering results of the initial high-density mesh, the adaptive simplified mesh, and the final driven mesh were compared. Observations confirm that the simplification mechanism proposed in this invention can still perfectly maintain the geometric features of the object while significantly reducing the total number of vertices. Thanks to the budget allocation strategy that tilts towards the core observation region, the simplified lightweight mesh accurately preserves key geometric features such as edges and corners. The rightmost rendering view further demonstrates that, supported by a highly compressed geometric skeleton, the model can still maintain extremely high photometric fidelity. This fully demonstrates that Metric-GS has successfully found the optimal balance between lightweight geometric representation and high-quality visual rendering, achieving a leap from visual reproduction to reconstruction with physical scale metrics.

[0061] 2.3 Model Measurement and Editing To further validate the application potential of Metric-GS in real-world scenarios, this section evaluates the model's physical measurement accuracy and interactive editing capabilities. Compared to traditional methods that focus solely on visual reconstruction, this paper establishes a deep binding between geometric meshes and Gaussian units, enabling the reconstruction results to directly serve downstream tasks with strict requirements for spatial measurement and object boundaries.

[0062] In terms of physical scale measurement, thanks to the scale-aware registration strategy proposed in this invention, the reconstructed scene has been transformed into a real physical coordinate system. Based on this, this invention also developed a WebGL real-time analysis system, supporting users to directly perform point-to-point distance measurements and surface area calculations in the rendered view. Figure 5 As shown, experimental results demonstrate that the Gaussian elements and their underlying geometric mesh are highly aligned in space, and the measured values ​​accurately reflect the true physical dimensions of the object. This representation method with metric properties provides technical support for precise spatial analysis of 3D scenes.

[0063] In interactive object editing, Metric-GS demonstrates precise object extraction capabilities and cross-scene composition effects. Traditional 3DGS, due to a lack of clear geometric boundary constraints, often results in extracted objects with noticeable background artifacts at their edges. Metric-GS, however, achieves a strict association between Gaussian units and geometric surfaces through a binding mechanism based on local tangent space. For example... Figure 6 As shown, the target object can be completely peeled off along the explicit geometric boundaries, effectively solving the problem of edge fragmentation and background adhesion.

[0064] also, Figure 7The demonstration showcased the effect of cross-scene object recombination. By placing objects extracted from different source scenes into a new environment, it can be seen that the objects blend very naturally with the new background without any sense of incongruity. Thanks to the binding relationship between the mesh and Gaussian elements, when operations such as rotation or translation are performed on the objects, the Gaussian elements attached to the surface can synchronously maintain their original shape and structure. These experimental results demonstrate that Metric-GS has good practical value in constructing high-quality virtual scenes and creating 3D content tasks.

[0065] 3. Ablation test The following analysis, through a series of ablation experiments, examines the specific impact of each core component in Metric-GS on system performance.

[0066] First, regarding the adaptive simplification strategy guided by the camera distribution prior, Figure 8 The performance of the method of this invention was compared with that of the traditional uniform simplification algorithm. Experimental results show that, under the same patch budget, uniform simplification, due to its lack of viewpoint perception, causes severe geometric breaks in fine structures such as cables and wheel spokes at the bicycle handlebars, making it difficult to maintain the integrity of the overall shape of the object. In contrast, the method of this invention allocates more geometric resources to the core observation area, preserving fine geometric features while achieving a high compression ratio, effectively improving the geometric integrity of the subject.

[0067] Subsequently, the effects of various surface constraints on the final performance of the model were further analyzed. (See Table 3 and...) Figure 9 As shown, the regularization constraints are crucial for maintaining reconstruction accuracy. After removing the geometric binding constraints, the Gaussian normals exhibit severe random bias, resulting in noticeable black artifacts in the rendering, directly compromising the visual stability of local surfaces. Simultaneously, the absence of the stepped opacity penalty leads to a large number of floating objects detached from the surface, significantly increasing the outlier rate. This not only reduces image sharpness but also causes severe background adhesion during object extraction. Furthermore, removing the anisotropic scale constraint causes Gaussian units to tend to fit depth by expanding outwards, resulting in thick and blurred object boundaries due to volumetric effects.

[0068] Table 3. Results of Quantitative Ablation Experiments on Core Components Note: Bold text indicates the optimal value.

[0069] Based on the above ablation experiment results, only under a complete constraint system can Gaussian elements be accurately aligned to the geometric mesh surface. This synergistic effect not only ensures high-fidelity rendering results but also provides a reliable geometric basis for subsequent physical measurement and editing tasks.

[0070] The number of devices and processing scale described herein are for simplification of the invention. Applications, modifications, and variations of the present invention's 3D Gaussian modeling method with realistic physical scale and mesh-driven editing will be readily apparent to those skilled in the art.

[0071] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A 3D Gaussian modeling method with realistic physical scale and mesh-driven editing, characterized in that, Includes the following steps: S1. Acquire multi-view images of the target scene, perform 3D Gaussian splash modeling on the target scene to obtain an initial 3D Gaussian field, and extract an initial triangular mesh from the initial 3D Gaussian field; calculate the observation weights of the mesh vertices based on the camera distribution prior, construct the vertex cost function in combination with the quadratic error metric, and perform adaptive simplification on the initial triangular mesh to obtain a simplified triangular mesh. S2. Based on a simplified triangular mesh, adaptive sampling is performed according to the multidimensional geometric features of each facet to generate Gaussian elements that initially fit the mesh surface. During the photometric iteration calculation of the Gaussian elements, normal consistency loss, surface projection loss, stepped opacity penalty loss, and composite scale constraint loss are added as geometric regularization constraints on the basis of the photometric rendering loss of 3D Gaussian splashing. These constitute the total loss function, which restricts the Gaussian elements from detaching from the mesh surface, thus obtaining a scene representation in which the mesh is bound to Gaussian. S3. Based on the binding relationship between the mesh and Gaussian, a local tangent space coordinate system is constructed for each facet in the simplified triangular mesh. The position and rotation parameters of the Gaussian elements are re-parameterized as local attributes under the local coordinate system of the bound facet, so that the Gaussian elements can move synchronously with the spatial transformation of the facet, realizing object extraction and cross-scene combination editing along the geometric boundary of the simplified triangular mesh. S4. Obtain point cloud data of the target scene and uniformly sample the simplified triangular mesh surface to obtain a geometric point set. Register the geometric point set and point cloud data by solving a seven-degree-of-freedom similarity transformation that includes scale factor, rotation matrix and translation vector. Align the scene representation of the mesh and Gaussian binding from the relative coordinate system to the real physical coordinate system, so that the model has a real physical scale and provides a unified physical scale benchmark for cross-scene editing.

2. The 3D Gaussian modeling method with realistic physical scale and mesh-driven editing as described in claim 1, characterized in that, In S2, during the photometric iteration calculation of Gaussian primitives, when a Gaussian primitive needs to generate a sub-primitive to meet the compaction condition, the spatial position of the sub-primitive is calculated by centroid interpolation through the three vertices of the bound facet of the parent primitive; the sub-primitive automatically inherits the facet index corresponding to the parent primitive, ensuring that all newly added Gaussian primitives are always constrained within the surface range of the bound facet.

3. The 3D Gaussian modeling method with realistic physical scale and mesh-driven editing as described in claim 1, characterized in that, In S1, the vertex cost function C ( v )for: ， in, Q ( v ) is the quadratic error measure of vertex v, Φ( v , c i ) represents the position of vertex v in the camera's view. c i The visibility weight is α, which is the balance weight coefficient. N cam For the total number of training cameras, w i Contribute weighting coefficients to the viewpoint of the i-th training camera.

4. The 3D Gaussian modeling method with realistic physical scale and mesh-driven editing as described in claim 1, characterized in that, In S2, during adaptive sampling, the sampling weights of the patches... w j Determined in the following ways: ， in, H area ( f j ) is a sheet f j area, H curv ( f j ) is a sheet f j The local curvature, H cam ( f j ) is a sheet f j Camera observation degree, λ area , λ curv , λ cam These are the weighting coefficients. f j ∈F, where F is the set of all faces in the simplified triangular mesh; The number of Gaussian elements within the patch is allocated according to the sampling weight, and Gaussian elements are generated inside the patch using the centroid coordinates. The initial rotation direction of the generated Gaussian elements is aligned with the patch normal, and the initial scale is set to a flat shape with tangential extension and normal compression.

5. The 3D Gaussian modeling method with realistic physical scale and mesh-driven editing as described in claim 1, characterized in that, In S2, the normal consistency loss is used to minimize the cosine distance between the shortest axis of the Gaussian element and the normal vector of the bound surface. The normal consistency loss function L normal for: in, N g The total number of effective Gaussian elements participating in the geometric constraint calculation. n gi Let be the unit normal vector corresponding to the shortest axis of the i-th Gaussian element; n fi The unit normal vector of the patch bound to the i-th Gaussian element; Surface projection loss is used to penalize the vertical displacement from the center of the Gaussian element to the plane containing the geometric center of the bound patch, thus limiting the Gaussian element from detaching from the mesh surface along the normal. The surface projection loss function L proj for: in, N g This represents the total number of effective Gaussian elements participating in the geometric constraint calculation. μ i Let the coordinates be the center position coordinates of the i-th Gaussian element; C fi The coordinates of the geometric center of the patch bound to the i-th Gaussian element; n fi The unit normal vector of the facet bound to the i-th Gaussian element.

6. The 3D Gaussian modeling method with realistic physical scale and mesh-driven editing as described in claim 1, characterized in that, In S2, the stepped opacity penalty applies a graded gradient penalty based on the Euclidean distance from the center of the Gaussian cell to the surface of the bound patch. Outlier Gaussian cells exceeding a threshold are penalized with double the penalty to eliminate floating artifacts. The stepped opacity penalty loss function L... opacity for: in, τ far This is the near-field distance threshold; α i Let be the opacity value of the i-th Gaussian element; β 1 is the attenuation coefficient of the first-level penalty, used to control the rate at which the near-field region opacity penalty increases with distance; τ vfar The far-field distance threshold; β 2 is the attenuation coefficient of the second-level penalty, used to control the rate at which the opacity penalty in the far-field outlier region increases with distance; γ is the far-field doubling penalty factor, used to amplify the penalty intensity of outlier Gaussian units that are severely deviated from the surface. d i Let be the Euclidean distance from the center of the i-th Gaussian element to the surface of the bound patch.

7. The 3D Gaussian modeling method with realistic physical scale and mesh-driven editing as described in claim 1, characterized in that, In S2, composite scale constraints are used to limit the anisotropy ratio of the Gaussian element normal and tangential scales, while also varying with surface distance. d i The decaying envelope function constrains the extent of the Gaussian element space, avoiding mesh penetration and depth conflicts; the composite scale constrains the loss function L. scale for: ， in, s x,i , s y,i , s z,i These represent the scales of the Gaussian element along the three principal axes; τ ratio The anisotropy ratio threshold; τ scale ( d i () represents the distance from the Gaussian element to the mesh surface. d i The decaying envelope function, N g The total number of valid Gaussian elements participating in the geometric constraint calculation; s i Let be the three-dimensional scale vector of the i-th Gaussian element.

8. The 3D Gaussian modeling method with realistic physical scale and mesh-driven editing as described in claim 1, characterized in that, In S3, the local tangent space coordinate system is constructed as follows: for the coordinate system formed by the vertex { v 1 , v 2 , v 3 Any facet formed by} f j Define its local rotation basis R t = [r1, r2, r3], where: ， in, n f The normal vector of the surface; Gaussian elements are reparameterized as local properties of the bound patch's local coordinate system. μ l , q l , s l }; when the geometric center of the bound facet μ t or rotating base R t When changes occur, the global spatial position of the Gaussian element μ The global rotation R is updated as follows: ， in, R l This is the local rotation matrix of the Gaussian element relative to the local tangent space coordinate system of the bound surface, used to describe the rotational offset of the Gaussian element itself relative to the tangent plane of the surface.

9. The 3D Gaussian modeling method with realistic physical scale and mesh-driven editing as described in claim 1, characterized in that, In S4, a coarse registration step is included before registration: at least three pairs of non-collinear corresponding feature points are selected, and the initial rigid transformation matrix is ​​solved by combining singular value decomposition. The global scale factor is estimated by the distance ratio of corresponding points to obtain the initial seven-degree-of-freedom similarity transformation parameters. Using these initial parameters as initial values, the scale-aware iterative nearest-point algorithm is used to iteratively solve the optimal seven-degree-of-freedom similarity transformation by minimizing the weighted Huber loss function. ， Where s is the global scale factor, R is the rotation matrix, and t is the translation vector. p i For points in the geometric point set, q i These are the corresponding points in the point cloud data; ρ () is the Huber kernel function, used to reduce the impact of outliers on the registration results and improve the robustness of the algorithm to environmental noise; N p The total number of corresponding point pairs to be registered; w pi Let be the weight coefficient of the corresponding point pair in the i-th group.

10. The 3D Gaussian modeling method with realistic physical scale and mesh-driven editing as described in claim 1, characterized in that, In S3, object extraction along the simplified triangular mesh geometric boundary is achieved through semantic guidance: first, the input multi-view image is semantically annotated at the pixel level through a pre-trained semantic segmentation network; then, the pixel-level semantic labels are fused and mapped onto the facets of the simplified triangular mesh through multi-view projection to obtain mesh facets with semantic labels; the corresponding facets are selected according to the semantic labels of the target object, and the bound Gaussian units are automatically stripped along the facet boundaries, so that no adhesion is generated during the extraction process.

Images