Method, apparatus, equipment, and storage medium for 3D Gaussian scene reconstruction based on mesh tangent plane geometric constraints.

Abstract

This application provides a method, apparatus, device, and storage medium for reconstructing a 3D Gaussian scene based on the geometric constraints of a mesh tangent plane, relating to the field of computer vision technology. The method includes: acquiring an image sequence of a target scene and reconstructing the initial geometric topology of the target scene; constructing a local tangent plane coordinate system attached to a triangular mesh, and parameterizing the center of the 3D Gaussian primitives as coordinates in the local tangent plane coordinate system; rendering and iteratively optimizing the parameterized 3D Gaussian primitives to update their attributes; during the iterative optimization process, adaptively controlling the density of the 3D Gaussian primitives attached to the mesh surface based on the geometric curvature information of the triangular mesh; and generating a 3D scene model with a triangular mesh as the geometric skeleton and optimized 3D Gaussian primitives as the appearance representation. This application achieves high-fidelity reconstruction of both visual appearance and physical geometry, and improves the resolution capability of complex textures while maintaining real-time rendering speed.

Description

Technical Field

[0001] This application relates to the fields of computer vision and computer graphics technology, and in particular to a method, apparatus, device and storage medium for three-dimensional Gaussian scene reconstruction based on mesh tangent plane geometric constraints. Background Technology

[0002] With the development of virtual reality (VR) and digital twin technologies, how to quickly reconstruct high-quality 3D scenes from 2D images or videos has become a research hotspot in the field of computer vision and graphics. Existing technologies can be mainly divided into the following two categories.

[0003] The first category is explicit mesh-based reconstruction methods, such as Multi-View Stereo (MVS). These methods can generate triangular mesh models with well-defined geometric topology, providing a solid geometric foundation for subsequent applications such as physics simulation and collision detection. However, their drawback is that texture mapping struggles to handle complex lighting, high-frequency details, and semi-transparent objects, resulting in insufficient realism in the final rendering.

[0004] The second category is implicit reconstruction methods based on 3D Gaussian Splatting (3DGS). These methods fit the scene using a large number of discrete ellipsoidal primitives, offering advantages such as fast rendering speed and good visual effects. However, they are essentially unstructured collections of scattered points, lacking explicit geometric and topological constraints.

[0005] The main problem with existing technologies is that traditional 3DGS methods directly optimize the global coordinates (x, y, z) of Gaussian points. During the optimization process, Gaussian points are prone to detaching from the object surface, producing floaters artifacts. This results in fragmented and discontinuous reconstructed geometry, which cannot meet the geometric accuracy requirements of downstream tasks such as physical simulation or collision detection.

[0006] To address the aforementioned issues, existing improvements include: some works attempt to initialize Gaussian points on the mesh surface in order to improve geometric consistency through initial geometric constraints. However, during training, Gaussian points still drift without additional constraints; directly using a clip function to force Gaussian points back to the mesh surface leads to gradient truncation, preventing the network from learning effectively and thus affecting reconstruction quality. Summary of the Invention

[0007] This application provides a method, apparatus, device, and storage medium for 3D Gaussian scene reconstruction based on mesh tangent plane geometric constraints, aiming to solve the problems of poor geometric accuracy and easy generation of floating artifacts in existing 3D Gaussian reconstruction methods. By establishing a local tangent plane coordinate system based on mesh triangular faces, the Gaussian plane is strictly anchored to the geometric surface, achieving high-fidelity reconstruction of both visual appearance and physical geometry, and improving the resolution of complex textures while maintaining real-time rendering speed.

[0008] In a first aspect, this application provides a method for reconstructing a 3D Gaussian scene based on the geometric constraints of a mesh tangent plane, the method comprising: Acquire an image sequence of the target scene, and reconstruct an initial geometric topology of the target scene based on the image sequence, wherein the initial geometric topology includes a triangular mesh; A local tangent plane coordinate system is constructed based on the triangular mesh, and the center of the three-dimensional Gaussian primitive is parameterized as the coordinate in the local tangent plane coordinate system, so that the three-dimensional Gaussian primitive is anchored to the surface of the triangular mesh. Based on the image sequence and the triangular mesh, the parameterized 3D Gaussian primitive is rendered and iteratively optimized to update its properties; During the iterative optimization process, based on the geometric curvature information of the triangular mesh, adaptive density control is performed on the three-dimensional Gaussian primitives attached to the mesh surface to increase the density of Gaussian primitives in high curvature regions and decrease the density of Gaussian primitives in low curvature regions. Generate a 3D scene model with the triangular mesh as the geometric skeleton and the optimized 3D Gaussian primitives as the appearance representation.

[0009] In one possible design, the local tangent plane coordinate system is constructed based on each triangular facet in the triangular mesh; for any triangular facet, the center of the three-dimensional Gaussian primitive is defined by the two-dimensional centroid coordinates within the triangular facet and the offset along the normal direction of the facet.

[0010] In one possible design, the center of the three-dimensional Gaussian primitive... Calculated using the following formula: in, This is the reference vertex of the triangular facet. To define two basis vectors for the triangular plane, Let be the unit normal vector of the triangular facet. and The coordinates of the two-dimensional barycenter within the triangular plane. This is the offset of the Gaussian center along the normal direction.

[0011] In one possible design, the two-dimensional centroid coordinates Limited to Within the range; the offset of the Gaussian center along the normal direction The value range is limited to the dynamic range based on the radius of the triangular facet.

[0012] In one possible design, the method of reconstructing the initial geometric topology of the target scene based on the image sequence includes: reconstructing the zero-level set of the target scene using a neural signed distance field or multi-view geometry method, and extracting triangular meshes using a traveling cube algorithm; during the reconstruction of the zero-level set, a monocular geometric prior is introduced as a geometric consistency loss term for optimization, wherein the monocular geometric prior includes a depth map generated by a pre-trained monocular depth estimation model and a normal map generated by a pre-trained monocular normal estimation model.

[0013] In one possible design, adaptive density control is performed on the three-dimensional Gaussian primitives attached to the mesh surface based on the geometric curvature information of the triangular mesh, including: Calculate the included angle between the normals of adjacent triangular faces in the triangular mesh. When the included angle of the normals When the preset threshold is exceeded, the corresponding region is determined to be a high curvature region, and the three-dimensional Gaussian primitives attached to the triangular facets of that region are triggered to perform a splitting operation to increase the local density.

[0014] The condition for triggering a split operation is expressed by the following formula: in, A boolean value indicating whether a split operation is triggered. This is an indicator function that outputs true if the condition within the parentheses is true, and false otherwise. Represents a logical OR operation. The gradient of the loss function in the view space. For gradient threshold, Triangular facet curvature, Curvature threshold; when When true, it triggers a splitting operation on the 3D Gaussian primitives attached to the corresponding triangular facet to increase the local density.

[0015] In one possible design, during the iterative optimization of the 3D Gaussian primitive, a linear scheduling strategy is used to dynamically adjust the weights in the loss function. This linear scheduling strategy includes: In the early stages of training, the deep supervision loss is assigned a high weight to force the 3D Gaussian primitives to fit tightly onto the surface of the triangular mesh. In the later stages of training, the weights of the depth supervision loss are linearly reduced, allowing the 3D Gaussian primitives to deviate from the triangular mesh surface within a preset range to fit complex lighting effects.

[0016] Secondly, this application provides a three-dimensional Gaussian scene reconstruction device based on mesh tangent plane geometric constraints, the device comprising: A topology construction module is configured to acquire an image sequence of a target scene and reconstruct an initial geometric topology of the target scene based on the image sequence, the initial geometric topology including a triangular mesh; The primitive anchoring module is configured to construct a local tangent plane coordinate system attached to the triangular mesh, and parameterize the center of the three-dimensional Gaussian primitive to the coordinates in the local tangent plane coordinate system, so that the three-dimensional Gaussian primitive is anchored to the surface of the triangular mesh. The iterative optimization module is configured to render and iteratively optimize the parameterized 3D Gaussian primitives based on the image sequence and the triangular mesh, so as to update their properties; The density control module is configured to, during the iterative optimization process, perform adaptive density control on the three-dimensional Gaussian primitives attached to the mesh surface based on the geometric curvature information of the triangular mesh, so as to increase the density of Gaussian primitives in high curvature regions and decrease the density of Gaussian primitives in low curvature regions. The scene generation module is configured to generate a 3D scene model with the triangular mesh as the geometric skeleton and the optimized 3D Gaussian primitives as the appearance representation.

[0017] Thirdly, embodiments of this application provide an electronic device, including: at least one processor and a memory; the memory stores computer execution instructions; the at least one processor executes the computer execution instructions stored in the memory, causing the at least one processor to perform the three-dimensional Gaussian scene reconstruction method based on mesh tangent plane geometric constraints as described in the first aspect and various possible designs of the first aspect.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions. When a processor executes the computer-executable instructions, it implements the three-dimensional Gaussian scene reconstruction method based on mesh tangent plane geometric constraints as described in the first aspect and various possible designs of the first aspect.

[0019] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the three-dimensional Gaussian scene reconstruction method based on mesh tangent plane geometric constraints as described in the first aspect and various possible designs of the first aspect.

[0020] The method, apparatus, device, and storage medium for reconstructing 3D Gaussian scenes based on mesh tangent plane geometric constraints provided in this application have at least the following beneficial effects: 1) This application parameterizes the three-dimensional Gaussian primitives into tangent plane coordinates attached to the surface of the triangular mesh (i.e., using... (Parameterized approach), forcing all Gaussian primitives to remain flush with the mesh surface throughout the optimization process. Compared to traditional 3DGS directly optimizing global coordinates ( x , y , z This method eliminates the "floating objects" and "cloud" artifacts caused by Gaussian points detaching from the object surface, significantly improving the geometric integrity and continuity of the reconstructed scene, enabling it to meet the stringent geometric accuracy requirements of downstream tasks such as physical simulation and collision detection.

[0021] 2) This application constructs a "dual representation" framework, using an explicit mesh as the geometric skeleton and 3D Gaussian primitives as the appearance carrier, achieving efficient decoupling between rendering performance and geometric structure. Under the constraints of the tangent plane parameterization model, it allows for the fitting of complex surface textures (high-frequency textures) and non-Lambertian volume lighting effects using 3D Gaussian primitives attached to the mesh surface while maintaining the low-poly mesh as a physical proxy. Combined with the linear scheduling strategy used during training, the geometry is solidified first and then the details are released, achieving an optimal balance between rendering efficiency and geometric accuracy.

[0022] 3) This application introduces a curvature-aware mechanism in the adaptive density control process, which determines the local geometric complexity by calculating the angle between the normals of adjacent triangular facets. When the angle between the normals exceeds a preset threshold, Gaussian primitive splitting is forcibly triggered in high-curvature regions (such as sharp edges) to enhance detail representation; in low-curvature flat regions, splitting is automatically suppressed and redundant primitives are pruned. This strategy ensures that the density distribution of Gaussian primitives is highly matched with the geometric complexity of the scene, significantly reducing storage overhead and greatly improving rendering and transmission efficiency while ensuring reconstruction quality. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0024] Figure 1 A flowchart illustrating a 3D Gaussian scene reconstruction method based on mesh tangent plane geometric constraints provided in this application embodiment; Figure 2 A flowchart illustrating the specific implementation of a 3D Gaussian scene reconstruction method based on mesh tangent plane geometric constraints, provided in this application embodiment; Figure 3 This is a schematic diagram of geometric parameterization based on a mesh tangent plane, provided for an embodiment of this application. Figure 4 This is a schematic diagram of an adaptive density enhancement strategy provided in an embodiment of this application; Figure 5 A schematic diagram illustrating the reconstruction effect of the 3D Gaussian scene reconstruction method provided in this application embodiment in a real indoor scene; Figure 6 A structural diagram of a three-dimensional Gaussian scene reconstruction device based on mesh tangent plane geometric constraints provided in this application embodiment.

[0025] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concepts of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0027] The collection, storage, use, processing, transmission, provision, and disclosure of relevant data and information in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0029] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0030] This application provides a 3D Gaussian scene reconstruction method based on mesh tangent plane geometric constraints. This method constructs a 3D scene reconstruction and generation process based on "dual representation," combining the topological advantages of explicit meshes with the rendering advantages of 3D Gaussian Splatting (3DGS). By constructing a Gaussian distribution attached to the mesh surface, high-fidelity reconstruction under geometric constraints is achieved. Figure 1 As shown, the detailed process of this method is as follows: Based on the input real scene image / video and optional geometric consistency constraints, SDF reconstruction and zero level set extraction operations are performed. This operation can be implemented using NeuralSDF or Colmap methods to complete the construction of the scene's signed distance field and the accurate extraction of the scene's surface zero level set.

[0031] Based on the extracted zero-level set, a baseline mesh generation operation is performed to obtain the triangular mesh of the scene. This baseline mesh provides the topological structure of the scene, providing a stable topological baseline and geometric carrier for subsequent Gaussian geometric constraints.

[0032] For the generated reference mesh, a tangent plane space construction operation is performed, and for each triangular facet in the reference mesh, its corresponding tangent vector is calculated. tangent vector With normal Based on tangent vector tangent vector With normal Construct a local tangent plane coordinate system for each triangular facet to provide a local geometric space reference for the parameterized binding of Gaussians.

[0033] Based on the constructed local tangent plane coordinate system, a Gaussian parametric binding Anchor operation is performed to anchor the 3D Gaussian primitive to the surface of the corresponding triangular facet, achieving deep binding between the Gaussian primitive and the mesh topology. The Gaussian center P attached to each triangular facet is defined parametrically using the following formula: .

[0034] After completing Gaussian parameterization binding, differentiable rendering and STE backpropagation operations are performed. Differentiable rendering is performed based on the anchored 3D Gaussian primitives to obtain the rendered image. A loss function is constructed based on the rendered image and the input real scene image / video, and backpropagation is performed through the pass-through estimator (STE) to iteratively update the various parameters of the 3D Gaussian primitives.

[0035] During the aforementioned iterative update process, a curvature-aware adaptive densification operation is executed synchronously. This operation includes three core processing logics: Split, Clone, and Cull. Based on the geometric curvature of the baseline mesh and the rendering contribution of Gaussian primitives, the distribution density of Gaussian primitives in the scene is dynamically adjusted. The Gaussian primitives adjusted by the curvature-aware adaptive densification operation are then re-entered into the differentiable rendering and STE reverse optimization stages to complete the closed-loop iterative optimization cycle.

[0036] When the iterative optimization reaches the preset convergence threshold or the maximum number of iterations, the iteration loop terminates and the final high-fidelity 3D scene is output. This scene is a dual representation scene model that integrates explicit mesh geometric constraints and implicit 3D Gaussian appearance representation, and can be directly used for subsequent real-time rendering, physical simulation and digital twin scenarios.

[0037] The following section will elaborate on the implementation principle and effects of the proposed method by combining the data structure foundation of 3D representation with the specific implementation process of the 3D Gaussian scene reconstruction method based on mesh tangent plane geometric constraints.

[0038] Digital representation of 3D scenes is a core problem in computer vision and graphics. Common representation forms include implicit representations (such as neural signed distance fields SDF) and explicit representations (such as meshes and point clouds).

[0039] A mesh is an explicit geometric data structure, typically represented as... .in Indicating the scene vertex coordinates A mesh represents a set of triangular faces that connect vertices. Meshes can accurately describe the topology and surface manifold of a scene, providing a physical benchmark for texture mapping and geometric constraints.

[0040] 3D Gaussian is a discrete primitive used for rendering radiation fields, consisting of a center position, a covariance matrix (scaling and rotation), opacity, and spherical harmonic coefficients. The "dual representation" proposed in this embodiment aims to leverage the geometric stability of the mesh to constrain the degrees of freedom of the 3D Gaussian, addressing the geometric collapse problem in traditional 3DGS in areas with weak textures or limited viewpoints.

[0041] Specifically, such as Figure 2 As shown, the 3D Gaussian scene reconstruction method based on mesh tangent plane geometric constraints can be implemented through the following steps S10-S50.

[0042] S10: Obtain the image sequence of the target scene and reconstruct the initial geometric topology of the target scene based on the image sequence. The initial geometric topology includes a triangular mesh.

[0043] It should be noted that the image sequence of the target scene described in this article can be an image or video sequence of a real scene.

[0044] In some embodiments, to obtain the underlying topology (i.e., the initial geometric topology) for constraining Gaussians, a geometric prior based on Neural Signed Distance Field (Neural SDF) is first established. Based on the acquired image or video sequence of the real scene, the zero-level set of the scene is reconstructed using Neural SDF or traditional multi-view geometric methods (such as Colmap), and the triangular mesh is extracted using the Marching Cubes algorithm.

[0045] In some embodiments, monocular geometric priors are introduced to enhance the robustness of the initial geometry. Specifically, a pre-trained monocular depth and normal estimation model is used to generate a depth map of the observation view. With normal diagram This is incorporated as a geometric consistency loss term into the SDF optimization process.

[0046] After optimization, the extracted mesh is simplified and texture baked to form a baseline mesh with a clear structure and basic texture information. , serving as the geometric carrier for subsequent dual representations.

[0047] S20: Construct a local tangent plane coordinate system attached to the triangular mesh, and parameterize the center of the 3D Gaussian primitive into the coordinates in the local tangent plane coordinate system, so that the 3D Gaussian primitive is anchored to the surface of the triangular mesh.

[0048] In some embodiments, based on the reference mesh obtained in step S10 Instead of directly initializing the Gaussian in free space, a parametric Gaussian model attached to a local tangent plane is constructed to anchor the 3D Gaussian primitives to the surface of the triangular mesh. Specifically, for any triangular facet on the mesh... Construct its local tangent plane coordinate system. The Gaussian center of each triangular facet is attached to this facet. No longer based on global coordinates The definition is not based on the two-dimensional barycentric coordinates within the plane. Offset along the normal direction This is jointly determined. The specific transformation formula is as follows: in, This is the reference vertex of the triangular facet. To define two basis vectors for the triangular plane, Let be the unit normal vector of the triangular facet.

[0049] For easier understanding, please refer to Figure 3 , Figure 3 This visually demonstrates the parametric geometric principle of the tangent plane defined by the above formula. The reference point (derived from the centroid coordinates) Constrained within the triangular facet, the Gaussian center is offset by a distance along the normal direction. This geometric anchoring mechanism ensures that Gaussian primitives are always attached to the mesh topology surface, achieving a combination of physical constraints and differentiability optimization.

[0050] In some embodiments, to further ensure that the three-dimensional Gaussian primitives are anchored to the surface of the triangular mesh, the following in-plane constraints and normal constraints are set.

[0051] In-plane constraints: After centroid coordinate clipping and normalization, it is strictly limited to Within the range, ensure that the Gaussian center is always projected inside the triangle.

[0052] Normal constraint: This represents the tiny undulations of the Gaussian relative to the mesh surface, and its value range is dynamically limited by the face radius. This ensures that the Gaussian can both closely adhere to the mesh surface to maintain the overall geometry and possess small degrees of freedom in detail to fit high-frequency textures.

[0053] S30: Based on image sequences and triangular meshes, render and iteratively optimize the parameterized 3D Gaussian primitives to update their properties.

[0054] In step S30, based on the image sequence and triangular mesh, the parameterized 3D Gaussian primitives are rendered and iteratively optimized. Specifically, this involves: using differentiable rendering technology, projecting the current parameterized 3D Gaussian primitives onto a 2D image plane to generate a rendered image; calculating the pixel-level difference between the rendered image and the corresponding viewpoint image of the input image sequence to construct a photometric loss (e.g., L1 loss or L2 loss); simultaneously, for each viewpoint, a rendered depth map is obtained through Gaussian splashing technology, and a geometric prior depth map is obtained by rasterizing the triangular mesh or inferring using a monocular depth estimation model, calculating the difference between the two to construct a depth supervision loss; the photometric loss and the depth supervision loss are weighted and summed to form a total loss function; subsequently, the attributes of the 3D Gaussian primitives, including their opacity, covariance matrix, and spherical harmonic coefficients, are iteratively updated using the backpropagation algorithm to minimize the total loss function. In this optimization process, the triangular mesh serves as a geometric constraint carrier, anchoring the Gaussian primitives to the mesh surface through a tangent plane parameterization model, ensuring that the optimization update is always performed under geometric constraints, thereby improving the realism of the rendering while maintaining the accuracy and continuity of the reconstructed geometric structure.

[0055] S40: During the iterative optimization process, based on the geometric curvature information of the triangular mesh, adaptive density control is performed on the three-dimensional Gaussian primitives attached to the mesh surface to increase the density of Gaussian primitives in high curvature regions and decrease the density of Gaussian primitives in low curvature regions.

[0056] In this embodiment, a curvature-aware adaptive splitting mechanism is constructed in step S40. Unlike traditional 3DGS which only relies on gradient magnitude for splitting, this adaptive splitting mechanism introduces geometric curvature as an important trigger signal for densification.

[0057] Figure 4 The adaptive density control strategies under different geometric curvatures were compared. Case A is shown in a flat region (low curvature), where the algorithm adopts a preservation or merging strategy to reduce redundancy; Case B is shown in a sharp-edge region (high curvature), where the algorithm automatically triggers a splitting mechanism when a sudden change in the angle between the normals of adjacent patches is detected, generating a high-density Gaussian distribution at the geometric edge, thereby effectively improving the reconstruction quality of high-frequency details.

[0058] The specific process is as follows: Calculate the included angle between the normals of adjacent triangular faces in the mesh. .when When the threshold is exceeded, the region is determined to be a high curvature region or a sharp edge. At this time, regardless of the gradient strength, the Gaussian surface attached to the surface is forcibly triggered to perform a split operation to increase the local density.

[0059] in, A boolean value indicating whether a split operation is triggered. This is an indicator function that outputs true if the condition within the parentheses is true, and false otherwise. Represents a logical OR operation. The gradient of the loss function in the view space. For gradient threshold, Triangular facet curvature, This is the curvature threshold.

[0060] when When true, it triggers the splitting operation of the 3D Gaussian primitives attached to the corresponding triangular facet to increase local density. Combined with adaptive culling, it can periodically remove low-contribution or abnormally inflated noise points according to the opacity, scale and proportion of Gaussian in screen space. This effectively suppresses floating noise in the scene while enhancing high-frequency detail fitting and keeping the model size controllable.

[0061] In some embodiments, to balance the stability of the geometric structure and the richness of rendering details, this scheme designs a temporal linear scheduling strategy to dynamically adjust the weight allocation in the loss function. The training process is divided into two stages: Geometry solidification stage (initial stage): This stage assigns a high weight to depth loss while setting a low accumulation target. It forces Gaussian geometry to adhere tightly to the mesh surface to prevent geometry drift.

[0062] Detail release phase (late stage): As the number of training steps increases, the depth supervision weights are linearly reduced, the Accumulation objective is gradually increased, and depth weighting based on Accumulation is enabled. This allows the Gaussian to "offset" from the mesh within a certain range to fit complex non-Lambertian lighting effects. This "stabilize geometry first, then release details" strategy allows the model to naturally achieve a smooth transition from coarse geometry to fine rendering during training.

[0063] S50: Generate a 3D scene model with a triangular mesh as the geometric skeleton and optimized 3D Gaussian primitives as the appearance representation.

[0064] To verify the effectiveness of the method in this application, this embodiment was tested on a real indoor scene dataset. For example... Figure 5 As shown, the input data is a continuous video frame sequence (real indoor scene) captured by a monocular camera. After the mesh cutting plane parameterization and adaptive optimization process described in steps S10-S50 above, a high-fidelity 3D reconstructed scene is output. Figure 5 As can be seen, the reconstructed scene not only maintains the accuracy of the global structure (such as the flatness of walls and cabinets), but also exhibits extremely high geometric density and clarity in local details (such as high-frequency areas like sinks and faucets). This fully demonstrates that the curvature-aware adaptive densification strategy proposed in this application can effectively capture complex geometric details, while the mesh constraint mechanism successfully suppresses the floating artifacts common in traditional methods.

[0065] This application also provides a 3D Gaussian scene reconstruction device based on mesh tangent plane geometric constraints, used to implement the methods described in any of the above embodiments, such as... Figure 6 As shown, the 3D Gaussian scene reconstruction device based on mesh tangent plane geometric constraints includes: The topology construction module 601 is configured to acquire an image sequence of a target scene and reconstruct an initial geometric topology of the target scene based on the image sequence, wherein the initial geometric topology includes a triangular mesh. The primitive anchoring module 602 is configured to construct a local tangent plane coordinate system attached to the triangular mesh, and parameterize the center of the three-dimensional Gaussian primitive to the coordinates in the local tangent plane coordinate system, so that the three-dimensional Gaussian primitive is anchored to the surface of the triangular mesh. The iterative optimization module 603 is configured to render and iteratively optimize the parameterized three-dimensional Gaussian primitives based on the image sequence and the triangular mesh, so as to update their properties; The density control module 604 is configured to, during the iterative optimization process, perform adaptive density control on the three-dimensional Gaussian primitives attached to the mesh surface based on the geometric curvature information of the triangular mesh, so as to increase the density of Gaussian primitives in high curvature regions and decrease the density of Gaussian primitives in low curvature regions. The scene generation module 605 is configured to generate a 3D scene model with the triangular mesh as the geometric skeleton and the optimized 3D Gaussian primitives as the appearance representation.

[0066] This application provides an electronic device. The electronic device may include a processor and a memory, wherein the processor and the memory can communicate; exemplarily, the processor and the memory communicate via a communication bus.

[0067] The processor executes computer execution instructions stored in memory, causing the processor to perform the scheme in the above embodiments. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0068] The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0069] The electronic device provided in this application embodiment can be the terminal device described in the above embodiments.

[0070] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the three-dimensional Gaussian scene reconstruction method based on mesh tangent plane geometric constraints described in the above embodiments.

[0071] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solution of the three-dimensional Gaussian scene reconstruction method based on mesh tangent plane geometric constraints in the above embodiments.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0073] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.

[0074] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.

[0075] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.

[0076] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0077] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.

[0078] Buses can be Industry Standard Architecture (ISA) buses, Peripheral Component Interconnect (PCI) buses, or Extended Industry Standard Architecture (EISA) buses, etc. Buses can be categorized into address buses, data buses, control buses, etc.

[0079] The aforementioned storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0080] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. The processor and storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic control unit or main control device.

[0081] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for 3D Gaussian scene reconstruction based on mesh plane geometry constraints, characterized in that, The method includes: Acquire an image sequence of the target scene, and reconstruct an initial geometric topology of the target scene based on the image sequence, wherein the initial geometric topology includes a triangular mesh; A local tangent plane coordinate system is constructed based on the triangular mesh, and the center of the three-dimensional Gaussian primitive is parameterized as the coordinate in the local tangent plane coordinate system, so that the three-dimensional Gaussian primitive is anchored to the surface of the triangular mesh. Based on the image sequence and the triangular mesh, the parameterized 3D Gaussian primitive is rendered and iteratively optimized to update its properties; During the iterative optimization process, based on the geometric curvature information of the triangular mesh, adaptive density control is performed on the three-dimensional Gaussian primitives attached to the mesh surface to increase the density of Gaussian primitives in high curvature regions and decrease the density of Gaussian primitives in low curvature regions. Generate a 3D scene model with the triangular mesh as the geometric skeleton and the optimized 3D Gaussian primitives as the appearance representation.

2. The method of claim 1, wherein, The local tangent plane coordinate system is constructed based on each triangular facet in the triangular mesh; for any triangular facet, the center of the three-dimensional Gaussian primitive is defined by the two-dimensional centroid coordinates within the triangular facet and the offset along the normal direction of the facet.

3. The 3D Gaussian scene reconstruction method based on mesh tangent plane geometric constraints according to claim 2, characterized in that, the center of the three-dimensional Gaussian bin is calculated by the following equation: in, This is the reference vertex of the triangular facet. To define two basis vectors for the triangular plane, Let be the unit normal vector of the triangular facet. and The coordinates of the two-dimensional barycenter within the triangular plane. This is the offset of the Gaussian center along the normal direction.

4. The 3D Gaussian scene reconstruction method based on mesh tangent plane geometric constraints according to claim 3, characterized in that, Two-dimensional centroid coordinates Limited to Within the range; the offset of the Gaussian center along the normal direction The value range is limited to the dynamic range based on the radius of the triangular facet.

5. The method for reconstructing a 3D Gaussian scene based on mesh tangent plane geometric constraints according to claim 1, characterized in that, The method for reconstructing the initial geometric topology of the target scene based on the image sequence includes: reconstructing the zero-level set of the target scene using neural signed distance field or multi-view geometry methods, and extracting triangular meshes using the traveling cube algorithm; in the process of reconstructing the zero-level set, monocular geometric priors are introduced as geometric consistency loss terms for optimization, and the monocular geometric priors include the depth map generated by the pre-trained monocular depth estimation model and the normal map generated by the pre-trained monocular normal estimation model.

6. The method for reconstructing a 3D Gaussian scene based on mesh tangent plane geometric constraints according to claim 1, characterized in that, Based on the geometric curvature information of the triangular mesh, adaptive density control is performed on the three-dimensional Gaussian primitives attached to the mesh surface, including: Calculate the included angle between the normals of adjacent triangular faces in the triangular mesh. When the included angle of the normals When the preset threshold is exceeded, the corresponding region is determined to be a high curvature region, and the three-dimensional Gaussian primitives attached to the triangular facets of that region are triggered to perform a splitting operation to increase the local density. The condition for triggering a split operation is expressed by the following formula: in, A boolean value indicating whether a split operation is triggered. This is an indicator function that outputs true if the condition within the parentheses is true, and false otherwise. Represents a logical OR operation. The gradient of the loss function in the view space. For gradient threshold, Triangular facet curvature, Curvature threshold; when When true, it triggers a splitting operation on the 3D Gaussian primitives attached to the corresponding triangular facet to increase the local density.

7. The method for reconstructing a 3D Gaussian scene based on mesh tangent plane geometric constraints according to claim 1, characterized in that, During the iterative optimization of the three-dimensional Gaussian primitive, a linear scheduling strategy is used to dynamically adjust the weights in the loss function. The linear scheduling strategy includes: In the early stages of training, the deep supervision loss is assigned a high weight to force the 3D Gaussian primitives to fit tightly onto the surface of the triangular mesh. In the later stages of training, the weights of the depth supervision loss are linearly reduced, allowing the 3D Gaussian primitives to deviate from the triangular mesh surface within a preset range to fit complex lighting effects.

8. A three-dimensional Gaussian scene reconstruction device based on mesh tangent plane geometric constraints, characterized in that, The device includes: A topology construction module is configured to acquire an image sequence of a target scene and reconstruct an initial geometric topology of the target scene based on the image sequence, the initial geometric topology including a triangular mesh; The primitive anchoring module is configured to construct a local tangent plane coordinate system attached to the triangular mesh, and parameterize the center of the three-dimensional Gaussian primitive to the coordinates in the local tangent plane coordinate system, so that the three-dimensional Gaussian primitive is anchored to the surface of the triangular mesh. The iterative optimization module is configured to render and iteratively optimize the parameterized 3D Gaussian primitives based on the image sequence and the triangular mesh, so as to update their properties; The density control module is configured to, during the iterative optimization process, perform adaptive density control on the three-dimensional Gaussian primitives attached to the mesh surface based on the geometric curvature information of the triangular mesh, so as to increase the density of Gaussian primitives in high curvature regions and decrease the density of Gaussian primitives in low curvature regions. The scene generation module is configured to generate a 3D scene model with the triangular mesh as the geometric skeleton and the optimized 3D Gaussian primitives as the appearance representation.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes the computer execution instructions stored in the memory to implement the three-dimensional Gaussian scene reconstruction method based on mesh tangent plane geometric constraints as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the three-dimensional Gaussian scene reconstruction method based on mesh tangent plane geometric constraints as described in any one of claims 1-7.

Images