Multi-level fusion construction method, device, equipment and medium of 3DGS and mesh

Abstract

This application provides a method, apparatus, device, and medium for multi-level fusion construction of 3DGS and meshes, including: rendering multi-view depth maps based on 3DGS scene models; performing truncated signed distance function fusion processing on the multi-view depth maps to reconstruct a mesh model matching the 3DGS scene model; performing multi-level detail modeling on both the 3DGS scene model and the mesh model, and spatially dividing the 3DGS scene model and mesh model at each level based on a preset spatial partitioning strategy to form multiple hierarchical spatial sub-regions; and performing data structure construction processing on the multiple spatial sub-regions to generate a hybrid multi-level data structure. This approach retains the high-fidelity rendering capabilities of 3DGS while also possessing the geometric topology support capabilities of Mesh, solving the core pain point that existing single data structures cannot simultaneously handle high-fidelity rendering and physical simulation, and achieving integrated support for "rendering-interaction-simulation".

Description

Technical Field

[0001] This application relates to the fields of computer graphics and 3D scene modeling technology, and in particular to a method, apparatus, device and medium for multi-level fusion construction of 3DGS and mesh. Background Technology

[0002] 3D Gaussian Splatting (3DGS) is a novel implicit scene representation method based on point clouds. It describes the appearance of a scene by distributing a large number of 3D Gaussian spheres with attributes such as position, scale, orientation, color, and opacity in space. However, 3DGS also has significant drawbacks: on the one hand, its data format is a massive unordered set of points, which can easily lead to excessive memory consumption and low loading efficiency in ultra-large scenes; on the other hand, due to the lack of a clear surface topology, it cannot be directly used for collision detection or scene object manipulation in physics engines, limiting its application in interactive systems. In contrast, meshes, as a traditional explicit geometric representation, are composed of vertices, edges, and triangular faces. They can accurately describe the geometric boundaries and topological relationships of objects and are currently the most commonly used data foundation in physical simulation, CAD modeling, and game engines. However, their rendering quality is highly dependent on texture maps and normal details. Without high-quality material information, it is difficult to reproduce realistic visual textures, and high-density meshes can lead to a decrease in rendering performance, making it difficult to meet the requirements of high frame rate real-time rendering. Therefore, there is an urgent need to propose a new hybrid 3D data structure that can maintain the high realism rendering capabilities of 3DGS while integrating the geometric integrity and physical operability of Mesh. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a multi-level fusion construction method, device, equipment and medium involving 3DGS and mesh. Through the collaborative fusion of 3DGS and Mesh, it not only retains the high realism rendering capability of 3DGS, but also has the geometric topology support capability of Mesh. It solves the core pain point that the existing single data structure cannot take into account both high-fidelity rendering and physical simulation, and realizes integrated support of "rendering-interaction-simulation".

[0004] This application provides a multi-level fusion construction method for 3DGS and mesh, the multi-level fusion construction method including:

[0005] Obtain a 3DGS scene model and render a multi-view depth map based on the 3DGS scene model;

[0006] The multi-view depth map is subjected to truncated symbolic distance function fusion processing to reconstruct a mesh model that matches the 3DGS scene model;

[0007] Multi-level detail modeling is performed on the 3DGS scene model and the mesh model respectively. Based on a preset spatial partitioning strategy, the 3DGS scene model and the mesh model at each level are spatially partitioned to form multiple hierarchical spatial sub-regions.

[0008] Data structure construction processing is performed on multiple spatial sub-regions to generate a hybrid multi-level data structure.

[0009] In one possible implementation, the step of performing truncated signed distance function fusion processing on the multi-view depth map to reconstruct a mesh model matching the 3DGS scene model includes:

[0010] The multi-view depth map and its camera pose are voxelized in a unified world coordinate system, and the truncation symbol distance value of each voxel is calculated.

[0011] The zero-level set extraction process is performed on the truncation symbol distance values ​​of the voxels based on the isosurface extraction algorithm to generate the mesh model.

[0012] In one possible implementation, for the 3DGS scene model, multi-level detail modeling is performed through the following steps:

[0013] Based on the mesh model, the effective spatial region of the scene is determined. Within the effective spatial region, the three-dimensional Gaussian spheres in the 3DGS scene model are randomly downsampled to obtain the sparse Gaussian sphere set corresponding to the target level.

[0014] The position, scale, and orientation attributes of the sparse Gaussian sphere are fixed, and only the color and opacity attributes of the sparse Gaussian sphere are iteratively optimized;

[0015] Using the multi-view depth map rendered by the 3DGS scene model from multiple perspectives as a visual reference benchmark, the visual error between the rendering output of the optimized model at the same perspective and the benchmark is minimized until convergence, generating 3DGS data for that level. The logic is iterated repeatedly until all levels are constructed.

[0016] In one possible implementation, the 3DGS scene model and the mesh model at each level are spatially divided based on a preset spatial partitioning strategy to form multiple hierarchical spatial sub-regions, including:

[0017] Based on the spatial partitioning strategy, spatial partitioning operations are performed on the 3DGS scene model at each level to form a set of spatial sub-regions at that level, and the 3DGS spatial partitioning of all levels is completed layer by layer.

[0018] The mesh models at the same level are clipped and allocated according to the spatial sub-region boundaries obtained by dividing the corresponding 3DGS scene model, so that each spatial sub-region contains aligned 3DGS data and mesh data.

[0019] In one possible implementation, the spatial partitioning operation performed on the 3DGS scene model at each level based on the spatial partitioning strategy to form a set of spatial sub-regions at that level includes:

[0020] For the lowest level, the 3DGS scene model is used as the initial spatial unit. The longest axis direction of its bounding box is calculated, and the space is divided into two sub-regions along the longest axis. If the number of 3DGS points in either sub-region exceeds the preset maximum node point threshold, the sub-region is recursively divided until all sub-regions meet the point number constraint.

[0021] For higher-level layers, each spatial sub-region in the previous level is used as the initial division unit, inheriting its boundary range. It is determined whether the number of 3DGS points inside it exceeds the upper limit of the current level. If so, it is divided again along the longest axis of the previous level to generate the spatial sub-region of that level.

[0022] In one possible implementation, the data structure construction process for the plurality of spatial sub-regions to generate a hybrid multi-level data structure includes:

[0023] Each spatial sub-region is instantiated as a Node, and the entity information of the spatial sub-region is filled into the data field of the Node; wherein, the Node includes index information, spatial information, 3DGS data index, mesh data index, and child node information;

[0024] Construct a top-level Tree node to store metadata about the 3DGS scene model and manage all levels of Node nodes;

[0025] The Nodes at each level are organized into a tree-like hierarchical structure to generate the hybrid multi-level data structure.

[0026] In one possible implementation, after instantiating each spatial sub-region as a Node, the multi-level fusion construction method further includes:

[0027] For the 3DGS data at each level, traverse the adjacent Nodes at the same level. If the sum of the number of points in the 3DGS data of two adjacent Nodes is lower than the preset merging threshold, then merge the 3DGS data of the two adjacent Nodes and store them as a single data block. The merged 3DGS data is arranged and compressed using a spatially ordered Gaussian format.

[0028] For each level of mesh data, a corresponding boundary volume hierarchy tree is independently constructed, and the boundary volume hierarchy tree is associated with the mesh data index in the corresponding Node and stored together; wherein, the boundary volume hierarchy tree forms a hierarchical node structure with a spatial bounding box by recursively dividing the set of triangular facets of the mesh data of that level.

[0029] This application embodiment also provides a multi-level fusion construction device for 3DGS and mesh, the multi-level fusion construction device comprising:

[0030] The rendering module is used to acquire a 3DGS scene model and render a multi-view depth map based on the 3DGS scene model.

[0031] The mesh model construction module is used to perform truncation symbolic distance function fusion processing on the multi-view depth map to reconstruct a mesh model that matches the 3DGS scene model.

[0032] The hierarchical partitioning module is used to perform multi-level detailed modeling on the 3DGS scene model and the mesh model respectively. Based on the preset spatial partitioning strategy, the 3DGS scene model and the mesh model under each level are spatially partitioned to form multiple hierarchical spatial sub-regions.

[0033] The generation module is used to perform data structure construction processing on multiple spatial sub-regions to generate a hybrid multi-level data structure.

[0034] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the multi-level fusion construction method of 3DGS and mesh described above are performed.

[0035] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the multi-level fusion construction method of 3DGS and mesh as described above.

[0036] This application provides a method, apparatus, device, and medium for multi-level fusion construction of 3DGS and mesh. The multi-level fusion construction method includes: acquiring a 3DGS scene model; rendering a multi-view depth map based on the 3DGS scene model; performing truncated signed distance function fusion processing on the multi-view depth map to reconstruct a mesh model matching the 3DGS scene model; performing multi-level detail modeling on both the 3DGS scene model and the mesh model; dividing the 3DGS scene model and the mesh model at each level into multiple hierarchical spatial sub-regions based on a preset spatial partitioning strategy; and performing data structure construction processing on the multiple spatial sub-regions to generate a hybrid multi-level data structure. Through the collaborative fusion of 3DGS and Mesh, both the high-fidelity rendering capability of 3DGS and the geometric topology support capability of Mesh are retained, solving the core pain point that existing single data structures cannot simultaneously handle high-fidelity rendering and physical simulation, and achieving integrated support for "rendering-interaction-simulation".

[0037] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating a multi-level fusion construction method of 3DGS and mesh provided in an embodiment of this application;

[0040] Figure 2 One of the structural schematic diagrams of a multi-level fusion construction device for 3DGS and mesh provided in an embodiment of this application;

[0041] Figure 3 A second schematic diagram of a multi-level fusion construction device for 3DGS and mesh provided in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0044] First, the applicable scenarios for this application are introduced. This application can be applied to the fields of computer graphics and 3D scene modeling technology.

[0045] Research has revealed that, to overcome the limitations of single-modality rendering, existing studies attempt to integrate implicit representations such as NeRF and point clouds with meshes through post-processing, for example, generating coarse meshes through volumetric rendering or depth map reconstruction. However, these methods typically suffer from low reconstruction accuracy, misalignment of heterogeneous data spaces, and a lack of multi-level collaborative management, failing to achieve efficient and unified support for rendering and simulation. Therefore, there is an urgent need to propose a novel hybrid 3D data structure that can maintain the high realism of 3DGS rendering while integrating the geometric integrity and physical manipulability of meshes.

[0046] Based on this, the embodiments of this application provide a multi-level fusion construction method of 3DGS and mesh. Through the collaborative fusion of 3DGS and Mesh, it not only retains the high realism rendering capability of 3DGS, but also has the geometric topology support capability of Mesh. It solves the core pain point that the existing single data structure cannot take into account both high-fidelity rendering and physical simulation, and realizes integrated support of "rendering-interaction-simulation".

[0047] Please see Figure 1 , Figure 1 This is a flowchart illustrating a multi-level fusion construction method for 3DGS and meshes provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the multi-level fusion construction method includes:

[0048] S101: Obtain a 3DGS scene model and render a multi-view depth map based on the 3DGS scene model.

[0049] It should be noted that 3DGS uses a series of 3D semi-transparent Gaussian spheres with attributes such as position, orientation, scale, translucency, and color deployed in 3D space as the basic representation unit of the scene. During the rendering process, for key viewpoints, the 3D Gaussian spheres are projected onto the 2D image plane through projection transformation to obtain 2D Gaussian ellipses. Then, by performing alpha blending and color overlay on all 2D Gaussian ellipses, a depth map that conforms to that viewpoint is finally rendered.

[0050] S102: Perform truncated symbolic distance function fusion processing on the multi-view depth map to reconstruct a mesh model that matches the 3DGS scene model.

[0051] In this step, the multi-view depth map is truncated and fused using a symbolic distance function to reconstruct a mesh model that matches the 3DGS scene model.

[0052] In one possible implementation, the step of performing truncated signed distance function fusion processing on the multi-view depth map to reconstruct a mesh model matching the 3DGS scene model includes:

[0053] The multi-view depth map and its camera pose are voxelized in a unified world coordinate system, and the truncation symbol distance value of each voxel is calculated. The truncation symbol distance value of the voxel is then extracted using an isosurface extraction algorithm to generate the mesh model.

[0054] Here, we first use the constructed 3DGS scene model to render depth maps of the scene from multiple different perspectives to ensure that the depth maps cover the complete geometric information of the scene. Then, we input the multi-view depth maps into the TSDFFusion framework, and through spatial voxelization, we calculate the truncation symbol distance value of each voxel, and then fuse them to generate a globally consistent mesh model. This approach can make full use of the high-fidelity geometric information of 3DGS to quickly generate a mesh model with matching accuracy.

[0055] The algorithm iterates through each cubic voxel unit, determines the TSDF symbol combination of its eight vertices, and looks up a table to determine the connection method of the triangular faces within that unit. Zero-intersection points are precisely located on the edges using linear interpolation, constructing a triangular mesh that approximates the real surface. The resulting mesh possesses closure, continuity, and high geometric fidelity, accurately reproducing the complex indoor and outdoor structures expressed by the original 3DGS.

[0056] S103: Perform multi-level detail modeling on the 3DGS scene model and the mesh model respectively, and divide the 3DGS scene model and the mesh model at each level into multiple hierarchical spatial sub-regions based on a preset spatial division strategy.

[0057] In this step, multi-level detailed modeling is performed on the 3DGS scene model and mesh model respectively. Using a preset spatial partitioning strategy, the 3DGS scene model and mesh model at each level are spatially partitioned to form multiple hierarchical spatial sub-regions.

[0058] In one possible implementation, for the 3DGS scene model, multi-level detail modeling is performed through the following steps:

[0059] A: Based on the mesh model, the effective spatial region of the scene is determined. Within the effective spatial region, the three-dimensional Gaussian spheres in the 3DGS scene model are randomly downsampled to obtain the sparse Gaussian sphere set corresponding to the target level.

[0060] Here, based on the reconstructed Mesh model, its bounding volume is extracted and the effective free space of the scene is determined to eliminate redundant distribution of Gaussian spheres inside cavities or objects; at the target level, according to the preset maximum node number threshold (e.g., 500,000 points), the Gaussian sphere set in the 3DGS scene model is randomly downsampled to obtain the sparse Gaussian sphere set corresponding to the target level.

[0061] B: Fix the position, scale, and orientation attributes of the sparse Gaussian sphere, and only iteratively optimize the color and opacity attributes of the sparse Gaussian sphere.

[0062] Here, the position, scale, and orientation parameters of the selected Gaussian sphere are kept constant, and backpropagation optimization driven by differentiable rendering is performed only on its color and opacity properties.

[0063] C: Using the multi-view depth map rendered by the 3DGS scene model from multiple perspectives as a visual reference benchmark, minimize the visual error between the rendering output of the optimized model from the same perspective and the benchmark until convergence, generate the 3DGS data of that level, and repeat the iterative construction logic until all levels are constructed.

[0064] Here, multi-view depth maps of the 3DGS scene model from multiple key perspectives are used as a reference benchmark. By minimizing the pixel-level difference between the optimized model's RGB output and the benchmark at the same perspective, visual details lost due to the reduction in the number of pixels are recovered until convergence, generating 3DGS data for that level. The above process is repeated to generate multiple levels from coarse to fine, with each level corresponding to a different Gaussian sphere density and visual accuracy.

[0065] It should be noted that for the mesh model, the modeling adopts the mature topology-preserving simplification algorithm, and preferably uses the Quadric Error Metric (QEM) method based on edge folding: setting the upper limit of the number of target triangles and the maximum allowable geometric deviation at each level; iteratively selecting the edges with the least impact on local curvature for folding operations, while updating vertex positions and adjacency relationships; strictly maintaining the connectivity and watertightness of the mesh during the simplification process to avoid holes or self-intersections; finally generating a set of multi-resolution mesh representations from coarse outlines to fine structures, corresponding to different levels of rendering and simulation requirements.

[0066] In one possible implementation, the 3DGS scene model and the mesh model at each level are spatially divided based on a preset spatial partitioning strategy to form multiple hierarchical spatial sub-regions, including:

[0067] a: Based on the spatial partitioning strategy, perform spatial partitioning operations on the 3DGS scene model at each level to form a set of spatial sub-regions at that level, and complete the 3DGS spatial partitioning of all levels layer by layer.

[0068] Here, a spatial partitioning strategy is used to perform spatial partitioning operations on the 3DGS scene model at each level, forming a set of spatial sub-regions at that level, and the 3DGS spatial partitioning of all levels is completed layer by layer.

[0069] It should be noted that the spatial partitioning strategy adopts the "longest axis recursive partitioning" strategy, that is, each partitioning is performed by bisecting along the longest dimension of the current spatial region's bounding box to ensure that the sub-region is close to a cube shape, thereby improving the efficiency of subsequent view frustum culling and collision detection.

[0070] In one possible implementation, the spatial partitioning operation performed on the 3DGS scene model at each level based on the spatial partitioning strategy to form a set of spatial sub-regions at that level includes:

[0071] (1): For the lowest level, the 3DGS scene model is used as the initial spatial unit. The longest axis direction of its bounding box is calculated, and the space is divided into two sub-regions along the longest axis. If the number of 3DGS points in any sub-region exceeds the preset maximum node point threshold, the sub-region is recursively divided until all sub-regions meet the point number constraint.

[0072] Here, the lowest level LoD0 is divided as follows: the entire scene is used as the initial spatial unit, its axis-aligned bounding box is calculated, and it is divided into two sub-regions along the longest axis. The number of 3DGS points contained in each sub-region is counted. If the number of points in any sub-region exceeds the preset maximum node point threshold (e.g., 500,000), the same division operation is recursively performed on that sub-region until all sub-regions meet the point number constraint, and the division stops, forming a set of spatial sub-regions at the LoD0 level.

[0073] (2): For the higher level, each spatial sub-region in the previous level is used as the initial division unit, inheriting its boundary range. It is determined whether the number of 3DGS points inside it exceeds the upper limit of the number of points in the current level. If so, it is divided again along the longest axis of the previous level to generate the spatial sub-region of the level.

[0074] Here, the higher-level LoD k (k>0) partitioning: the next higher level of LoD k-1 Each spatial sub-region in the process serves as the initial division unit for the current level, inheriting its spatial boundary range. It is determined whether the number of 3DGS points corresponding to the current LoD level within the initial division unit exceeds the upper limit of the number of points set for this level (which can be decreased layer by layer). If the limit is exceeded, it is divided again along the longest axis of its bounding box, and this process is repeated until all sub-regions meet the constraints, thus completing the spatial division of the current level.

[0075] b: The mesh models at the same level are clipped and allocated according to the spatial sub-region boundaries obtained by dividing the corresponding 3DGS scene model, so that each spatial sub-region contains aligned 3DGS data and mesh data.

[0076] Here, for each LoD level, the boundaries of all spatial sub-regions obtained by 3DGS partitioning under this level are used; correspondingly, all vertices and triangles located in a certain sub-region in the simplified Mesh model of this level are clipped out and associated with the corresponding Node node of the sub-region through spatial indexing; thus achieving accurate alignment of Mesh data and 3DGS data under the same spatial partitioning system, avoiding misalignment or omission.

[0077] This application realizes the synchronous spatial organization of 3DGS and Mesh at multiple levels and granularities, which not only ensures the consistency of heterogeneous data in spatial distribution, but also provides a solid foundation for the subsequent construction of Tree-Node hierarchical data structure and the realization of on-demand loading and dynamic detail switching.

[0078] This application employs a partitioning mechanism based on 3DGS spatial partitioning and passive mesh alignment to ensure strict correspondence and consistent boundaries between the 3DGS point cloud and mesh geometry within each spatial sub-region at each level. This avoids rendering flickering and collision penetration issues caused by data misalignment, improves system stability and user experience, provides a reliable data foundation for subsequent multi-module collaborative processing (such as joint culling and cross-modal queries), and guarantees the spatial consistency and collaboration of heterogeneous data.

[0079] S104: Perform data structure construction processing on multiple spatial sub-regions to generate a hybrid multi-level data structure.

[0080] In this step, data structure construction is performed on multiple spatial sub-regions to generate a hybrid multi-level data structure that collaboratively integrates 3DGS and Mesh.

[0081] In one possible implementation, the data structure construction process for the plurality of spatial sub-regions to generate a hybrid multi-level data structure includes:

[0082] I: Instantiate each spatial sub-region as a Node, and fill the entity information of the spatial sub-region into the data field of the Node; wherein, the Node includes index information, spatial information, 3DGS data index, mesh data index and child node information.

[0083] Here, for each spatially divided sub-region, the system maps it to an independent Node. Each Node corresponds to a local spatial range at a specific level and serves as a container for 3DGS point clouds and mesh models.

[0084] It should be noted that the Node is the core management unit of the data structure. Each Node corresponds to a spatial sub-region under a specific level and is the key carrier for realizing heterogeneous data collaboration and spatial association. Its specific composition is as follows: (1) Index information: marking the unique identifier (UID) of the Node, the LoD level number, the parent node index and the adjacent Node index, used to realize the fast location of the Node, hierarchical association and neighborhood retrieval, and ensure the efficiency of data scheduling; (2) Spatial information: including the boundary coordinates (minimum / maximum coordinate values) of the spatial sub-region corresponding to the Node and the coordinates of the spatial center point, used to support spatial division, collision detection and view frustum culling and other operations, and to clarify the spatial information of the Node. Coverage range; (3) 3DGS data index: records the storage path, data offset and data size of the 3DGS data associated with the Node under the corresponding level, and can quickly index the 3DGS Gaussian sphere set in the sub-region of the space; (4) Mesh data index: records the storage path, data offset and data size of the Mesh data associated with the Node under the corresponding level, and can quickly index the simplified Mesh model in the sub-region of the space; (5) Child node information: records the number of child nodes of the current Node, the unique identifier (UID) of the child node and the metadata of the child node (including simplified information such as the spatial boundary of the child node and the data volume), which is used to realize hierarchical data traversal and loading, and support dynamic adjustment of detail precision.

[0085] II: Construct a top-level Tree node to store metadata of the 3DGS scene model and manage all levels of Node nodes.

[0086] Here, after instantiating all Node nodes, the system constructs a top-level management structure Tree node, which serves as the entry point and control center for the entire hybrid multi-level data structure.

[0087] The Tree node contains the following metadata: Version number: identifies the compatibility version of the data structure, facilitating subsequent upgrades and migrations; Scene description information: includes semantic attributes such as name, creation time, author, and geographic coordinate system (e.g., WGS84, ENU); Total number of levels: indicates the number of detailed levels contained in the current data structure; Precision parameters for each level: includes configuration items such as the maximum 3DGS point threshold for each level, the upper limit of Mesh triangles, and the spatial partitioning strategy type; Root Node list: points to all initial Node nodes at the highest level, forming the starting point of the entire tree structure.

[0088] Here, through the Tree node, the application can quickly obtain overall scene information and dynamically select the appropriate LoD level and spatial region for loading and rendering based on the current viewpoint, hardware performance, or user interaction needs.

[0089] III: Organize the Node nodes under each level into a tree-like hierarchical structure to generate the hybrid multi-level data structure.

[0090] Here, all Nodes are organized into a multi-branch tree structure according to the "parent-child, neighborhood" relationship. In this structure, the upper-level Node is the parent node of two or more child Nodes below it. Nodes at the same level are connected horizontally through adjacency indexes. Vertical refinement chains are formed between different levels to support progressive detail loading.

[0091] This application introduces a Level-of-Detail (LoD) modeling mechanism and combines it with a spatial partitioning strategy based on the longest axis recursive segmentation to construct a tree-like hierarchical "Tree-Node" data structure. This structure can dynamically select the appropriate level of detail to load based on viewpoint distance, hardware performance, or network bandwidth, significantly reducing peak memory usage and I / O latency. It is suitable for lightweight deployment in large-scale complex scenarios such as city-level digital twins and metaverse spaces, and is particularly well-suited for application needs in resource-constrained environments such as mobile terminals and web applications.

[0092] In one possible implementation, after instantiating each spatial sub-region as a Node, the multi-level fusion construction method further includes:

[0093] i: For the 3DGS data under each level, traverse the adjacent Nodes under the same level. If the sum of the number of points of the 3DGS data of two adjacent Nodes is lower than the preset merging threshold, then merge the 3DGS data of the two adjacent Nodes and store them as a single data block. The merged 3DGS data is arranged and compressed using a spatially ordered Gaussian format.

[0094] Here, after completing the spatial partitioning of all levels, all Nodes under the same level are traversed; if the sum of the 3DGS points of adjacent Nodes is lower than the merging threshold, they are merged and stored. This optimization can effectively reduce data indexing and storage overhead and improve data loading efficiency; the merged 3DGS data is stored in SOG (Spatially Ordered Gaussians) format, further compressing the storage volume and reducing storage space occupation.

[0095] ii: For each level of mesh data, an independent boundary volume hierarchy tree is constructed, and the boundary volume hierarchy tree is associated with the mesh data index in the corresponding Node and stored together; wherein, the boundary volume hierarchy tree forms a hierarchical node structure with a spatial bounding box by recursively dividing the set of triangular facets of the mesh data of that level.

[0096] Here, mesh data (vertices, edges, triangles, and topological relationships) is stored in a lightweight format (such as PLY simplified format) to ensure data integrity and editability. At the same time, a separate BVHTree (boundary volume hierarchy tree) is constructed for each level of mesh data, dividing the mesh triangles into different boundary volume nodes according to spatial distribution, thus constructing a hierarchical collision detection structure. The storage of BVHTree can significantly accelerate the collision detection speed in physical simulation, reduce interactive computation overhead, and improve the real-time interactive performance of the scene.

[0097] It should be noted that the recursive subdivision is as follows: divide the triangular facets within the current node into two non-empty subsets, calculate the bounding boxes of the two subsets respectively, create left and right child nodes, and repeat this process until the termination condition is met (such as the number of triangular facets within the child node being less than or equal to a preset threshold, or reaching the maximum tree depth).

[0098] In this application, for 3DGS data, an adjacent node merging mechanism is introduced and spatially ordered Gaussian (SOG) format is used for compressed storage to reduce index overhead and cache miss rate, thereby improving the loading efficiency of the rendering pipeline. For Mesh data, a boundary volume hierarchy tree (BVHTree) is independently constructed for each LoD level and stored in strong association with the mesh index in the Node, which greatly accelerates the calculation tasks such as ray detection and object collision in the physics engine and meets the low latency requirements in real-time interactive scenarios.

[0099] This application provides a multi-level fusion construction method for 3DGS and mesh, comprising: acquiring a 3DGS scene model; rendering a multi-view depth map based on the 3DGS scene model; performing truncated signed distance function fusion processing on the multi-view depth map to reconstruct a mesh model matching the 3DGS scene model; performing multi-level detail modeling on the 3DGS scene model and the mesh model respectively; dividing the 3DGS scene model and the mesh model at each level into multiple hierarchical spatial sub-regions based on a preset spatial partitioning strategy; and performing data structure construction processing on the multiple spatial sub-regions to generate a hybrid multi-level data structure. Through the collaborative fusion of 3DGS and Mesh, both the high-fidelity rendering capability of 3DGS and the geometric topology support capability of Mesh are retained, solving the core pain point that existing single data structures cannot simultaneously handle high-fidelity rendering and physical simulation, and achieving integrated support for "rendering-interaction-simulation".

[0100] Please see Figure 2 , Figure 3 , Figure 2 One of the structural schematic diagrams of a multi-level fusion construction device for 3DGS and mesh provided in an embodiment of this application; Figure 3This is a second schematic diagram of a multi-level fusion construction device for 3DGS and mesh provided in an embodiment of this application. Figure 2 As shown, the multi-level fusion construction apparatus 200 of 3DGS and mesh includes:

[0101] Rendering module 210 is used to acquire a 3DGS scene model and render a multi-view depth map based on the 3DGS scene model.

[0102] The mesh model construction module 220 is used to perform truncated symbolic distance function fusion processing on the multi-view depth map to reconstruct a mesh model that matches the 3DGS scene model.

[0103] The hierarchical division module 230 is used to perform multi-level detailed modeling on the 3DGS scene model and the mesh model respectively, and to perform spatial division on the 3DGS scene model and the mesh model under each level based on a preset spatial division strategy to form a hierarchical multiple spatial sub-regions.

[0104] The generation module 240 is used to perform data structure construction processing on multiple spatial sub-regions to generate a hybrid multi-level data structure.

[0105] Furthermore, the mesh model construction module 220 is used to perform truncated signed distance function fusion processing on the multi-view depth map to reconstruct a mesh model that matches the 3DGS scene model:

[0106] The multi-view depth map and its camera pose are voxelized in a unified world coordinate system, and the truncation symbol distance value of each voxel is calculated.

[0107] The zero-level set extraction process is performed on the truncation symbol distance values ​​of the voxels based on the isosurface extraction algorithm to generate the mesh model.

[0108] Furthermore, the hierarchy partitioning module 230 is used to perform multi-level detail modeling for the 3DGS scene model through the following steps:

[0109] Based on the mesh model, the effective spatial region of the scene is determined. Within the effective spatial region, the three-dimensional Gaussian spheres in the 3DGS scene model are randomly downsampled to obtain the sparse Gaussian sphere set corresponding to the target level.

[0110] The position, scale, and orientation attributes of the sparse Gaussian sphere are fixed, and only the color and opacity attributes of the sparse Gaussian sphere are iteratively optimized;

[0111] Using the multi-view depth map rendered by the 3DGS scene model from multiple perspectives as a visual reference benchmark, the visual error between the rendering output of the optimized model at the same perspective and the benchmark is minimized until convergence, generating 3DGS data for that level. The logic is iterated repeatedly until all levels are constructed.

[0112] Furthermore, the hierarchical partitioning module 230 is used to spatially partition the 3DGS scene model and the mesh model under each level based on a preset spatial partitioning strategy, forming multiple hierarchical spatial sub-regions:

[0113] Based on the spatial partitioning strategy, spatial partitioning operations are performed on the 3DGS scene model at each level to form a set of spatial sub-regions at that level, and the 3DGS spatial partitioning of all levels is completed layer by layer.

[0114] The mesh models at the same level are clipped and allocated according to the spatial sub-region boundaries obtained by dividing the corresponding 3DGS scene model, so that each spatial sub-region contains aligned 3DGS data and mesh data.

[0115] Furthermore, the hierarchical partitioning module 230 is used to perform spatial partitioning operations on the 3DGS scene model under each level based on the spatial partitioning strategy, forming a set of spatial sub-regions under that level:

[0116] For the lowest level, the 3DGS scene model is used as the initial spatial unit. The longest axis direction of its bounding box is calculated, and the space is divided into two sub-regions along the longest axis. If the number of 3DGS points in either sub-region exceeds the preset maximum node point threshold, the sub-region is recursively divided until all sub-regions meet the point number constraint.

[0117] For higher-level layers, each spatial sub-region in the previous level is used as the initial division unit, inheriting its boundary range. It is determined whether the number of 3DGS points inside it exceeds the upper limit of the current level. If so, it is divided again along the longest axis of the previous level to generate the spatial sub-region of that level.

[0118] Furthermore, the generation module 240 is used to perform data structure construction processing on the multiple spatial sub-regions to generate a hybrid multi-level data structure:

[0119] Each spatial sub-region is instantiated as a Node, and the entity information of the spatial sub-region is filled into the data field of the Node; wherein, the Node includes index information, spatial information, 3DGS data index, mesh data index, and child node information;

[0120] Construct a top-level Tree node to store metadata about the 3DGS scene model and manage all levels of Node nodes;

[0121] The Nodes at each level are organized into a tree-like hierarchical structure to generate the hybrid multi-level data structure.

[0122] Furthermore, such as Figure 3 As shown, the multi-level fusion construction device 200 of 3DGS and mesh also includes an optimization module 250, which is used for:

[0123] For the 3DGS data at each level, traverse the adjacent Nodes at the same level. If the sum of the number of points in the 3DGS data of two adjacent Nodes is lower than the preset merging threshold, then merge the 3DGS data of the two adjacent Nodes and store them as a single data block. The merged 3DGS data is arranged and compressed using a spatially ordered Gaussian format.

[0124] For each level of mesh data, a corresponding boundary volume hierarchy tree is independently constructed, and the boundary volume hierarchy tree is associated with the mesh data index in the corresponding Node and stored together; wherein, the boundary volume hierarchy tree forms a hierarchical node structure with a spatial bounding box by recursively dividing the set of triangular facets of the mesh data of that level.

[0125] This application provides a multi-level fusion construction device for 3DGS and mesh, comprising: a rendering module for acquiring a 3DGS scene model and rendering a multi-view depth map based on the 3DGS scene model; a mesh model construction module for performing truncated signed distance function fusion processing on the multi-view depth map to reconstruct a mesh model matching the 3DGS scene model; a hierarchy partitioning module for performing multi-level detail modeling on the 3DGS scene model and the mesh model respectively, and performing spatial partitioning on the 3DGS scene model and the mesh model at each level based on a preset spatial partitioning strategy to form multiple hierarchical spatial sub-regions; and a generation module for performing data structure construction processing on the multiple spatial sub-regions to generate a hybrid multi-level data structure. Through the collaborative fusion of 3DGS and Mesh, both the high-fidelity rendering capability of 3DGS and the geometric topology support capability of Mesh are retained, solving the core pain point that existing single data structures cannot simultaneously handle high-fidelity rendering and physical simulation, and achieving integrated support for "rendering-interaction-simulation".

[0126] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0127] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the multi-level fusion construction method of 3DGS and mesh in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0128] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the multi-level fusion construction method of 3DGS and mesh in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0129] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0130] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0132] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0133] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0134] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-level fusion construction method for 3DGS and mesh, characterized in that, The multi-level fusion construction method includes: Obtain a 3DGS scene model and render a multi-view depth map based on the 3DGS scene model; The multi-view depth map is subjected to truncated symbolic distance function fusion processing to reconstruct a mesh model that matches the 3DGS scene model; Multi-level detail modeling is performed on the 3DGS scene model and the mesh model respectively. Based on a preset spatial partitioning strategy, the 3DGS scene model and the mesh model at each level are spatially partitioned to form multiple hierarchical spatial sub-regions. Data structure construction processing is performed on multiple spatial sub-regions to generate a hybrid multi-level data structure; For the aforementioned 3DGS scene model, multi-level detail modeling is performed through the following steps: Based on the mesh model, the effective spatial region of the scene is determined. Within the effective spatial region, the three-dimensional Gaussian spheres in the 3DGS scene model are randomly downsampled to obtain the sparse Gaussian sphere set corresponding to the target level. The position, scale, and orientation attributes of the sparse Gaussian sphere are fixed, and only the color and opacity attributes of the sparse Gaussian sphere are iteratively optimized; Using the multi-view depth map rendered by the 3DGS scene model from multiple perspectives as a visual reference benchmark, the visual error between the rendering output of the optimized model at the same perspective and the benchmark is minimized until convergence, generating 3DGS data for that level. The logic is iterated repeatedly until all levels are constructed.

2. The multi-level fusion construction method according to claim 1, characterized in that, The step of performing truncated signed distance function fusion processing on the multi-view depth map to reconstruct a mesh model that matches the 3DGS scene model includes: The multi-view depth map and its camera pose are voxelized in a unified world coordinate system, and the truncation symbol distance value of each voxel is calculated. The zero-level set extraction process is performed on the truncation symbol distance values ​​of the voxels based on the isosurface extraction algorithm to generate the mesh model.

3. The multi-level fusion construction method according to claim 1, characterized in that, The 3DGS scene model and mesh model at each level are spatially divided based on a preset spatial partitioning strategy, forming multiple hierarchical spatial sub-regions, including: Based on the spatial partitioning strategy, spatial partitioning operations are performed on the 3DGS scene model at each level to form a set of spatial sub-regions at that level, and the 3DGS spatial partitioning of all levels is completed layer by layer. The mesh models at the same level are clipped and allocated according to the spatial sub-region boundaries obtained by dividing the corresponding 3DGS scene model, so that each spatial sub-region contains aligned 3DGS data and mesh data.

4. The multi-level fusion construction method according to claim 3, characterized in that, The spatial partitioning strategy is used to perform spatial partitioning operations on the 3DGS scene model at each level, forming a set of spatial sub-regions at that level, including: For the lowest level, the 3DGS scene model is used as the initial spatial unit. The longest axis direction of its bounding box is calculated, and the space is divided into two sub-regions along the longest axis. If the number of 3DGS points in either sub-region exceeds the preset maximum node point threshold, the sub-region is recursively divided until all sub-regions meet the point number constraint. For higher-level layers, each spatial sub-region in the previous level is used as the initial division unit, inheriting its boundary range. It is determined whether the number of 3DGS points inside it exceeds the upper limit of the current level. If so, it is divided again along the longest axis of the previous level to generate the spatial sub-region of that level.

5. The multi-level fusion construction method according to claim 1, characterized in that, The data structure construction process for multiple spatial sub-regions to generate a hybrid multi-level data structure includes: Each spatial sub-region is instantiated as a Node, and the entity information of the spatial sub-region is filled into the data field of the Node; wherein, the Node includes index information, spatial information, 3DGS data index, mesh data index, and child node information; Construct a top-level Tree node to store metadata about the 3DGS scene model and manage all levels of Node nodes; The Nodes at each level are organized into a tree-like hierarchical structure to generate the hybrid multi-level data structure.

6. The multi-level fusion construction method according to claim 5, characterized in that, After instantiating each spatial sub-region as a Node, the multi-level fusion construction method further includes: For the 3DGS data at each level, traverse the adjacent Nodes at the same level. If the sum of the number of points in the 3DGS data of two adjacent Nodes is lower than the preset merging threshold, then merge the 3DGS data of the two adjacent Nodes and store them as a single data block. The merged 3DGS data is arranged and compressed using a spatially ordered Gaussian format. For each level of mesh data, a corresponding boundary volume hierarchy tree is independently constructed, and the boundary volume hierarchy tree is associated with the mesh data index in the corresponding Node and stored together; wherein, the boundary volume hierarchy tree forms a hierarchical node structure with a spatial bounding box by recursively dividing the set of triangular facets of the mesh data of that level.

7. A multi-level fusion construction device for 3DGS and mesh, characterized in that, The multi-level fusion construction device includes: The rendering module is used to acquire a 3DGS scene model and render a multi-view depth map based on the 3DGS scene model. The mesh model construction module is used to perform truncation symbolic distance function fusion processing on the multi-view depth map to reconstruct a mesh model that matches the 3DGS scene model. The hierarchical partitioning module is used to perform multi-level detailed modeling on the 3DGS scene model and the mesh model respectively. Based on the preset spatial partitioning strategy, the 3DGS scene model and the mesh model under each level are spatially partitioned to form multiple hierarchical spatial sub-regions. The generation module is used to perform data structure construction processing on multiple spatial sub-regions to generate a hybrid multi-level data structure. The hierarchy partitioning module is used to perform multi-level detail modeling for the 3DGS scene model through the following steps: Based on the mesh model, the effective spatial region of the scene is determined. Within the effective spatial region, the three-dimensional Gaussian spheres in the 3DGS scene model are randomly downsampled to obtain the sparse Gaussian sphere set corresponding to the target level. The position, scale, and orientation attributes of the sparse Gaussian sphere are fixed, and only the color and opacity attributes of the sparse Gaussian sphere are iteratively optimized; Using the multi-view depth map rendered by the 3DGS scene model from multiple perspectives as a visual reference benchmark, the visual error between the rendering output of the optimized model at the same perspective and the benchmark is minimized until convergence, generating 3DGS data for that level. The logic is iterated repeatedly until all levels are constructed.

8. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the multi-level fusion construction method of 3DGS and mesh as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the multi-level fusion construction method of 3DGS and mesh as described in any one of claims 1 to 6.

Images