3DGS reconstruction method and system based on unified instance coding

Abstract

The invention discloses a 3DGS reconstruction method and system based on unified instance coding, and the method comprises the steps: introducing instance ID embedding in a 3DGS frame, and adding a vector to each Gaussian sphere to represent an object to which the vector belongs; for the instance object obtained by segmentation, extracting a Gaussian ball, converting the Gaussian ball into an initial sparse point cloud with normal and color attributes, and performing point cloud completion by using a physical guide diffusion completion network; the physical guide diffusion completion network starts from a noise point cloud based on a diffusion model, and a differentiable physical guide item is superposed during denoising sampling; and bidirectional iteration grid extraction and optimization are carried out, primary grid extraction, physical collision detection and Gaussian correction and secondary grid extraction are carried out on the complemented Gaussian cloud, a high-precision non-penetration grid is reconstructed, and scene reconstruction is completed. According to the method, the reconstruction efficiency can be improved, the phenomena of physical law violation such as object position deviation and cross overlapping during reconstruction are avoided, and the authenticity of reconstruction is improved.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction, specifically to a 3DGS reconstruction method and system based on unified instance coding. Background Technology

[0002] In the field of computer vision, 3D scene reconstruction has always been a core research topic that has attracted much attention. Although the traditional Neural Radiation Field (NeRF) method can synthesize new perspective images with photorealistic quality from multi-view images, its reconstruction process is time-consuming and the output results are mostly implicit voxels or scenes, making it difficult to directly obtain explicit geometric structures, which greatly limits the subsequent application expansion.

[0003] To overcome these bottlenecks, Google proposed the 3D Gaussian Splatting (3DGS) technology, which uses Gaussian spherical clouds to represent the color and density distribution of a scene and leverages differentiable rendering to achieve efficient training and real-time rendering of new perspectives. However, this technology still has significant drawbacks: the original 3DGS model consists of a massive number of Gaussian points, and after optimization, these points are randomly distributed in space, making it impossible to directly extract the geometry of individual objects and 3D meshes; at the same time, it lacks the ability to understand scene semantics, making it difficult to meet the needs of interactive editing and downstream scene understanding. Although methods such as Gaussian Grouping and LangSplat attempt to endow 3DGS with scene semantic segmentation capabilities, these methods require additional parameters and loss terms during training, heavily rely on a large amount of multi-view 2D segmentation supervision information, have high optimization costs, and only support offline operation, making it impossible to separate individual objects in real time during reconstruction.

[0004] Even with preliminary scene representations and instance-level point clouds, the sparsity and incompleteness of point cloud data remain significant issues. Due to factors such as shooting angle and occlusion, data on the back and bottom of objects are often missing. While point cloud completion techniques such as GeoFormer and SuperPC can effectively restore the geometric details of point clouds, they generally neglect physical plausibility, and the completed models are prone to phenomena that violate physical laws, such as floating elements and overlapping parts.

[0005] Extracting meshes from Gaussian clouds also presents challenges. While methods such as SuGaR and GS2Mesh can quickly generate high-quality meshes, they all assume that the Gaussian cloud data is accurate and conforms to physical rules. This makes it difficult to resolve errors such as object interpenetration and overlap in the original scene reconstruction, and to handle geometric conflicts between objects in real-world scenes.

[0006] In terms of scene layout, traditional multi-view reconstruction often suffers from model interleaving due to object position deviations and sensor noise. While methods such as PhyScene, FirePlace, and Agentic 3D Scene Generation improve scene generation quality by introducing physical and semantic constraints, they mostly focus on new scene synthesis or object insertion and rely on offline optimization or large models, lacking automatic physical correction methods for reconstructing scenes from real data. Currently, the industry urgently needs a solution that integrates reconstruction, segmentation, completion, and object spatial rationality optimization to ensure the physical feasibility of object shapes and layouts in real time during the reconstruction process. Summary of the Invention

[0007] In view of the above problems, this application provides a 3DGS reconstruction method based on unified instance coding to solve the technical problems of low accuracy and easy overlap in the reconstruction of the above scenes, which violate the laws of physics.

[0008] To achieve the above objectives, this application provides a 3DGS reconstruction method based on unified instance coding, comprising the following steps:

[0009] Taking multi-view image sequences as input, instance ID embedding is introduced into the 3DGS framework, and a vector is attached to each Gaussian sphere to represent the object to which it belongs. By constraining the 2D-3D consistent clustering loss, the 2D instance segmentation results in the multi-view images are compared with the Gaussian embedding, which promotes the Gaussian sharing of similar features of the same object, realizes instance-level segmentation of objects in the scene, and outputs a Gaussian radiation field model with instance labels.

[0010] Physically guided object-level point cloud completion: For the segmented instance objects, Gaussian spheres are extracted and converted into initial sparse point clouds with normal and color attributes. The point cloud is then completed using a physically guided diffusion completion network. The physically guided diffusion completion network starts from noisy point clouds based on a diffusion model and superimposes differentiable physical guidance terms during denoising sampling, including penetration penalties and support stability constraints, to generate physically reasonable dense point clouds.

[0011] Bidirectional iterative mesh extraction and optimization: Perform the following operations on the completed Gaussian cloud:

[0012] Initial mesh extraction: An improved SuGaR algorithm is used to sample surface points by taking advantage of the Gaussian distribution and the alignment characteristics of the object surface to generate a coarse mesh;

[0013] Physics collision detection and Gaussian correction: Utilizes a GPU-accelerated AABB+BVH engine to detect mesh intersecting regions, locate conflicting Gaussian points, and adjust their radius, density, or center position to eliminate geometric conflicts;

[0014] Secondary mesh extraction: Using a volume fusion strategy similar to GS2Mesh, the updated Gaussian field is rendered as a multi-view depth / voxel, and a high-precision non-penetrating mesh is reconstructed to complete the scene reconstruction.

[0015] Furthermore, the 2D-3D consistent clustering loss is jointly optimized with pixel error during training.

[0016] Furthermore, the weighting coefficients of the penetration penalty and support stability constraint in the physical loss are λ1 and λ2, which are weighted and summed to form a joint physical guidance term.

[0017] Furthermore, the bidirectional iterative mesh extraction process iterates 1-2 times. The first iteration can eliminate more than 90% of the interlacing noise, and after the second iteration, the conflicts are basically eliminated by visual inspection.

[0018] Furthermore, it also includes physical feasible layout optimization based on scene hypergraph: objects are used as hypergraph nodes to construct a hypergraph structure that includes first-order support relationships, second-order orientational proximity relationships and higher-order semantic combination relationships. Differentiable graph neural networks are used with object poses and relationship tensors as inputs, and iterative optimization is performed through message passing to make the scene meet the constraints of support stability, collision-free and common sense layout.

[0019] Furthermore, it also includes unified GPU scheduling and real-time rendering: a multi-stream parallel CUDA graph is built based on the OpenAI Triton framework, and the tasks of each module are executed asynchronously in a pipeline to achieve a scene processing and update speed of ≥25 frames per second, supporting real-time editing and backfilling.

[0020] Furthermore, in the 3DGS framework, the initialization of Gaussian clouds is generated by transforming sparse point clouds output by SLAM, and the Gaussian distribution is optimized through adaptive density control.

[0021] To address the aforementioned technical problems, this application also provides another technical solution:

[0022] A 3DGS reconstruction system based on unified instance coding includes:

[0023] The 3DGS reconstruction module takes multi-view image sequences as input, introduces instance ID embedding into the 3DGS framework, and adds a vector to each Gaussian sphere to represent the object to which it belongs. Through 2D-3D consistent clustering loss constraint, it compares the 2D instance segmentation results in the multi-view images with the Gaussian embedding, prompting the Gaussians of the same object to share similar features, realizing instance-level segmentation of objects in the scene, and outputting a Gaussian radiation field model with instance labels.

[0024] The point cloud completion module is used to extract Gaussian spheres from the segmented instance objects and convert them into an initial sparse point cloud with normal and color attributes. The point cloud is then completed using a physically guided diffusion completion network. The physically guided diffusion completion network starts from the noisy point cloud based on the diffusion model and superimposes a differentiable physical guidance term during denoising sampling, including penetration penalty and support stability constraint, to generate a physically reasonable dense point cloud.

[0025] The mesh extraction and Gaussian refinement module is used to generate a coarse mesh by sampling surface points through the Gaussian distribution and the alignment characteristics of the object surface using the improved SuGaR algorithm; and to detect mesh intersecting regions using the GPU-accelerated AABB+BVH engine, locate conflicting Gaussian points and adjust their radius, density or center position to eliminate geometric conflicts.

[0026] The scene layout module is used for physically feasible layout optimization based on the scene hypergraph. It includes: taking objects as hypergraph nodes, constructing a hypergraph structure containing first-order support relationships, second-order orientational proximity relationships, and higher-order semantic combination relationships; using a differentiable graph neural network with object pose and relationship tensors as input, iteratively optimizing through message passing to ensure that the scene meets support stability, collision-free, and common-sense layout constraints; and using a volume fusion strategy similar to GS2Mesh to render the updated Gaussian field as multi-view depth / voxels, reconstructing a high-precision non-penetrating mesh to complete scene reconstruction.

[0027] The rendering module is used to render the reconstructed scene.

[0028] Furthermore, the 2D-3D consistent clustering loss is jointly optimized with pixel error during training.

[0029] Furthermore, the rendering module constructs a multi-stream parallel CUDA graph based on the OpenAITriton framework, and executes the tasks of each module asynchronously in a pipeline, achieving a scene processing and update speed of ≥25 frames per second, and supporting real-time editing and backfilling.

[0030] Unlike existing technologies, the above technical solution is based on a unified instance coding 3DGS reconstruction method and system. It introduces instance-level understanding of the scene and real-world physical laws, segmenting objects within the scene and reconstructing individual object meshes. After segmentation, a generative object reconstruction model can be used to complete and reconstruct the objects, allowing each object in the reconstruction result to be independently extracted, refined, and appropriately placed within the scene. This improves reconstruction efficiency and avoids physical errors such as object positional deviations and overlaps, enhancing the realism of the reconstruction. Furthermore, the entire reconstruction process is highly parallelized and optimized, meeting the demands of real-time applications. In cutting-edge fields such as metaverse content creation and digital twin scene construction, this solution will significantly improve the efficiency and quality of 3D data acquisition and processing.

[0031] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0032] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on this application.

[0033] In the accompanying drawings of the instruction manual:

[0034] Figure 1 A flowchart illustrating the 3DGS reconstruction method based on unified instance coding as described in the specific implementation;

[0035] Figure 2 This is a flowchart illustrating the bidirectional iterative mesh extraction and optimization process described in the specific implementation method;

[0036] Figure 3 A flowchart illustrating the 3DGS reconstruction method based on unified instance coding as described in the specific implementation;

[0037] Figure 4 This is a block diagram of the 3DGS reconstruction system based on unified instance coding as described in a specific implementation.

[0038] The reference numerals used in the above figures are explained as follows:

[0039] 400. 3DGS Reconstruction System Based on Unified Instance Encoding; 401. 3DGS Reconstruction Module; 402. Point Cloud Completion Module; 403. Mesh Extraction and Gaussian Refinement Module; 404. Scene Layout Module; 405. Rendering Module. Detailed Implementation

[0040] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0041] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0042] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0043] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0044] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0045] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0046] Similar to the understanding in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0047] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0048] Unless otherwise expressly specified or limited, the terms "installation," "connection," "linking," "fixing," and "setting," as used in the description of the embodiments of this application, should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral setting; it can be a mechanical connection, an electrical connection, or a communication connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be the internal connection of two components or the interaction between two components. For those skilled in the art to which this application pertains, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0049] Please see Figure 1 and Figure 2 This embodiment provides a 3DGS reconstruction method based on Uniform Instance Coding. This 3DGS reconstruction method based on Uniform Instance Coding can be applied to metaverse content creation, digital twin scene construction, etc. Figure 1 As shown, the 3DGS reconstruction method based on unified instance coding includes the following steps:

[0050] S101. Instance-level segmentation of objects in the scene: The input multi-view image sequence is processed, and scene reconstruction is achieved through an improved 3D Gaussian cloud reconstruction algorithm, segmenting objects within the scene during the reconstruction process. Specifically, using the multi-view image sequence as input, instance ID embedding is introduced into the 3DGS framework, attaching a vector to each Gaussian sphere to represent its associated object. Through 2D-3D consistent clustering loss constraints, the 2D instance segmentation results in the multi-view images are compared with the Gaussian embedding, prompting the Gaussians of the same object to share similar features, achieving instance-level segmentation of objects in the scene, and outputting a Gaussian radiation field model with instance labels. The 2D-3D consistent clustering loss is jointly optimized with pixel error during training. In the 3DGS framework, the initialization of the Gaussian cloud is generated by transforming the sparse point cloud output from SLAM, and the Gaussian distribution is optimized through adaptive density control.

[0051] S102. Physically Guided Object-Level Point Cloud Completion: For each segmented instance object, the corresponding Gaussian sphere is extracted and converted into an initial sparse point cloud. A physically guided diffusion completion network is then used to complete the point cloud containing missing viewpoints. Specifically, this includes: for each segmented instance object, extracting a Gaussian sphere and converting it into an initial sparse point cloud with normal and color attributes; then using a physically guided diffusion completion network to complete the point cloud. The physically guided diffusion completion network starts from noisy point clouds based on a diffusion model, and during denoising sampling, a differentiable physical guidance term is superimposed, including penetration penalty and support stability constraints, to generate a physically reasonable dense point cloud. The weight coefficients of the penetration penalty and support stability constraints in the physical loss are λ1 and λ2, respectively, which are weighted and summed to form a joint physical guidance term.

[0052] S103. Bidirectional iterative mesh extraction and optimization to complete scene reconstruction. For example... Figure 2 As shown, bidirectional iterative mesh extraction and optimization: Perform the following operations on the completed Gaussian cloud:

[0053] S201, Initial Mesh Extraction: An improved SuGaR algorithm is used to sample surface points by taking advantage of the Gaussian distribution and the alignment characteristics of the object surface to generate a coarse mesh;

[0054] S202, Physical Collision Detection and Gaussian Correction: Utilizes the GPU-accelerated AABB+BVH engine to detect mesh intersecting regions, locate conflicting Gaussian points, and adjust their radius, density, or center position to eliminate geometric conflicts;

[0055] S203, Secondary Mesh Extraction: Using a volume fusion strategy similar to GS2Mesh, the updated Gaussian field is rendered as a multi-view depth / voxel, reconstructing a high-precision, non-penetrating mesh to complete scene reconstruction. The bidirectional iterative mesh extraction process iterates 1-2 times. The first iteration can eliminate more than 90% of the interlacing noise, and after the second iteration, the visible conflicts are basically eliminated.

[0056] In this embodiment, for multiple image and video inputs, real-time performance and rendering realism are the final results. The dynamic scene (i.e., the multiple images and input video) is captured using an RGB or RGB-D camera, and the RGB or RGB-D camera outputs: image sequence, camera pose, and sparse point cloud.

[0057] The data is then preprocessed, including keyframe selection and coordinate system alignment. During keyframe selection, blurred / highly motion-blurred frames are removed, retaining only frames that fully cover the scene. During coordinate system alignment, the SLAM coordinate system is converted to the 3DGS world coordinate system. After data preprocessing, 3DGS initialization and training are performed. Finally, real-time rendering and output are performed, and a PLY Gaussian model is exported after rendering. The 3DGS initialization and training includes:

[0058] Convert the sparse point cloud into an initial Gaussian. Adaptive density control: (1) clone the Gaussian in the high variance region based on the gradient magnitude; (2) trim the Gaussian with low opacity.

[0059] This embodiment also includes physically feasible layout optimization based on a scene hypergraph: Objects are used as hypergraph nodes to construct a hypergraph structure containing first-order support relationships, second-order directional proximity relationships, and higher-order semantic combination relationships. Differentiable graph neural networks are used with object poses and relationship tensors as input, and iterative optimization is performed through message passing to ensure the scene meets support stability, collision-free, and common-sense layout constraints. It also includes unified GPU scheduling and real-time rendering: a multi-stream parallel CUDA graph is constructed based on the OpenAI Triton framework, and the tasks of each module are executed asynchronously in a pipeline, achieving a scene processing and update speed of ≥25 frames per second, supporting real-time editing and backfilling.

[0060] like Figure 3 The diagram shown is a flowchart of the 3DGS reconstruction method based on unified instance coding; this 3DGS reconstruction method based on unified instance coding includes:

[0061] Step S1: 3DGS Unified Instance Encoding. In step S1, a multi-view image sequence is used as input, and an improved 3D Gaussian cloud reconstruction algorithm is employed. In this embodiment, instance ID embedding is introduced into the classic 3D Gaussian Splatting reconstruction framework, that is, each Gaussian sphere is assigned a vector representing the object it belongs to. During training, in addition to minimizing the pixel error of the reconstructed image, a 2D-3D consistency clustering loss is added: by comparing with the 2D instance segmentation results in the multi-view images, Gaussians of the same object are encouraged to share similar embeddings. This is similar to the idea of ​​Gaussian Grouping, which learns semantic features in Gaussian optimization. However, unlike existing methods, this embodiment incorporates 2D segmentation results such as Segment-Anything in a lighter way, and 3D instance clustering can be completed without additional large network training. The output of step S1 is a scene radiation field model composed of Gaussian parameters, with each Gaussian carrying its own instance label. In this way, the fast and high-fidelity reconstruction capability of 3DGS is maintained, and the representation of each independent object in the scene is obtained directly during the reconstruction stage.

[0062] Step S2: Object-level diffusion point cloud completion. In step S2, for each instance object obtained in step S1, the corresponding Gaussian sphere is extracted and converted into an initial sparse point cloud (which can carry attributes such as normal and color). Due to insufficient coverage of some multi-view images, the reconstructed objects have some missing viewpoints. For objects that are not fully reconstructed, a Physics-Guided Diffusion Completer (PG-DiffC) network is introduced to improve the point cloud. Specifically, PG-DiffC refines and generates the point cloud step by step based on a diffusion model: starting from the noisy point cloud, a differentiable physical guidance term, i.e., physical loss $L_{\mathrm{phys}}$, is added at each step of denoising and sampling. The loss includes: (a) Penetration: For newly generated points, a penalty is imposed for falling into the geometry of other objects already present in the scene; (b) Support Stability: This encourages generated point clouds to satisfy support relationships in the direction of gravity, for example, the bottom surface of an object should be flat on a supporting surface to avoid suspension or tipping. By superimposing these two types of physical constraints during the diffusion process (balancing coefficients $\lambda_1$ and $\lambda_2$ respectively), PG-DiffC can complete the scene model with missing perspectives while ensuring the physical rationality of object shapes, i.e., objects will not produce unrealistic structures such as cross-embedding or floating. This module outputs the completed dense point cloud for each instance, laying the foundation for subsequent fine mesh modeling.

[0063] Bidirectional Iterative Mesh Extraction and Gaussian Refinement Module: This module solves the problems of converting Gaussian representation to explicit mesh and eliminating geometric conflicts between objects. Its processing flow is shown in the attached diagram. Figure 2 As shown, it includes two iterative phases:

[0064] Step S3: Initial Mesh Extraction; In step S3, an improved SuGaR method is first used, taking the completed Gaussian cloud as input, to quickly generate a coarse mesh that fits the surface. SuGaR aligns the Gaussian distribution with the real surface, thus efficiently sampling surface points and reconstructing the mesh. Using this method, we obtain a preliminary triangular mesh representation while preserving details.

[0065] Step S4, Physical Collision Detection and Gaussian Write-back: In step S4, a comprehensive collision detection is performed on the initial mesh. Utilizing a GPU-accelerated AABB+BVH collision detection engine, intersecting and overlapping areas of objects in the scene are efficiently detected. If the meshes of two objects significantly intrude, the spatial location of the conflict and the involved Gaussian points are identified. Then, local geometric refinement is achieved by adjusting the parameters of these Gaussian points, such as reducing the Gaussian radius, density, or slightly shifting the Gaussian center, so that the meshes in the corresponding areas are pushed back within the boundaries of their respective objects. This "Gaussian Write-back" step is equivalent to correcting the implicit representation based on physical feedback, eliminating overlap at its source.

[0066] Step S5: Secondary Mesh Extraction; After Gaussian adjustment in step S4, mesh extraction is performed on the scene again. This time, a volume fusion fine mesh generation strategy similar to GS2Mesh is adopted: the updated Gaussian field is rendered into multi-view depth / voxels, and then fused and reconstructed to generate a high-precision mesh. The mesh obtained by secondary extraction is more refined than before and does not have serious cross-penetration.

[0067] The above process can be iterated 1-2 times as needed. Practice has shown that only one iteration is sufficient to eliminate the vast majority (>90%) of noise interference issues; after two iterations, visible mesh conflicts in the scene are almost completely eliminated. Compared to one-time extraction, the bidirectional iterative strategy of this invention ensures that the final generated explicit mesh model is cleaner both visually and physically, laying a reliable geometric foundation for downstream physical layout optimization and rendering.

[0068] Step S6: Scene Hypergraph Optimization Outputs Physically Reasonable Poses. After obtaining a high-quality object mesh, step S6 further adjusts the pose and layout of each object in the scene to meet global physical and spatial rationality. Step S6 proposes a layout optimization method based on a scene hypergraph: each object in the scene is treated as a node, and a hypergraph is constructed to describe multi-order relationships. Specifically, first-order relationships represent support (e.g., whether object A directly supports object B); second-order relationships represent the relative orientation / proximity between objects (e.g., orientation, alignment, distance); third-order and higher-order relationships can describe combinatorial semantics (e.g., a set of furniture forming a typical living room configuration like "sofa + coffee table + rug"). With the hypergraph structure, a differentiable graph neural network (GNN) is used to perform message passing and joint optimization: the GNN takes the initial state of the object and the relationship tensor as input, and iteratively updates the pose parameters (position, orientation) of each object and the value of the relationship tensor through multi-layer message passing. During optimization, constraints in the hypergraph (such as support relationships requiring objects to be in contact and on the surface of the object below, and orientation relationships requiring coordinated orientation) are transformed into part of the loss function, thereby guiding the GNN to adjust the poses of each object. After several iterations, the entire scene tends to be physically stable: all objects are placed securely without overlapping or collisions, and the layout conforms to common sense (such as chairs arranged around a table, and decorations placed on the tabletop). The final output is the physically corrected poses of each object in the scene. This module essentially automates the scene layout steps that previously required manual adjustment, ensuring that the results can be used for physical simulation and interaction. Unlike previous placement methods that rely on heuristic rules or LLM semantics, our method directly integrates physical constraints into the learnable model, accurately and automatically improving the layout quality of the reconstructed scene.

[0069] Step S7, GPU unified scheduling and real-time rendering module; In order to efficiently connect the above stages, a special GPU scheduling strategy was designed in step S7. Based on the OpenAI Triton framework, a multi-stream parallel kernel was written, and the subtasks of modules (10) to (40) were encapsulated into an asynchronous CUDA graph for unified execution. This fully utilizes the parallel computing power and memory locality of the GPU, minimizing the copying and transmission of data between modules. The entire process, from image input to output of the complete scene, is completed on the GPU, and the asynchronous pipelines of each part are parallel, which speeds up the processing speed. According to the test, our system achieved a scene processing and update speed of more than 25 frames per second. This means that the present invention can support real-time scene editing and backfilling: when the user modifies an object or viewpoint on the front end, the background can immediately re-render the adjusted Gaussian model and mesh results and dynamically feed them back to the user. This performance far exceeds that of typical offline reconstruction pipelines, laying the foundation for deployment in interactive applications (such as scene reconstruction and editing in the Unity engine).

[0070] The 3DGS reconstruction method based on unified instance encoding described above can be applied to a variety of different scenarios and has significant advantages.

[0071] In the realms of gaming and the metaverse, this method can rapidly digitize real-world scenes into virtual scenes within Unity or Unreal Engine. Since the reconstructed results have already undergone physical correction, they can be imported into the engine for physical simulation and interaction without the need for manual fixes of issues like hovering or clipping. For example, by scanning a room with a mobile phone or AR glasses, the system can instantly generate a corresponding virtual room, which users can experience in real-time from a first-person perspective within the metaverse.

[0072] In interior design and digital twins, designers or robots can scan existing room layouts, reconstruct accurate 3D models, and attempt to rearrange furniture. Because we introduce support relationships and physical stability constraints, the adjusted layout will be realistically feasible, reducing design errors and safety hazards.

[0073] In robot navigation and Embedded AI, physically reliable scene reconstruction can provide a training environment for agents. Our method can generate high-quality scenes with contact relationships, which helps to simulate robot interaction with the environment and improve training effectiveness.

[0074] like Figure 4 As shown, in another embodiment, a 3DGS reconstruction system 400 based on Uniform Instance Coding is provided. This 3DGS reconstruction system 400 based on Uniform Instance Coding includes: a 3DGS reconstruction module 401, a point cloud completion module 402, a mesh extraction and Gaussian refinement module 403, a scene layout module, and a rendering module 405.

[0075] The 3DGS reconstruction module 401 takes a multi-view image sequence as input, introduces instance ID embedding in the 3DGS framework, and adds a vector to each Gaussian sphere to represent the object to which it belongs. Through 2D-3D consistent clustering loss constraint, it compares the 2D instance segmentation results in the multi-view images with the Gaussian embedding, prompting the Gaussians of the same object to share similar features, realizing instance-level segmentation of objects in the scene, and outputting a Gaussian radiation field model with instance labels.

[0076] The point cloud completion module 402 is used to extract Gaussian spheres from the segmented instance objects and convert them into an initial sparse point cloud with normal and color attributes. The point cloud is then completed using a physically guided diffusion completion network. The physically guided diffusion completion network starts from the noisy point cloud based on the diffusion model and superimposes a differentiable physical guidance term during denoising sampling, including penetration penalty and support stability constraint, to generate a physically reasonable dense point cloud.

[0077] The mesh extraction and Gaussian refinement module 403 is used to generate a coarse mesh by sampling surface points through the Gaussian distribution and the alignment characteristics of the object surface using the improved SuGaR algorithm; and to detect mesh intersecting regions by using the GPU-accelerated AABB+BVH engine, locate conflicting Gaussian points and adjust their radius, density or center position to eliminate geometric conflicts.

[0078] The scene layout module 404 is used for physically feasible layout optimization based on the scene hypergraph, including: taking objects as hypergraph nodes, constructing a hypergraph structure containing first-order support relationships, second-order orientational proximity relationships, and higher-order semantic combination relationships; using a differentiable graph neural network with object pose and relationship tensors as input, iteratively optimizing through message passing to ensure that the scene meets support stability, collision-free, and common-sense layout constraints; and using a volume fusion strategy similar to GS2Mesh to render the updated Gaussian field as multi-view depth / voxels, reconstructing a high-precision non-penetrating mesh to complete scene reconstruction.

[0079] The rendering module 405 is used to render the reconstructed scene.

[0080] The 2D-3D consistent clustering loss is jointly optimized with pixel error during training. The rendering module constructs a multi-stream parallel CUDA graph based on the OpenAI Triton framework, executing each module's tasks asynchronously in a pipeline to achieve a scene processing and update speed of ≥25 frames per second, supporting real-time editing and backfilling.

[0081] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A 3DGS reconstruction method based on unified instance coding, characterized in that, Includes the following steps: Taking multi-view image sequences as input, instance ID embedding is introduced into the 3DGS framework, and a vector is attached to each Gaussian sphere to represent the object to which it belongs. By constraining the 2D-3D consistent clustering loss, the 2D instance segmentation results in the multi-view images are compared with the Gaussian embedding, which promotes the Gaussian sharing of similar features of the same object, realizes instance-level segmentation of objects in the scene, and outputs a Gaussian radiation field model with instance labels. Physically guided object-level point cloud completion: For the segmented instance objects, Gaussian spheres are extracted and converted into initial sparse point clouds with normal and color attributes. The point cloud is then completed using a physically guided diffusion completion network. The physically guided diffusion completion network starts from noisy point clouds based on a diffusion model and superimposes differentiable physical guidance terms during denoising sampling, including penetration penalties and support stability constraints, to generate physically reasonable dense point clouds. Bidirectional iterative mesh extraction and optimization: Perform the following operations on the completed Gaussian cloud: Initial mesh extraction: An improved SuGaR algorithm is used to sample surface points by taking advantage of the Gaussian distribution and the alignment characteristics of the object surface to generate a coarse mesh; Physics collision detection and Gaussian correction: Utilizes a GPU-accelerated AABB+BVH engine to detect mesh intersecting regions, locate conflicting Gaussian points, and adjust their radius, density, or center position to eliminate geometric conflicts; Secondary mesh extraction: Using a volume fusion strategy similar to GS2Mesh, the updated Gaussian field is rendered as a multi-view depth / voxel, and a high-precision non-penetrating mesh is reconstructed to complete the scene reconstruction.

2. The 3DGS reconstruction method based on unified instance coding according to claim 1, characterized in that, The 2D-3D consistent clustering loss is jointly optimized with pixel error during training.

3. The 3DGS reconstruction method based on unified instance coding according to claim 1, characterized in that, The weighting coefficients of the penetration penalty and support stability constraint in the physical loss are λ1 and λ2, respectively, which are weighted and summed to form a joint physical guidance term.

4. The 3DGS reconstruction method based on unified instance coding according to claim 1, characterized in that, The bidirectional iterative mesh extraction process iterates 1-2 times. The first iteration can eliminate more than 90% of the interlacing noise, and after the second iteration, the conflicts are basically eliminated by visual inspection.

5. The 3DGS reconstruction method based on unified instance coding according to claim 1, characterized in that, It also includes physically feasible layout optimization based on scene hypergraph: objects are used as hypergraph nodes to construct a hypergraph structure containing first-order support relationships, second-order orientational proximity relationships and higher-order semantic combination relationships. Differentiable graph neural networks are used with object pose and relationship tensors as input, and iterative optimization is performed through message passing to make the scene meet the constraints of support stability, collision-free and common sense layout.

6. The 3DGS reconstruction method based on unified instance coding according to claim 1, characterized in that, It also includes unified GPU scheduling and real-time rendering: a multi-stream parallel CUDA graph is built based on the OpenAI Triton framework, and the tasks of each module are executed asynchronously in a pipeline to achieve a scene processing and update speed of ≥25 frames per second, supporting real-time editing and backfilling.

7. The 3DGS reconstruction method based on unified instance coding according to claim 1, characterized in that, In the 3DGS framework, the initialization of Gaussian clouds is generated by transforming sparse point clouds output by SLAM, and the Gaussian distribution is optimized by adaptive density control.

8. A 3DGS reconstruction system based on unified instance coding, characterized in that, include: The 3DGS reconstruction module takes multi-view image sequences as input, introduces instance ID embedding into the 3DGS framework, and adds a vector to each Gaussian sphere to represent the object to which it belongs. Through 2D-3D consistent clustering loss constraint, it compares the 2D instance segmentation results in the multi-view images with the Gaussian embedding, prompting the Gaussians of the same object to share similar features, realizing instance-level segmentation of objects in the scene, and outputting a Gaussian radiation field model with instance labels. The point cloud completion module is used to extract Gaussian spheres from the segmented instance objects and convert them into an initial sparse point cloud with normal and color attributes. The point cloud is then completed using a physically guided diffusion completion network. The physically guided diffusion completion network starts from the noisy point cloud based on the diffusion model and superimposes a differentiable physical guidance term during denoising sampling, including penetration penalty and support stability constraint, to generate a physically reasonable dense point cloud. The mesh extraction and Gaussian refinement module is used to generate a coarse mesh by sampling surface points through the Gaussian distribution and the alignment characteristics of the object surface using the improved SuGaR algorithm; and to detect mesh intersecting regions using the GPU-accelerated AABB+BVH engine, locate conflicting Gaussian points and adjust their radius, density or center position to eliminate geometric conflicts. The scene layout module is used for physically feasible layout optimization based on the scene hypergraph. It includes: taking objects as hypergraph nodes, constructing a hypergraph structure containing first-order support relationships, second-order orientational proximity relationships, and higher-order semantic combination relationships; using a differentiable graph neural network with object pose and relationship tensors as input, iteratively optimizing through message passing to ensure that the scene meets support stability, collision-free, and common-sense layout constraints; and using a volume fusion strategy similar to GS2Mesh to render the updated Gaussian field as multi-view depth / voxels, reconstructing a high-precision non-penetrating mesh to complete scene reconstruction. The rendering module is used to render the reconstructed scene.

9. The 3DGS reconstruction system based on unified instance coding according to claim 8, characterized in that, The 2D-3D consistent clustering loss is jointly optimized with pixel error during training.

10. The 3DGS reconstruction system based on unified instance coding according to claim 8, characterized in that, The rendering module is based on the OpenAI Triton framework to build a multi-stream parallel CUDA graph, and executes the tasks of each module asynchronously in a pipeline to achieve a scene processing and update speed of ≥25 frames per second, supporting real-time editing and backfilling.

Images