Three-dimensional dynamic scene graph construction method based on 3D Gaussian representation

Abstract

The invention discloses a three-dimensional dynamic scene graph construction method based on 3D Gauss, and belongs to the field of computer body intelligence. The implementation method comprises the following steps of: realizing object perception and semantic feature extraction of open vocabularies by utilizing a visual basic model; a 3D Gaussian scene with high fidelity and continuous object semantics is constructed through multi-view multi-dimension optimization of 3D Gaussian representation; constructing a multi-level three-dimensional scene graph, extracting spatial levels and semantic relationships among objects by using 3D spatial positions and semantic tags of instance objects existing in a semantic Gaussian graph, and constructing a multi-level spatial semantic topology to accurately represent an environment layout; according to the method for realizing local updating for the Gaussian scene graph based on the environmental structural similarity, environmental change detection is carried out through real-time RGB-D observation and the structural similarity between high-quality rendering views of the Gaussian scene graph, and corresponding local updating is carried out by using rapid training and differentiable rendering of 3D Gaussian representation. And the capability of adapting to a complex dynamic environment of the 3D Gaussian scene graph is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer embodied intelligence and relates to a method for constructing a three-dimensional dynamic scene graph based on 3D Gaussian representation. Background Art

[0002] 3D scene graphs have extensive practical applications in autonomous robot navigation, mobile manipulation, spatial semantic understanding, and environmental structure representation. In particular, in autonomous robot navigation using natural language commands, 3D scene graphs serve as a key technology for agent command analysis, task decomposition and planning, and target positioning and navigation. They extract the spatial structure and semantic information of the current scene from sensor data, providing a 3D scene representation rich in semantic information. Currently, 3D scene graphs can be constructed using various environmental representation techniques, including mesh-based, point cloud-based, and 3D Gaussian-based methods. However, existing scene graphs typically assume a static scene and represent the spatial topology and semantic relationships of the scene at a specific moment. This makes them difficult to scale to the real world, as the layout of a robot's work environment often changes due to the activities of humans or other agents. The pre-stored scene graphs used to guide task planning and execution differ from the real environment, significantly reducing the success rate of autonomous robot navigation and subsequent tasks. Further research on 3D scene graph construction methods that can adapt to complex and dynamic environments is of vital practical significance.

[0003] The goal of a 3D scene graph is to construct a multi-layer topological structure of the robot's spatial environment, which can represent the spatial and semantic relationships between objects in the environment. Traditional mesh- and point cloud-based environment representation methods were first applied to 3D scene graph construction due to their relative simplicity and efficiency. However, these methods suffer from poor reconstruction quality of fine-grained environmental features and lack methods for local dynamic updating. With the continuous advancement of computer graphics technology, 3D Gaussian-based environment representation methods have been introduced into 3D scene understanding. Compared with traditional environment representation techniques, these 3D Gaussian-based methods have demonstrated significant advantages in environment reconstruction quality and capturing geometric details. 3D Gaussian-based methods have overcome the limitations of traditional methods in reconstructing details and have been widely used in semantic environment understanding tasks. However, 3D Gaussian-based methods often require differentiable rendering to optimize the semantic properties of the Gaussian and cluster Gaussians belonging to the same object, making it challenging to capture high-precision 3D instance object classification. At the same time, by introducing additional object segmentation priors from an advanced computer vision foundation model (Segment Anything Model, SAM), the accuracy of 3D instance object segmentation can be significantly improved by jointly optimizing the semantic properties of 3D Gaussians from multiple views. However, multi-view joint optimization typically requires additional semantic embedding in pre-built 3D Gaussian geometric scenes or data preprocessing using a visual foundation model before joint gradient-based optimization. This offline process inherently conflicts with the robot's operating paradigm in dynamic environments and fails to provide a method for coping with scene changes.

[0004] In summary, although there are many existing methods for constructing 3D scene graphs, they all have their own limitations, especially the lack of methods that can adapt to dynamic environments. Summary of the Invention

[0005] To solve the problem that robots have difficulty adapting to dynamic environments when performing long-term complex tasks, a multi-layer scene semantic topological representation is provided to robots that is temporally and spatially consistent with the real workspace. The purpose of the present invention is to provide a method for constructing a dynamic scene graph based on 3D Gaussian, which uses a visual basic model to realize object perception and semantic feature extraction of open vocabulary; design a 3D-2D Gaussian object matching method based on composite similarity, and construct a joint feature loss (RGB color, depth, semantic features), optimize the 3D Gaussian representation to construct a high-fidelity and semantically continuous 3D Gaussian graph; construct a multi-level three-dimensional scene graph, use the 3D spatial position and semantic labels of the existing instance objects in the semantic Gaussian graph, extract the spatial hierarchy and semantic relationship between objects, and construct a multi-level spatial semantic topology to accurately represent the environment layout; based on the similarity of environmental structure, a local update method for the topological scene graph is implemented, and environmental change detection is performed through real-time RGB-D observation and the structural similarity between high-quality rendered views of the Gaussian scene graph, and the rapid training and differentiable rendering of the 3D Gaussian representation are used to perform corresponding local updates, thereby improving the ability of the 3D Gaussian scene graph to adapt to complex dynamic environments, thereby helping robots to perform complex tasks in dynamic environments for a long time.

[0006] In order to achieve the above objectives, the present invention is achieved through the following technical solutions:

[0007] The present invention discloses a method for constructing a dynamic scene graph based on 3D Gaussian, comprising the following steps:

[0008] Step 1: Implement open vocabulary object perception based on the visual foundation model and detect objects of interest

[0009] The data required for step 1 comes from public datasets or time-aligned color images obtained from RGB-D cameras. Depth Map and camera pose T t Data sequence, where t represents the input data at time t. The visual foundation models described in step 1 include YOLO-World, SAM (Segment Anything Model), and CLIP. Robots face various complex or diversely shaped objects in real-world work scenarios. Deep learning models such as object detection or instance segmentation trained on a single closed dataset are less applicable in complex environments. Therefore, when constructing an open-vocabulary semantic scene graph, an open-vocabulary 2D visual foundation model is selected for scene object perception and semantic feature extraction.

[0010] Step 1.1: Use the visual base model to obtain the current frame color image Instance masks that do not occlude each other For each color frame Use YOLO-World to obtain the target detection frame of all objects of interest in the current frame These detected object candidate regions are then used as input to SAM to generate the corresponding instance object masks. By constructing a mask post-processing method, it is ensured that there is no mutual occlusion relationship between all instance masks on the 2D image plane, and the instance masks that do not occlude each other in the current frame perspective are obtained.

[0011] Step 1.2: Extract the instance mask of the current frame using the basic visual language model The corresponding semantic feature vector is the open vocabulary object perception based on the visual basic model. Since the semantic descriptors extracted by CLIP are aligned with the input image, each mask m in the current frame instance mask set is t,i Input CLIP separately to extract the corresponding instance-level object semantic features f t,i All detected objects in the current frame are expressed as

[0012] Step 2: 3D-2D Gaussian Object Association (3D-2D GOA) matching method based on composite similarity. The object of interest detected in step 1 At each moment t, they are independent of each other and lack continuity in the time dimension. Therefore, based on step 1, a 3D Gaussian scene graph is constructed, and the detection object of the current frame needs to be calculated. and objects in the map 3D-2D Gaussian object matching is performed based on the composite similarity in the current frame perspective, thereby ensuring that the scene graph has spatiotemporal semantic consistency, that is, obtaining the detection object mask and semantic feature description that have spatiotemporal semantic consistency with the Gaussian scene graph.

[0013] Step 2.1: Render the 2D mask of the objects in the Gaussian scene graph of the camera pose at the current time t through the instance level rendering method ILR (Instance Level Rendering) and obtain the corresponding semantic feature vector to form the visible objects in the current frame perspective

[0014] Step 2.2: 3D-2D Gaussian Object Association (3D-2D GOA) based on composite similarity is used to detect the match between the object and the existing object. 3D-2D GOA weights the detected object in the current frame. and scene graph visible objects The geometric feature similarity and semantic feature similarity between them are used for joint similarity matching. First, the geometric similarity s geo By all 2D detection masks Projection mask of visible 3D Gaussian objects The intersection over Union (IoU) between them is expressed as follows: sem Detect objects by computing the CLIP semantic descriptor and the semantic feature vector of visible objects Finally, the joint similarity s is obtained by weighted summation of geometric similarity and semantic similarity. associate , and performs a greedy matching strategy by joint similarity, assigning each detection to the visible Gaussian object with the maximum similarity score, and assigning it to the visible Gaussian object with the maximum similarity score when no match is found with a joint similarity greater than δ sim When a visible object is detected, a new Gaussian object is initialized from the detection.

[0015]

[0016]

[0017] s associate (i,j)=w g s geo (i,j)+w s s sem (i,j) (3)

[0018] Among them, ω g and ω s is the proportion of geometric similarity and semantic similarity in the composite similarity.

[0019] Step 3: Multi-view, multi-dimensional Gaussian scene optimization based on keyframe sequences. Based on the detected object masks and semantic feature descriptions obtained in Step 2, which are temporally and spatially consistent with the Gaussian scene graph, a new Gaussian representation is incrementally initialized frame by frame. The scene graph is then optimized from multiple perspectives and dimensions based on multi-dimensional input observations to obtain an instance-level semantic scene representation with high geometric reconstruction quality and semantic consistency.

[0020] Step 3.1: Gaussian densification method based on opacity. Initialize a new 3D Gaussian based on the input observations processed by steps 1 and 2 to incrementally construct a 3D Gaussian scene graph. Using the influence g(x) of each 3D Gaussian at a point x in 3D space, calculate the opacity accumulation S of each pixel u in the 2D image plane of the Gaussian scene at the current perspective. t :

[0021]

[0022] Based on the current frame depth observation And the rendering depth of the Gaussian scene at the current camera perspective Construct the initialization mask M t To cover the newly observed areas and the pixels where the newly appeared geometric figures occlude the existing scene structure. Based on this mask, a new Gaussian sphere is added for each pixel. The initial attributes of the Gaussian sphere are the color of the pixel, the aligned depth, the eigenvalue of the corresponding Gaussian object index and the initial opacity:

[0023]

[0024] Among them, λ MDE represents the median depth error, and the instance eigenvalues ​​of all 3D Gaussians are then projected onto The 2D mask area corresponding to the detected object On top, construct the true value instance feature Semantic feature supervision for multi-dimensional scene optimization.

[0025] Step 3.2: Gaussian regularization method based on instance features. In the process of incrementally constructing the Gaussian scene graph, each attribute of the Gaussian sphere will be rendered differently according to the input observations of each dimension to obtain the corresponding optimization gradient, and will be optimized and fine-tuned based on the gradient of each attribute. Supervising the optimization process of the Gaussian scene only in the color and depth dimensions will cause the Gaussians initially grouped by object index to gradually shift to the area occupied by other objects, thereby causing 3D semantic segmentation artifacts and reduced boundary accuracy. Use the semantic features of the Gaussian scene to render the perspective of the 2D image plane at the current frame camera pose The ground-truth instance features constructed in step 3.1 Construct feature consistency loss L feature , strengthen the Gaussian semantic consistency within the instance and improve the semantic segmentation accuracy:

[0026]

[0027] Among them, λ1 and λ2 represent instance feature rendering views and the true value The weight ratio between L1 loss and structural similarity loss.

[0028] Step 3.3: Multi-view and multi-dimensional Gaussian scene optimization based on joint feature loss (JFO). In step 3.1, the mask M is initialized according to the calculation of the current frame. t Initialize the new Gaussian and construct the instance-level Gaussian semantic feature supervision loss L in step 3.2 feature Then, combined with the current frame color image and depth map Rendering perspective color map with Gaussian scene graph and depth map Construct joint feature loss L mapping Simultaneously optimize the color, depth, and semantic features of Gaussian scenes:

[0029]

[0030] L mapping =w c L color +w d L depth +w f L feature (10)

[0031] Among them, λ3, λ4 represent RGB rendering views and the true value The weight ratio between L1 loss and structural similarity loss. ω c ,ω d and ω f The weight ratio of RGB color loss, depth loss, and feature loss to the joint feature loss is calculated separately, and the sum of the three is always 1. Multi-dimensional optimization of the Gaussian scene is performed based on the gradient of each dimension to ensure the consistency of semantic segmentation and high-quality reconstruction of the scene's geometric features.

[0032] To avoid overfitting optimization for single-view observations, every n frames are saved as a keyframe to form a keyframe sequence. Based on the time interval and spatial similarity, k frames are selected from these keyframes as a similar keyframe sequence to perform multi-view optimization on the current frame. The similar keyframe sequence includes the current frame, the most recent keyframe, and the k-2 previous keyframes with the highest overlap with the current frame. The overlap is determined by determining the number of points in the point cloud of the current frame's depth map within the camera frustum of each keyframe. During the multi-dimensional optimization of Gaussian scenes using the joint feature loss, keyframes at different times are randomly selected from the similar keyframe sequence for multi-view and multi-dimensional optimization. Redundant Gaussian spheres with near-zero opacity or large covariance are removed during the optimization process to improve reconstruction quality.

[0033] Step 4: Based on the spatial hierarchy and semantic relationship, a multi-layer Gaussian scene topology graph is constructed to quickly retrieve the position of the target object in three-dimensional space through a hierarchical query method.

[0034] By incrementally constructing a Gaussian map with semantic consistency and centered on the object, the spatial structure layering and object semantic relationship of the current robot workspace are extracted to construct a hierarchical 3D scene graph G t ={O t ,E t}, where O t and E t They represent the geometry of object nodes and the set of relationship edges between objects. Spatial hierarchical relationships are calculated by the spatial positions between different objects. Object semantic relationships are judged by combining spatial hierarchical relationships and the semantic information of the objects themselves. Objects are divided into asset objects. Ordinary objects and independent objects A large visual language model is used to identify asset objects in indoor environments. Expected asset objects typically include furniture on the ground, rather than small portable containers or decorations. The ground in the Gaussian map is fitted by RANSAC, and all Gaussian spheres are transformed to the ground coordinate system. j With asset objects The spatial position relationship of ordinary objects is defined If the object o j The center position is located at the asset object If the object is within or above the spatial range of And create an edge e between them k (i,j)∈E t This edge not only represents the spatial hierarchy, but also indicates Depend on Independent objects are defined as the complement of assets and common objects. Typical examples include stools, windows, and similar entities that exhibit inherent independence within the operational environment. The constructed multi-layer scene graph, for a human query command "A{relationship}B" containing a series of hierarchical structures or connected relationships, first identifies three asset objects that exhibit the highest semantic similarity with "B." It then performs hierarchical localization, examining the child nodes and all independent objects of these selected asset objects to retrieve the object most relevant to "A" and locate its three-dimensional spatial position in the 3D scene.

[0035] Step 5: In a dynamic human-robot collaborative environment, changes in object positions and spatial relationships pose a significant challenge to robots performing long-term tasks. A local update method for the 3D Gaussian scene graph based on environmental structural similarity leverages real-time RGB-D observations, high-fidelity 3D Gaussian reconstruction, and fast differential rendering to detect structural similarity-based changes and dynamically update the 3D Gaussian scene graph constructed in the previous steps. This ensures spatiotemporal consistency between the robot's in-memory representation of the environment and its physical workspace, improving the 3D Gaussian scene graph's ability to adapt to complex and dynamic environments.

[0036] Step 5.1: In real-world scenarios, the lack of reliable pose priors fundamentally limits the construction and modification of Gaussian scene graphs. To bridge this perception gap, we obtain an initial pose estimate T for each frame observation using a visual-inertial odometry framework. t,est , and align the colors by minimizing camera observations and depth Iterative pose refinement is performed using the L1 loss between the rendered view and the camera pose T to obtain the accurate camera pose T t :

[0037]

[0038] Among them, λ5, λ6 represent the color and depth L1 loss in the camera pose optimization loss L tracking The weight ratio in , and only the visibility S t Only higher Gaussian balls will participate in camera pose optimization. At the same time, at this stage, all Gaussian parameters are frozen and not optimized, and only the camera pose parameters are refined.

[0039] Step 5.2: Environmental change detection method based on structural similarity. The precise camera pose T obtained by the above method t , detect local environment changes based on the observation of the current frame, including object disappearance, object displacement, and new object appearance. For object disappearance and object displacement, high-fidelity reconstruction and fast differentiable rendering of 3D Gaussian are used to obtain the camera pose T through ILR t Color object mask in the 2D image plane Calculate its RGB observation The structural similarity (SSIM) of the corresponding area within the image is used to determine whether the corresponding object has changed:

[0040]

[0041] If an object o j The corresponding structural similarity s change (j) is less than δ change , then it means object o j has been moved or replaced by another object. All the missing objects are grouped into the set O delete , and delete all redundant Gaussian balls belonging to the objects. For the appearance of new objects, by re-executing 3D-2D GOA, the new object set O appearing in the current frame is detected new The union of deleted objects and new objects constitutes the update set O update ={O delete ,O new}.if Clear the keyframe list constructed in step 3 to prevent outdated keyframes from participating in the subsequent Gaussian scene optimization. Otherwise, perform keyframe selection and multi-view multi-dimensional Gaussian scene optimization normally.

[0042] Step 5.3: Update the object set O based on the objects constructed in steps 5.1 and 5.2 update As well as the types of objects in the hierarchical scene graph, it realizes the local update of the environment representation in the robot's memory after the scene change is detected, thereby constructing a three-dimensional dynamic scene graph with 3D Gaussian representation, ensuring that there is no delay between the scene graph guiding task planning and execution and the real environment in the process of performing long-term and complex tasks, thereby improving the success rate of task completion.

[0043] Gaussian scene graph G t-1 Need to be based on O update The categories of objects in the are adjusted accordingly. j ∈O delete If the object is marked as a common object or an independent object, it only needs to be marked together with its parent edge e k Together from G t-1 On the contrary, if the object is an asset object, the deletion must be extended to all its child nodes and related edges. For the newly appeared object, determine its type by the method in step 4 and add it to G t-1 Insert the corresponding type of nodes and establish relationship edges to construct an updated 3D Gaussian scene graph G t , to adapt to complex dynamic environments and improve the ability of 3D Gaussian scene graphs to adapt to complex dynamic environments.

[0044] It also includes step six: based on steps two to five, the 3D spatial positions and semantic labels of the existing instance objects in the semantic Gaussian graph are used to extract the spatial hierarchy and semantic relationship between objects, and construct a multi-level spatial semantic topology to accurately represent the environment layout; based on the similarity of the environmental structure, a local update method for the topological scene graph is implemented, and environmental changes are detected through real-time RGB-D observations and the structural similarity between high-quality rendering views of the Gaussian scene graph, and corresponding local updates are performed using the rapid training and differentiable rendering of the 3D Gaussian representation to achieve layered scene semantic understanding and object positioning, assisting the robot in completing scene object retrieval and positioning, autonomous navigation, and final mobile operations.

[0045] Beneficial effects:

[0046] 1. The present invention discloses a method for constructing a three-dimensional dynamic scene graph based on 3D Gaussian representation. It uses high-quality instance rendering of 3D Gaussian and compares it with real-time RGB-D observation to detect environmental structural changes, and then performs corresponding local updates on the scene graph. This improves the ability of the three-dimensional Gaussian scene graph to cope with complex dynamic scenes and ensures the structural consistency of the scene graph and the robot's real workspace over a long time series.

[0047] 2. The present invention discloses a method for constructing a three-dimensional dynamic scene graph based on 3D Gaussian representation. Through a 3D-2D Gaussian object matching method based on composite similarity, the 2D mask and semantic feature vector of the 3D objects existing in the Gaussian scene graph under the current RGB-D observation perspective are obtained through instance rendering, and composite matching of geometric similarity and semantic similarity is performed with the observed objects, which significantly enhances the semantic consistency of the 3D Gaussian within the instance and improves the 3D semantic segmentation accuracy of the scene graph.

[0048] 3. The present invention discloses a method for constructing a three-dimensional dynamic scene graph based on 3D Gaussian representation. Through a three-dimensional scene graph construction and optimization method based on 3D Gaussian representation, the ability of 3D Gaussian representation for fast differentiable rendering is utilized to incrementally construct a three-dimensional scene graph and perform joint feature optimization, thereby fully capturing the geometric details of the environment and improving the quality and realism of the environment reconstruction.

[0049] 4. The present invention discloses a method for constructing a three-dimensional dynamic scene graph based on 3D Gaussian representation. Through a layered scene semantic understanding and object positioning method based on a multi-layer 3D Gaussian scene graph, hierarchical object retrieval is performed based on natural language instructions using the spatial hierarchy and semantic relationships between objects stored in the scene graph, thereby improving the accuracy of target object positioning.

[0050] 5. The present invention discloses a method for constructing a three-dimensional dynamic scene graph based on 3D Gaussian representation, which utilizes the 3D spatial positions and semantic labels of existing instance objects in the semantic Gaussian graph to extract the spatial hierarchy and semantic relationship between objects, and construct a multi-level spatial semantic topology to accurately represent the environment layout; based on the similarity of environmental structure, a local update method for the topological scene graph is implemented, and environmental change detection is performed through real-time RGB-D observation and structural similarity between high-quality rendering views of the Gaussian scene graph, and corresponding local updates are performed using fast training and differentiable rendering of 3D Gaussian representation, thereby realizing layered scene semantic understanding and object positioning, and assisting robots in completing scene object positioning, autonomous navigation and mobile operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of a method for constructing a dynamic scene graph based on 3D Gaussian disclosed by the present invention.

[0052] Figure 2This is a schematic diagram of the principle of 3D-2D Gaussian object matching in the present invention.

[0053] Figure 3 It is a construction example diagram of the 3D Gaussian dynamic scene graph of the Replica dataset in the embodiment.

[0054] Figure 4 2 is a schematic diagram of the visualization results of the joint feature optimization effect of the Replica dataset in the embodiment.

[0055] Figure 5 3 is a schematic diagram of the visualization result of the dynamic update of the real scene acquisition data set in the embodiment. DETAILED DESCRIPTION

[0056] The following, combined with the accompanying figures, clearly and completely describes the implementation steps of the present invention's method for constructing a 3D dynamic scene graph based on 3D Gaussian representation, and demonstrates its effectiveness through experimental results. This paper uses semantic segmentation and object querying in the Replica dataset, as well as dynamic changes constructed in real-world scenes, as examples.

[0057] Implementation Details: All of the following examples were implemented on a computer with an NVIDIA RTX 4090D GPU. The entire Gaussian optimization process was implemented using the deep learning framework Pytorch, and the visualization results were achieved using the Open3D framework. The optimization process used the Adam optimizer for gradient-based optimization. The learning rates for each Gaussian parameter were: xyz position: 0.0001, RGB color: 0.0025, semantic features: 0.0025, rotation: 0.001, and opacity: 0.05. The learning rates for each camera pose optimization were: rotation: 0.004, and translation: 0.002. Gaussian map optimization was performed for 60 epochs per iteration, and camera pose optimization was performed for 200 epochs per iteration. During Gaussian map optimization, all Gaussian parameters were set to learnable, while during camera pose optimization, all Gaussian parameters were frozen, and only the camera pose was optimized.

[0058] Example 1:

[0059] This example uses eight scene data sequences in the Replica dataset to construct a 3D Gaussian hierarchical scene graph.

[0060] like Figure 1 and Figure 2 As shown, the present embodiment discloses a method for constructing a 3D Gaussian layered scene graph, and the specific implementation steps are as follows:

[0061] Step 1: Implement open vocabulary object perception based on the visual foundation model and detect objects of interest

[0062] like Figure 3 As shown in the figure, the data required for step 1 comes from the color images of eight scenes in the Replica dataset. Depth Map and camera pose T t Data sequence, where t represents the input data at time t. The visual foundation models described in step 1 include YOLO-World, SAM (Segment Anything Model), and CLIP. Robots face various complex or diversely shaped objects in real-world work scenarios. Deep learning models such as object detection or instance segmentation trained on a single closed dataset are less applicable in complex environments. Therefore, when constructing an open-vocabulary semantic scene graph, an open-vocabulary 2D visual foundation model is selected for scene object perception and semantic feature extraction.

[0063] Step 1.1: Use the visual base model to obtain the current frame color image Instance masks that do not occlude each other For each color frame Use YOLO-World to obtain the target detection frame of all objects of interest in the current frame These detected object candidate regions are then used as input to SAM to generate the corresponding instance object masks. By constructing a mask post-processing method, it is ensured that there is no mutual occlusion relationship between all instance masks on the 2D image plane, and the instance masks that do not occlude each other in the current frame perspective are obtained.

[0064] Step 1.2: Extract the instance mask of the current frame using the basic visual language model The corresponding semantic feature vector is the open vocabulary object perception based on the visual basic model. Since the semantic descriptors extracted by CLIP are aligned with the input image, each mask m in the current frame instance mask set is t,i Input CLIP separately to extract the corresponding instance-level object semantic features f t,i All detected objects in the current frame are expressed as

[0065] Step 2: If Figure 2 As shown in the figure, the 3D-2D Gaussian Object Association (3D-2D GOA) method based on composite similarity is used. The object of interest detected in step 1 is At each moment t, they are independent of each other and lack continuity in the time dimension. Therefore, based on step 1, a 3D Gaussian scene graph is constructed, and the detection object of the current frame needs to be calculated. and objects in the map 3D-2D Gaussian object matching is performed based on the composite similarity in the current frame perspective, thereby ensuring that the scene graph has spatiotemporal semantic consistency, that is, obtaining the detection object mask and semantic feature description that have spatiotemporal semantic consistency with the Gaussian scene graph.

[0066] Step 2.1: Render the 2D mask of the objects in the Gaussian scene graph of the camera pose at the current time t through the instance level rendering method ILR (Instance Level Rendering) and obtain the corresponding semantic feature vector to form the visible objects in the current frame perspective

[0067] Step 2.2: 3D-2D Gaussian Object Association (3D-2D GOA) based on composite similarity is used to detect the match between the object and the existing object. 3D-2D GOA weights the detected object in the current frame. and scene graph visible objects The geometric feature similarity and semantic feature similarity between them are used for joint similarity matching. First, the geometric similarity s geo By all 2D detection masks Projection mask of visible 3D Gaussian objects The intersection over Union (IoU) between them is expressed as follows: sem Detect objects by computing the CLIP semantic descriptor and the semantic feature vector of visible objects Finally, the joint similarity s is obtained by weighted summation of geometric similarity and semantic similarity. associate , and performs a greedy matching strategy by joint similarity, assigning each detection to the visible Gaussian object with the maximum similarity score, and assigning it to the visible Gaussian object with the maximum similarity score when no match is found with a joint similarity greater than δ sim When a visible object is detected, a new Gaussian object is initialized from the detection.

[0068]

[0069]

[0070] s associate (i,j)=w g s geo (i,j)+w s s sem (i,j) (3)

[0071] Among them, ω g =ω s=0.5 is the proportion of geometric similarity and semantic similarity in the composite similarity.

[0072] Step 3: Multi-view, multi-dimensional Gaussian scene optimization based on keyframe sequences. Based on the detected object masks and semantic feature descriptions obtained in Step 2, which are temporally and spatially consistent with the Gaussian scene graph, a new Gaussian representation is incrementally initialized frame by frame. The scene graph is then optimized from multiple perspectives and dimensions based on multi-dimensional input observations to obtain an instance-level semantic scene representation with high geometric reconstruction quality and semantic consistency.

[0073] Step 3.1: Gaussian densification method based on opacity. Initialize a new 3D Gaussian based on the input observations processed by steps 1 and 2 to incrementally construct a 3D Gaussian scene graph. Using the influence g(x) of each 3D Gaussian at a point x in 3D space, calculate the opacity accumulation S of each pixel u in the 2D image plane of the Gaussian scene at the current perspective. t :

[0074]

[0075] Based on the current frame depth observation And the rendering depth of the Gaussian scene at the current camera perspective Construct the initialization mask M t To cover the newly observed areas and the pixels where the newly appeared geometric figures occlude the existing scene structure. Based on this mask, a new Gaussian sphere is added for each pixel. The initial attributes of the Gaussian sphere are the color of the pixel, the aligned depth, the eigenvalue of the corresponding Gaussian object index and the initial opacity:

[0076]

[0077] Among them, λ MDE represents 50 times the median depth error, and the instance eigenvalues ​​of all 3D Gaussians are then projected onto The 2D mask area corresponding to the detected object On top, construct the true value instance feature Semantic feature supervision for multi-dimensional scene optimization.

[0078] Step 3.2: Gaussian regularization method based on instance features. Figure 4As shown in the figure, in the process of incrementally constructing the Gaussian scene graph, each attribute of the Gaussian sphere is rendered differently according to the input observations of each dimension to obtain the corresponding optimization gradient, and the optimization is fine-tuned based on the gradient of each attribute. Supervising the optimization process of the Gaussian scene only in the color and depth dimensions will cause the Gaussians initially grouped by object index to gradually shift to the area occupied by other objects, thereby causing 3D semantic segmentation artifacts and reduced boundary accuracy. Use the semantic feature rendering perspective of the Gaussian scene in the 2D image plane of the current frame camera pose The ground-truth instance features constructed in step 3.1 Construct feature consistency loss L feature , strengthen the Gaussian semantic consistency within the instance and improve the semantic segmentation accuracy:

[0079]

[0080] Among them, λ1=0.8 and λ2=0.2 represent instance feature rendering views and the true value The weight ratio between L1 loss and structural similarity loss.

[0081] Step 3.3: Multi-view and multi-dimensional Gaussian scene optimization based on joint feature loss (JFO). In step 3.1, the mask M is initialized according to the calculation of the current frame. t Initialize the new Gaussian and construct the instance-level Gaussian semantic feature supervision loss L in step 3.2 feature Then, combined with the current frame color image and depth map Rendering perspective color map with Gaussian scene graph and depth map Construct joint feature loss L mapping Simultaneously optimize the color, depth, and semantic features of Gaussian scenes:

[0082]

[0083] L mapping =w c L color +w d L depth +w f L feature (10)

[0084] Among them, λ3=0.8 and λ4=0.2 represent RGB rendering view and the true value The weight ratio between L1 loss and structural similarity loss. ω c =ω f =0.5 and ω d= 1.0 represents the weight ratio of RGB color loss, feature loss, and depth loss to the joint feature loss, and ensures that the sum of the three is always 1. Multi-dimensional optimization of Gaussian scenes is performed based on the gradient of each dimension to ensure the consistency of semantic segmentation and high-quality reconstruction of scene geometric features.

[0085] In order to avoid overfitting optimization of single-view observations, every n frames are saved as a key frame to form a key frame sequence, and k frames are selected from them as similar key frame sequences based on time interval and spatial similarity to perform multi-view optimization on the current frame. The similar key frame sequence includes the current frame, the most recent key frame, and the k-2 previous key frames with the highest overlap with the current frame. The overlap is determined by judging the number of points in the point cloud of the current frame depth map within the camera frustum of each key frame. In the process of multi-dimensional optimization of Gaussian scenes using joint feature loss, key frames at different times are randomly selected from the similar key frame sequence for multi-view and multi-dimensional optimization. At the same time, redundant Gaussian spheres with opacity close to zero or large covariance during the optimization process are deleted to improve the reconstruction quality. As shown in Table 1, the results of 3D open vocabulary semantic segmentation in the Replica dataset show that multi-view and multi-dimensional Gaussian scene optimization based on joint feature loss significantly improves the consistency of scene semantics:

[0086] Table 1 Comparative experiments on 3D open vocabulary semantic segmentation on the Replica dataset

[0087]

[0088] Table 2 Comparative experiment of geometric reconstruction quality on Replica dataset

[0089]

[0090] As shown in Table 2, the geometric reconstruction quality on the Replica dataset, multi-view multi-dimensional Gaussian scene optimization based on joint feature loss, significantly outperforms the traditional point cloud-based scene graph construction methods HOV-SG and ConceptGraphs, and slightly outperforms the advanced 3D Gaussian-based SGS-SLAM.

[0091] Step 4: Based on the spatial hierarchy and semantic relationship, a multi-layer Gaussian scene topology graph is constructed to quickly retrieve the position of the target object in three-dimensional space through a hierarchical query method.

[0092] like Figure 3 As shown in the figure, through the incremental construction of the semantically consistent Gaussian map centered on the object, the spatial structure layering and object semantic relationship of the current robot workspace are extracted to construct a hierarchical 3D scene graph G t ={O t ,Et}, where O t and E t They represent the geometry of object nodes and the set of relationship edges between objects. Spatial hierarchical relationships are calculated by the spatial positions between different objects. Object semantic relationships are judged by combining spatial hierarchical relationships and the semantic information of the objects themselves. Objects are divided into asset objects. Ordinary objects and independent objects A large visual language model is used to identify asset objects in indoor environments. Expected asset objects typically include furniture on the ground, rather than small portable containers or decorations. The ground in the Gaussian map is fitted by RANSAC, and all Gaussian spheres are transformed to the ground coordinate system. j With asset objects The spatial position relationship of ordinary objects is defined If the object o j The center position is located at the asset object If the object is within or above the spatial range of And create an edge e between them k (i,j)∈E t This edge not only represents the spatial hierarchy, but also indicates Depend on Independent objects are defined as the complement of assets and common objects. Typical examples include stools, windows, and similar entities that exhibit inherent independence within the operational environment. The constructed multi-layer scene graph, for a human query command "A{relationship}B" containing a series of hierarchical structures or connected relationships, first identifies three asset objects that exhibit the highest semantic similarity with "B." It then performs hierarchical localization, examining the child nodes and all independent objects of these selected asset objects to retrieve the object most relevant to "A" and locate its three-dimensional spatial position in the 3D scene.

[0093] Example 2:

[0094] like Figure 5 As shown, this example uses a real scene acquisition dataset to dynamically update the 3D Gaussian scene. The specific implementation steps are as follows:

[0095] Step 5: In a dynamic human-robot collaborative environment, changes in object positions and spatial relationships pose a significant challenge to robots performing long-term tasks. A local update method for the 3D Gaussian scene graph based on environmental structural similarity leverages real-time RGB-D observations, high-fidelity 3D Gaussian reconstruction, and fast differential rendering to detect structural similarity-based changes and dynamically update the 3D Gaussian scene graph constructed in the previous steps. This ensures spatiotemporal consistency between the robot's in-memory representation of the environment and its physical workspace, improving the 3D Gaussian scene graph's ability to adapt to complex and dynamic environments.

[0096] Step 5.1: In real-world scenarios, the lack of reliable pose priors fundamentally limits the construction and modification of Gaussian scene graphs. To bridge this perception gap, we obtain an initial pose estimate T for each frame observation using a visual-inertial odometry framework. t,est , and align the colors by minimizing camera observations and depth Iterative pose refinement is performed using the L1 loss between the rendered view and the camera pose T to obtain the accurate camera pose T t :

[0097]

[0098] Among them, λ5=0.8 and λ6=0.2 indicate that the color and depth L1 losses are in the camera pose optimization loss L tracking The weight ratio in , and only the visibility S t Only higher Gaussian balls will participate in camera pose optimization. At the same time, at this stage, all Gaussian parameters are frozen and not optimized, and only the camera pose parameters are refined.

[0099] Step 5.2: Environmental change detection method based on structural similarity. The precise camera pose T obtained by the above method t , detect local environment changes based on the observation of the current frame, including object disappearance, object displacement, and new object appearance. For object disappearance and object displacement, high-fidelity reconstruction and fast differentiable rendering of 3D Gaussian are used to obtain the camera pose T through ILR t Color object mask in the 2D image plane Calculate its RGB observation The structural similarity (SSIM) of the corresponding area within the image is used to determine whether the corresponding object has changed:

[0100]

[0101] If an object o j The corresponding structural similarity s change (j) is less than δchange =0.15, then it means that the object o j has been moved or replaced by another object. All the missing objects are grouped into the set O delete , and delete all redundant Gaussian balls belonging to the objects. For the appearance of new objects, by re-executing 3D-2D GOA, the new object set O appearing in the current frame is detected new The union of deleted objects and new objects constitutes the update set O update ={O delete ,O new}.if Clear the keyframe list constructed in step 3 to prevent outdated keyframes from participating in the subsequent Gaussian scene optimization. Otherwise, perform keyframe selection and multi-view multi-dimensional Gaussian scene optimization normally.

[0102] Step 5.3: Update the object set O based on the objects constructed in steps 5.1 and 5.2 update As well as the types of objects in the hierarchical scene graph, it realizes the local update of the environment representation in the robot's memory after the scene change is detected, thereby constructing a three-dimensional dynamic scene graph with 3D Gaussian representation, ensuring that there is no delay between the scene graph guiding task planning and execution and the real environment in the process of performing long-term and complex tasks, thereby improving the success rate of task completion.

[0103] Gaussian scene graph G t-1 Need to be based on O update The categories of objects in the are adjusted accordingly. j ∈O delete If the object is marked as a common object or an independent object, it only needs to be marked together with its parent edge e k Together from G t-1 On the contrary, if the object is an asset object, the deletion must be extended to all its child nodes and related edges. For the newly appeared object, determine its type by the method in step 4 and add it to G t-1 Insert the corresponding type of nodes and establish relationship edges to construct an updated 3D Gaussian scene graph G t , to adapt to complex dynamic environments and improve the ability of 3D Gaussian scene graphs to adapt to complex dynamic environments.

[0104] It also includes step six: based on steps two to five, the 3D spatial positions and semantic labels of the existing instance objects in the semantic Gaussian graph are used to extract the spatial hierarchy and semantic relationship between objects, and construct a multi-level spatial semantic topology to accurately represent the environment layout; based on the similarity of the environmental structure, a local update method for the topological scene graph is implemented, and environmental changes are detected through real-time RGB-D observations and the structural similarity between high-quality rendering views of the Gaussian scene graph, and corresponding local updates are performed using the rapid training and differentiable rendering of the 3D Gaussian representation to achieve layered scene semantic understanding and object positioning, assisting the robot in completing scene object retrieval and positioning, autonomous navigation, and final mobile operations.

[0105] The above specific description further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for constructing a dynamic scene graph based on 3D Gaussian, characterized by: The steps include: Step 1: Implement open vocabulary object perception based on the visual foundation model and detect objects of interest Step 2: 3D-2D Gaussian Object Association (3D-2D GOA) based on composite similarity; Based on step 1, a 3D Gaussian scene graph is constructed, and the detection object of the current frame needs to be calculated. and objects in the map 3D-2D Gaussian object matching is performed based on the composite similarity in the current frame perspective, thereby ensuring that the scene graph has spatiotemporal semantic consistency, that is, obtaining the detection object mask and semantic feature description that have spatiotemporal semantic consistency with the Gaussian scene graph; Step 3: Multi-view and multi-dimensional Gaussian scene optimization based on keyframe sequences. Based on the detection object mask and semantic feature description obtained in Step 2, which are temporally and spatially semantically consistent with the Gaussian scene graph, a new Gaussian representation is incrementally initialized frame by frame. The scene graph is then optimized from multiple perspectives and dimensions based on multi-dimensional input observations to obtain an instance-level semantic scene representation with high geometric reconstruction quality and semantic consistency. Step 4: Based on the multi-layer Gaussian scene topology construction method of spatial hierarchy and semantic relationship, a multi-layer scene graph is constructed; through the hierarchical query method, the position of the target object in the three-dimensional space is quickly retrieved; Step 5: A local update method for the 3D Gaussian scene graph based on environmental structural similarity uses real-time RGB-D observations and high-fidelity reconstruction and fast differential rendering of 3D Gaussian to achieve change detection based on structural similarity and perform local dynamic updates on the 3D Gaussian scene graph constructed in the above steps. This ensures the spatiotemporal consistency between the environmental representation in the robot's memory and the physical workspace, and improves the ability of the 3D Gaussian scene graph to adapt to complex dynamic environments.

2. The method for constructing a dynamic scene graph based on 3D Gaussian according to claim 1, wherein: The sixth step is to extract the spatial hierarchy and semantic relationship between objects based on the 3D spatial positions and semantic labels of the existing instance objects in the semantic Gaussian graph, and construct a multi-level spatial semantic topology to accurately represent the environment layout. A local update method for the topological scene graph is implemented based on the similarity of environmental structure. Environmental changes are detected through the structural similarity between real-time RGB-D observations and high-quality rendered views of the Gaussian scene graph. Corresponding local updates are performed using fast training and differentiable rendering of 3D Gaussian representations to achieve layered scene semantic understanding and object positioning, assisting robots in scene object retrieval and positioning, autonomous navigation, and ultimately mobile operations.

3. The method for constructing a dynamic scene graph based on 3D Gaussian according to claim 1 or 2, characterized in that: The data required for step 1 comes from public datasets or time-aligned color images obtained from RGB-D cameras. Depth Map and camera pose T t data sequence, where t represents the input data at time t; the visual foundation models described in step 1 include YOLO-World, SAM (Segment Anything Model), and CLIP; when constructing an open-vocabulary semantic scene graph, an open-vocabulary 2D visual foundation model is selected for scene object perception and semantic feature extraction.

4. The method for constructing a dynamic scene graph based on 3D Gaussian according to claim 3, wherein: The implementation method of step one is: Step 1.1: Use the visual base model to obtain the current frame color image Instance masks that do not occlude each other For each color frame Use YOLO-World to obtain the target detection frame of all objects of interest in the current frame These detected object candidate regions are then used as input to SAM to generate corresponding instance object masks. By constructing a mask post-processing method, it is ensured that there is no mutual occlusion relationship between all instance masks on the 2D image plane, and the instance masks that do not occlude each other in the current frame perspective are obtained. Step 1.2: Extract the instance mask of the current frame using the basic visual language model The corresponding semantic feature vector is the open vocabulary object perception based on the visual basic model; since the semantic descriptors extracted by CLIP are aligned with the input image, each mask m in the current frame instance mask set is t,i Input CLIP separately to extract the corresponding instance-level object semantic features f t,i ; All detected objects in the current frame are expressed as 5. The method for constructing a dynamic scene graph based on 3D Gaussian according to claim 4, wherein: The implementation method of step 2 is: Step 2.1: Render the 2D mask of the objects in the Gaussian scene graph of the camera pose at the current time t through the instance level rendering method ILR (Instance Level Rendering) and obtain the corresponding semantic feature vector to form the visible objects in the current frame perspective Step 2.2: 3D-2D Gaussian Object Association (3D-2D GOA) based on composite similarity is used to detect the match between the object and the existing object. 3D-2D GOA weights the detected object in the current frame. and scene graph visible objects The geometric feature similarity and semantic feature similarity between them are used for joint similarity matching. First, the geometric similarity s geo By all 2D detection masks Projection mask of visible 3D Gaussian objects The intersection over Union (IoU) between them represents the semantic similarity s sem Detect objects by computing the CLIP semantic descriptor and the semantic feature vector of visible objects Finally, the joint similarity s is obtained by weighted summation of geometric similarity and semantic similarity. associate , and performs a greedy matching strategy by joint similarity, assigning each detection to the visible Gaussian object with the maximum similarity score, and assigning it to the visible Gaussian object with the maximum similarity score when no match is found with a joint similarity greater than δ sim When a visible object is detected, a new Gaussian object will be initialized from the detection; s associate (i,j)=w g s geo (i,j)+w s s sem (i,j) (3) Among them, ω g and ω s is the proportion of geometric similarity and semantic similarity in the composite similarity.

6. The method for constructing a dynamic scene graph based on 3D Gaussian according to claim 5, wherein: The implementation method of step three is: Step 3.1: Gaussian densification method based on opacity. Initialize a new 3D Gaussian based on the input observations processed by steps 1 and 2 to incrementally construct a 3D Gaussian scene graph. Using the influence g(x) of each 3D Gaussian at a point x in 3D space, calculate the opacity accumulation S of each pixel u in the 2D image plane of the Gaussian scene at the current perspective. t : Based on the current frame depth observation And the rendering depth of the Gaussian scene at the current camera perspective Construct the initialization mask M t To cover the newly observed areas and the pixels where the newly appeared geometric figures occlude the existing scene structure. Based on this mask, a new Gaussian sphere is added for each pixel. The initial attributes of the Gaussian sphere are the color of the pixel, the aligned depth, the eigenvalue of the corresponding Gaussian object index and the initial opacity: Among them, λ MDE represents the median depth error, and the instance eigenvalues ​​of all 3D Gaussians are then projected onto The 2D mask area corresponding to the detected object On top, construct the true value instance feature Used for semantic feature supervision in multi-dimensional scene optimization; Step 3.2: Gaussian regularization method based on instance features. In the process of incrementally constructing the Gaussian scene graph, each attribute of the Gaussian sphere will be rendered differently according to the input observations of each dimension to obtain the corresponding optimization gradient, and will be optimized and fine-tuned based on the gradient of each attribute. Supervising the optimization process of the Gaussian scene only in the color and depth dimensions will cause the Gaussians initially grouped by object index to gradually shift to the area occupied by other objects, thereby causing 3D semantic segmentation artifacts and reduced boundary accuracy. Use the semantic features of the Gaussian scene to render the perspective of the 2D image plane at the current frame camera pose The ground truth instance features constructed in step 3.1 Construct feature consistency loss L feature , strengthen the Gaussian semantic consistency within the instance and improve the semantic segmentation accuracy: Among them, λ1 and λ2 represent instance feature rendering views and the true value The weight ratio between L1 loss and structural similarity loss; Step 3.3: Multi-view and multi-dimensional Gaussian scene optimization based on joint feature loss (JFO). In step 3.1, the mask M is initialized according to the calculation of the current frame. t Initialize the new Gaussian and construct the instance-level Gaussian semantic feature supervision loss L in step 3.2 feature Then, combined with the current frame color image and depth map Rendering perspective color map with Gaussian scene graph and depth map Construct joint feature loss L mapping Simultaneously optimize the color, depth, and semantic features of Gaussian scenes: L mapping =w c L color +w d L depth +w f L feature (10) Among them, λ3, λ4 represent RGB rendering views and the true value The weight ratio between L1 loss and structural similarity loss. ω c ,ω d and ω f The weight ratio of RGB color loss, depth loss and feature loss to the joint feature loss is calculated separately, and the sum of the three is always 1. Multi-dimensional optimization of Gaussian scenes is performed based on the gradient of each dimension to ensure the consistency of semantic segmentation and high-quality reconstruction of scene geometric features; To avoid overfitting optimization for single-view observations, every n frames are saved as a keyframe to form a keyframe sequence. Based on the time interval and spatial similarity, k frames are selected from these keyframes as a similar keyframe sequence to perform multi-view optimization on the current frame. The similar keyframe sequence includes the current frame, the most recent keyframe, and the k-2 previous keyframes with the highest overlap with the current frame. The overlap is determined by determining the number of points in the point cloud of the current frame's depth map within the camera frustum of each keyframe. During the multi-dimensional optimization of Gaussian scenes using the joint feature loss, keyframes at different times are randomly selected from the similar keyframe sequence for multi-view and multi-dimensional optimization. Redundant Gaussian spheres with near-zero opacity or large covariance are removed during the optimization process to improve reconstruction quality.

7. The method for constructing a dynamic scene graph based on 3D Gaussian according to claim 6, wherein: The implementation method of step 4 is: By incrementally constructing a Gaussian map with semantic consistency and centered on the object, the spatial structure layering and object semantic relationship of the current robot workspace are extracted to construct a hierarchical 3D scene graph G t ={O t ,E t }, where O t and E t They represent the geometry of object nodes and the set of relationship edges between objects. Spatial hierarchical relationships are calculated by the spatial positions between different objects. Object semantic relationships are judged by combining spatial hierarchical relationships and the semantic information of the objects themselves. Objects are divided into asset objects. Ordinary objects and independent objects A large visual language model is used to identify asset objects in indoor environments. Expected asset objects typically include furniture on the ground, rather than small portable containers or decorations. The ground in the Gaussian map is fitted by RANSAC, and all Gaussian spheres are transformed to the ground coordinate system. j With asset objects The spatial position relationship of ordinary objects is defined If the object o j The center position is located at the asset object If the object is within or above the spatial range of And create an edge e between them k (i,j)∈E t This edge not only represents the spatial hierarchy, but also indicates Depend on Independent objects are defined as the complement of assets and common objects. Typical examples include stools, windows, and similar entities that exhibit inherent independence within the operational environment. The constructed multi-layer scene graph, for a human query command "A{relationship}B" consisting of a series of hierarchical structures or connected relationships, first identifies three asset objects that exhibit the highest semantic similarity to "B." It then performs hierarchical localization, examining the child nodes and all independent objects of these selected asset objects to retrieve the object most relevant to "A" and locate its three-dimensional spatial position within the 3D scene.

8. The method for constructing a dynamic scene graph based on 3D Gaussian according to claim 7, wherein: The implementation method of step five is: Step 5.1: In real-world scenarios, the lack of reliable pose priors fundamentally limits the construction and modification of Gaussian scene graphs. To bridge this perception gap, we obtain an initial pose estimate T for each frame observation using a visual-inertial odometry framework. t,est , and align the colors by minimizing camera observations and depth Iterative pose refinement is performed using the L1 loss between the rendered view and the camera pose T to obtain the accurate camera pose T t : Among them, λ5, λ6 represent the color and depth L1 loss in the camera pose optimization loss L tracking The weight ratio in , and only the visibility S t Only higher Gaussian balls will participate in camera pose optimization. At this stage, all Gaussian parameters are frozen and not optimized, and only the camera pose parameters are refined. Step 5.2: Environmental change detection method based on structural similarity. The precise camera pose T obtained by the above method t , detect local environment changes based on the observation of the current frame, including object disappearance, object displacement, and new object appearance. For object disappearance and object displacement, high-fidelity reconstruction and fast differentiable rendering of 3D Gaussian are used to obtain the camera pose T through ILR t Color object mask in the 2D image plane Calculate its RGB observation The structural similarity (SSIM) of the corresponding area within the image is used to determine whether the corresponding object has changed: If an object o j The corresponding structural similarity s change (j) is less than δ change , then it means object o j has been moved or replaced by another object. All the missing objects are grouped into the set O delete , and delete all redundant Gaussian balls belonging to the objects. For the appearance of new objects, by re-executing 3D-2D GOA, the new object set O appearing in the current frame is detected new The union of deleted objects and new objects constitutes the update set O update ={O delete ,O new }.if Clear the keyframe list constructed in step 3 to prevent outdated keyframes from participating in the subsequent Gaussian scene optimization. Otherwise, proceed with keyframe selection and multi-view and multi-dimensional Gaussian scene optimization as normal. Step 5.3: Update the object set O based on the objects constructed in steps 5.1 and 5.2 update The robot can then detect changes in the scene and the types of objects in the layered scene graph, thereby locally updating the environment representation in the robot's memory after detecting changes in the scene. This allows the robot to construct a three-dimensional dynamic scene graph with a 3D Gaussian representation, ensuring that there is no delay between the scene graph that guides task planning and execution and the real environment during the execution of long-term and complex tasks, thereby improving the success rate of task completion. Gaussian scene graph G t-1 Need to be based on O update The categories of objects in the are adjusted accordingly. j ∈O delete If the object is marked as a common object or an independent object, it only needs to be marked together with its parent edge e k Together from G t-1 On the contrary, if the object is an asset object, the deletion must be extended to all its child nodes and related edges. For the newly appeared object, determine its type by the method in step 4 and add it to G t-1 Insert the corresponding type of nodes and establish relationship edges to construct an updated 3D Gaussian scene graph G t , to adapt to complex dynamic environments and improve the ability of 3D Gaussian scene graphs to adapt to complex dynamic environments.

Images