Scene understanding oriented open-vocabulary 3d scene graph generation system and method, computer storage medium

Abstract

The application provides a scene understanding-oriented open-vocabulary three-dimensional scene graph generation system and method, and a computer storage medium, and relates to the technical field of three-dimensional scene graph generation.The method comprises the following steps: acquiring an RGB image;reconstructing a three-dimensional Gaussian representation scene based on multi-view RGB images with camera poses;constructing a plurality of view selectors based on the three-dimensional Gaussian representation scene;acquiring a SAM mask based on a SAM model and the RGB image, and performing clustering;independently constructing a local K-neighbor graph for each cluster based on the clustering result and the three-dimensional Gaussian representation scene;performing smoothing processing on the K-neighbor graph;performing Gaussian codebook encoding based on the image subjected to the smoothing processing to obtain a root cluster node and a leaf cluster node;and generating a three-dimensional scene graph based on the root cluster node and the leaf cluster node.The application can alleviate the boundary feature pollution caused by alpha blending in the three-dimensional Gaussian representation, and retain semantic distribution information under different views.

Description

Technical Field

[0001] This application relates to the field of 3D scene graph generation technology, specifically to an open-vocabulary 3D scene graph generation system and method for scene understanding, and a computer storage medium. Background Technology

[0002] 3D scene understanding is a fundamental task in computer vision, with wide applications in areas such as robot navigation, embodied intelligence, and augmented reality. Achieving comprehensive scene understanding requires not only identifying objects within the scene but also reasoning about the spatial relationships between them. To this end, 3D scene graphs provide a structured representation where objects are encoded as nodes and spatial relationships are encoded as directed edges. By explicitly characterizing these relationships, scene graphs can serve as a powerful intermediate representation, supporting robot task planning, scene question answering, and spatial reasoning.

[0003] Recently, 3D Gaussian Splatting (3DGS) has become the mainstream paradigm for 3D scene representation due to its efficient rendering and high visual fidelity. Research on incorporating open-lexical semantics into 3DGS is also rapidly developing. For example, existing methods have achieved object-level semantic retrieval based on Gaussian scenes by embedding features from a visual-language model (CLIP model) into Gaussian primitives. However, these methods primarily focus on object-level localization or retrieval and do not explicitly model the spatial relationships between objects, limiting the ability to construct 3D scene graphs. Meanwhile, existing 3D scene graph generation methods typically operate on pre-segmented point clouds or implicit volumetric features, which are fundamentally different from explicit but unstructured Gaussian primitives.

[0004] Therefore, there is still a lack of technical solutions that can simultaneously achieve open vocabulary object retrieval and 3D scene graph generation on a 3D Gaussian representation. Summary of the Invention

[0005] This application provides an open-vocabulary 3D scene graph generation system and method for scene understanding, using a computer storage medium. This system can alleviate boundary feature pollution caused by α mixing in 3D Gaussian representation, preserve semantic distribution information from different perspectives, and realize open-vocabulary object retrieval, spatial relationship prediction, and 3D scene graph generation on 3D Gaussian representation.

[0006] The specific technical solution of this embodiment is as follows: On the one hand, embodiments of this application provide an open-vocabulary 3D scene graph generation method for scene understanding, including: S10, Obtain the RGB image; S20. Reconstructing a 3D Gaussian representation of a scene based on multi-view RGB images with camera pose; S30. Construct multiple view selectors for a scene based on 3D Gaussian representation; S40. Based on the SAM model and RGB image, obtain the SAM mask and perform clustering; S50. Based on the clustering results and the scenario represented by three-dimensional Gaussian, a local K-nearest neighbor graph is independently constructed for each cluster; where neighbors are determined by a joint location-feature metric. S60. Smooth the K-nearest neighbor graph; S70. Based on the smoothed image, Gaussian codebook encoding is performed to obtain the root cluster node and leaf cluster node; S80. Generate a 3D scene graph based on the root cluster node and leaf cluster node.

[0007] In some embodiments, smoothing the K-nearest neighbor graph includes graph Laplacian smoothing of the K-nearest neighbor graph with normalization.

[0008] In some embodiments, after constructing multiple view selectors, CLIP image features are extracted at three spatial scales for each SAM mask in each view. The background is suppressed by compact cropping and is close to the mask area. The bounding box is expanded by local cropping to capture the local context. The scope is further expanded by global cropping to include scene-level information. The CLIP image features at the three scales are weighted and fused. The fused single-view semantic features are associated with the corresponding leaf clusters to construct a multi-view feature library of the leaf clusters.

[0009] In some embodiments, generating a 3D scene graph based on root cluster nodes and leaf cluster nodes includes the following steps: S801. Based on the root cluster node and leaf cluster node, and using the perspective feature library obtained by multiple perspective selectors, extract the subject global features and object global features for each cluster node respectively. S802. Encode the location of the main cluster center using the first hash grid encoder; encode the location of the object cluster center using the second hash grid encoder; S803. Obtain the relative positions of the global features of the first subject and the global features of the first object; S804. The global features of the first subject and the global features of the first object, as well as the relative positions of the global features of the first subject and the global features of the first object, are concatenated. S805. Obtain the semantic features of the subject and object from the perspective feature library, and then input them into the subject projection head and object projection head respectively to obtain the subject embedding and object embedding of the role perception. S806, the results of splicing output based on S804, and the results of output based on S805 are merged; S807 generates a 3D scene map based on the relation decoder.

[0010] In some embodiments, after obtaining the relative positions of the first subject global feature and the first object global feature, and before concatenating the first subject global feature and the first object global feature, and the relative positions of the first subject global feature and the first object global feature, the following steps are also included: S8031. Normalize the relative position data of the global features of the first subject and the global features of the first object.

[0011] In some embodiments, before generating a 3D scene map based on the relationship decoder, and after the result of S804 stitching output and the fusion of the main projection head and the object projection head, the following steps are also included: K10, based on the dual-path relationship cross-attention mechanism, performs bidirectional interactive processing on the subject embedding and object embedding of role perception. The forward path makes the subject embedding pay attention to the object embedding, and the reverse path makes the object embedding pay attention to the subject embedding. K20, based on the fusion output and the result of bidirectional interactive processing of the subject embedding and object embedding of role perception based on the dual-path relationship cross-attention mechanism, is incorporated into the relation representation through residual gating to obtain the relation embedding aligned with the pre-trained text embedding space.

[0012] In some embodiments, the first hash grid encoder and the second hash grid encoder are the same shared resolution hash grid encoder.

[0013] On the other hand, embodiments of this application provide an open vocabulary 3D scene graph generation system for scene understanding, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the open vocabulary 3D scene graph generation method for scene understanding of any of the above embodiments.

[0014] This application also provides a computer storage medium storing a computer program, which is executed to implement the open vocabulary 3D scene graph generation method for scene understanding described in any of the above embodiments.

[0015] Compared with the prior art, the embodiments of this application have the following beneficial effects: This application provides an open-vocabulary 3D scene graph generation method for scene understanding, which effectively alleviates the problem of alpha-mixed feature contamination. After smoothing by the K-nearest neighbor graph, the features of each Gaussian are corrected by referring to spatially and semantically similar Gaussians. The fuzzy features contaminated at the boundary can be effectively corrected, improving the accuracy of subsequent instance clustering and semantic retrieval. At the same time, the multi-view selector retains the features under different views and builds a feature library separately, without directly collapsing the multi-view semantics into a single representation. It retains the semantic distribution information under different views, which can improve the object recognition performance under view changes or partially occluded scenes. Then, based on the object nodes obtained by clustering, it can further complete relation reasoning and generate a structured 3D scene graph. It directly realizes open-vocabulary object retrieval and 3D scene graph generation on the 3D Gaussian representation, bridging the gap between Gaussian representation and structured scene understanding. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the overall framework of the open vocabulary 3D scene graph generation method for scene understanding provided in some embodiments of this application; Figure 2 These are schematic diagrams illustrating the results of real-world object retrieval and relationship prediction provided in some embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0020] The use of "applies to" or "configured to" in this application implies open and inclusive language, which does not exclude the applicability to or configuration to devices performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, because processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated.

[0021] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0022] Currently, there are three core bottlenecks in constructing scene graphs based on 3D Gaussians. First, alpha-mixed feature contamination: when learning instance features of each Gaussian object through differentiable rendering, the features rendered at object boundaries are a mixture of multiple object features weighted by transmissivity. This results in blurred or noisy feature representations for the boundary Gaussians, reducing the reliability of subsequent object clustering and semantic retrieval. Second, multi-view semantic collapse: when associating 2D CLIP features with 3D clustering, existing methods typically use simple averaging or pooling to aggregate multi-view features. However, the semantics of objects may change with viewpoint; this aggregation method collapses the rich multi-view semantic distribution into a single representation, potentially reducing the recognition performance for view-dependent or partially occluded objects. Third, the gap between object retrieval and relational reasoning: while current 3DGS semantic methods support object-level retrieval, they do not explicitly reason about spatial relationships. Scene graph construction requires jointly modeling the semantic identity and spatial relationships of objects, fusing geometric configuration with semantic cues. Bridging this gap remains a key challenge in achieving structured scene understanding.

[0023] In summary, due to the aforementioned core bottlenecks, there is still a lack of technical solutions that can simultaneously achieve open-vocabulary object retrieval and 3D scene graph generation on 3D Gaussian representations, in order to bridge the gap between Gaussian representations and graph-structured scene understanding.

[0024] In view of this, on the one hand, embodiments of this application provide an open-vocabulary 3D scene graph generation method for scene understanding, including the following steps: S10, Obtain RGB image.

[0025] S20. Reconstruct a three-dimensional Gaussian representation of the scene based on the RGB images with multiple viewpoints including camera pose.

[0026] S30. Based on the 3D Gaussian representation of the scene, construct multiple view selectors.

[0027] S40. Based on the SAM model and RGB image, obtain the SAM mask and perform clustering.

[0028] S50. Based on the clustering results and the scenario of 3D Gaussian representation, construct a K-nearest neighbor graph independently for each cluster, where neighbors are determined by a joint location-feature metric.

[0029] S60. Smooth the K-nearest neighbor graph.

[0030] S70. Based on the smoothed image, Gaussian codebook encoding is performed to obtain the root cluster node and leaf cluster node.

[0031] S80. Generate a 3D scene graph based on the root cluster node and leaf cluster node.

[0032] It can acquire continuous RGB images of the target content, and it can acquire RGB images of the target content from multiple angles at the same time.

[0033] In the process of reconstructing a 3D Gaussian representation of a scene based on multi-view RGB images with camera pose, learnable instance features can be configured for each 3D Gaussian primitive.

[0034] Multiple view selectors can be constructed based on the image itself or based on the camera itself, such as the camera's intrinsic and extrinsic parameter matrices.

[0035] The SAM model is an interactive segmentation model based on Transformer that can generate high-precision object masks based on input prompts. It can automatically segment object instances in multi-view RGB images without additional fine-tuning for the target scene. It can directly adapt to the segmentation needs of any object in open scenes and provide an accurate instance region foundation for subsequent clustering and scene graph generation.

[0036] The steps for determining neighbors using the location-feature joint metric can be as follows: calculate the Euclidean distance and cosine distance between the spatial locations of two Gaussians respectively, then sum the two distances by weight to obtain the joint metric distance, sort the Gaussians by joint metric distance from smallest to largest, and select the top K Gaussians as the neighbors of the current Gaussian to complete the construction of the K nearest neighbor graph.

[0037] The above embodiments effectively mitigate the α-mixed feature contamination problem: after smoothing by the K-nearest neighbor graph, each Gaussian feature is corrected by referencing spatially and semantically similar Gaussians. Contaminated and ambiguous features at the boundaries can be effectively corrected, improving the accuracy of subsequent instance clustering and semantic retrieval. Simultaneously, the multi-view selector retains features from different viewpoints and constructs feature libraries separately, avoiding direct collapse of multi-view semantics into a single representation. This preserves semantic distribution information from different viewpoints, improving object recognition performance in scenarios with changing viewpoints or partial occlusion. Furthermore, based on the clustered object nodes, it can further perform relational reasoning and generate a structured 3D scene graph, directly achieving open-vocabulary object retrieval and 3D scene graph generation on the 3D Gaussian representation, bridging the gap between Gaussian representation and structured scene understanding.

[0038] In some embodiments, smoothing the K-nearest neighbor graph includes graph Laplacian smoothing of the K-nearest neighbor graph with normalization.

[0039] By using the settings in the above embodiments, Gaussian features can be smoothed and denoised while preserving the semantic boundaries between different instances, reducing the decrease in instance distinguishability caused by excessive feature smoothing, and further improving the reliability of feature representation.

[0040] In other embodiments, the smoothing of the K-nearest neighbor graph can also be performed in the following way: based on the graph convolutional neural network, the features of each node in the K-nearest neighbor graph are weighted and aggregated. The aggregation weight is inversely proportional to the joint metric distance between nodes. The closer the Gaussian is, the greater its contribution to the correction of the current Gaussian features. While preserving local semantic consistency, the mixed feature pollution of the boundary Gaussian is specifically corrected.

[0041] In some embodiments, after constructing multiple view selectors based on the scene in step S30, for each SAM mask in each view, CLIP image features are extracted at three spatial scales. The background is suppressed by compact cropping and is close to the mask area. The bounding box is expanded by local cropping to capture the local context. The scope is further expanded by global cropping to include scene-level information. The CLIP image features at the three scales are weighted and fused. The fused single-view semantic features are associated with the corresponding leaf clusters to construct the multi-view feature library of the leaf clusters.

[0042] Through the above embodiments, multi-scale features can simultaneously capture the local appearance information and global scene context information of an object, reduce semantic recognition bias caused by incomplete extraction of single-scale features, and further improve the accuracy of semantic retrieval of objects under open vocabulary, especially adapting to the feature extraction needs of different objects with large size differences.

[0043] In some embodiments, step S80, generating a 3D scene graph based on the root cluster node and leaf cluster nodes, includes the following steps: S801. Based on the root cluster node and leaf cluster node, and using a view feature library obtained from multiple view selectors, extract the subject global features and object global features for each cluster node.

[0044] S802. Encode the location of the main cluster center using the first hash grid encoder; encode the location of the object cluster center using the second hash grid encoder; S803. Obtain the relative positions of the global features of the first subject and the global features of the first object; S804. The global features of the first subject and the global features of the first object, as well as the relative positions of the global features of the first subject and the global features of the first object, are concatenated. S805. Obtain the semantic features of the subject and object from the perspective feature library, and then input them into the subject projection head and object projection head respectively to obtain the subject embedding and object embedding of the role perception. S806, the results of splicing output based on S804, and the results of output based on S805 are merged; S807 generates a 3D scene map based on the relation decoder.

[0045] The view feature library obtained from multiple view selectors refers to a collection that stores the multi-scale CLIP features corresponding to each cluster node under all views. It can be that the first and second hash grid encoders are of the same type, or they can be of different types.

[0046] Through the above embodiments, the absolute position, relative position and multi-perspective semantic features of objects can be integrated simultaneously to perform explicit reasoning on the spatial relationship between any two objects, effectively bridging the gap between object retrieval and relationship reasoning, and finally obtaining a structurally complete and semantically accurate open-vocabulary 3D scene map.

[0047] In some embodiments, after step S803, obtaining the relative positions of the first subject global feature and the first object global feature, and before step S804, concatenating the first subject global feature and the first object global feature, and obtaining the relative positions of the first subject global feature and the first object global feature, the following steps are also included: S8031. Normalize the relative position data of the global features of the first subject and the global features of the first object.

[0048] By setting up the above embodiments, the model training instability caused by the difference in location data scale under different size scenarios can be eliminated, the generalization ability and prediction accuracy of relation reasoning can be improved, and the model can stably output accurate spatial relation prediction results in various scenarios of different sizes.

[0049] Step S806, based on the stitching output of S804, and after the fusion of the main projection head and the object projection head, before step S807, based on the relationship decoder, generates the 3D scene map, also includes the following steps: K10, based on the dual-path relationship cross-attention mechanism, performs bidirectional interactive processing on the subject embedding and object embedding of role perception. The forward path makes the subject embedding pay attention to the object embedding, and the reverse path makes the object embedding pay attention to the subject embedding. K20, based on the fusion output and the result of bidirectional interactive processing of the subject embedding and object embedding of role perception based on the dual-path relationship cross-attention mechanism, is incorporated into the relation representation through residual gating to obtain the relation embedding aligned with the pre-trained text embedding space.

[0050] Through the above embodiments, dual-path cross-attention enables the subject and object to rely on each other's semantic and positional information to optimize their own relation embedding representation, capture the dependency relationship between the two, improve the accuracy of relation reasoning, and align the relation embedding to the embedding space of the pre-trained text encoder, which can directly support relation category prediction of open vocabulary. After completing the relation embedding training, it can support open vocabulary relation query training outside the relation category by similarity matching with the relation text vocabulary, thus expanding the relation reasoning capability in open scenarios.

[0051] In some embodiments, the first hash grid encoder and the second hash grid encoder are the same shared resolution hash grid encoder.

[0052] By setting up the above embodiments, the total number of model parameters can be reduced, encoder parameter redundancy can be reduced, storage overhead and computational costs for model training and deployment can be lowered, while improving the consistency of position encoding and reducing the impact on relational inference accuracy.

[0053] Please see Figure 1 The open-vocabulary 3D scene graph generation method provided in this application embodiment, based on a unified framework of 3D Gaussian splash representation, can simultaneously complete open-vocabulary object retrieval and 3D scene graph generation, specifically including two stages: In the hierarchical open vocabulary clustering stage, we introduce an adaptive neighborhood consistency mechanism to optimize the instance features of each Gaussian. This mechanism applies normalized Laplacian smoothing to the K-nearest neighbor graph to alleviate boundary contamination caused by α-mixture. Based on this, Gaussian codebook encoding organizes these Gaussians into a coarse-to-fine hierarchical structure, producing coarse-grained root clusters and fine-grained leaf clusters. To associate the clustering with the semantics of the open vocabulary, visual-linguistic features are extracted at three spatial scales and stored in the viewpoint library module, retaining a set of high-confidence single-viewpoint features sorted by matching score.

[0054] In the relationship graph generation stage, for each pair of object clusters, the absolute position and relative geometric relationship of the object pair are first encoded by a multi-resolution hash grid encoder. Simultaneously, independent subject and object projection heads map visual-linguistic semantic features into subject and object embeddings for role perception. Then, a gated fusion module adaptively balances geometric and semantic features. The fused representation is first refined by a gated residual MLP, and then the directionality of the relationships is modeled by dual-path cross-attention. Ultimately, the relationship graph learns an embedding space that is discriminative of spatial relationships, achieving end-to-end inference from 3DGS representation to a structured 3D scene graph. We train and evaluate the model on the real-world indoor scene dataset LERF and the 3D scene graph dataset 3DSSG. The framework takes pose-integrated multi-view RGB images and SAM segmentation masks for each view as input, and the relation supervision signal comes from ground-value 3D scene annotations or relation labels generated by a large language model. The large language model can be a conventional model.

[0055] The scene is represented by a 3D Gaussian splash, where each Gaussian primitive, in addition to the standard center position, covariance, opacity, and spherical harmonic color parameters, adds a learnable instance feature. After the initial reconstruction converges, a SAM mask is used as pseudo-supervision, and instance feature maps are rendered and optimized through alpha mixing. To mitigate boundary feature contamination caused by alpha mixing, an adaptive neighborhood consistency mechanism is introduced. A local K-nearest neighbor graph is independently constructed for each cluster, with neighbors determined by a joint location-feature metric, structurally preventing cross-instance feature leakage. Edge weights are assigned by a Gaussian kernel based on feature distance, and degree-normalized graph Laplacian smoothing is applied to smooth features within clusters while maintaining sharp boundaries between clusters.

[0056] In some examples, the graph can be reconstructed periodically every 500 iterations to adapt to the evolving feature space. The number of neighbors in the K-nearest neighbor graph can be 16, and the smoothing bandwidth can be 0.05.

[0057] Gaussian Codebook Encoding organizes consecutive Gaussian features into hierarchical clusters. At the root level, Gaussian features with low variance in their neighborhood are first selected as stable anchors. Farthest point sampling ensures spatial coverage, and redundant seeds are merged using BIRCH clustering to generate R root cluster centers. The remaining Gaussian features are refined through two rounds of iteration and assigned to the nearest root centers. Clustering is performed in a joint 9-dimensional feature space concatenated from instance features and weighted spatial locations, with spatial location weights potentially set to 1.0. At the leaf level, each root cluster is further divided into at most L leaf clusters using BIRCH. When the number of sub-clusters exceeds L, the L largest clusters are retained, and smaller clusters are merged into the nearest retained cluster in the feature space. Each leaf cluster receives a globally unique identifier.

[0058] After clustering is completed, each leaf cluster is semantically associated with an open vocabulary through the View Bank Module. For each SAM mask in each view, visual-linguistic image features are extracted at three spatial scales. Tight cropping suppresses the background and closely adheres to the mask region. Local cropping expands the bounding box to capture local context, and global cropping further expands the scope to include scene-level information.

[0059] In some examples, the features at the three scales can be summed with weights of 0.50, 0.30, and 0.20.

[0060] To prevent the collapse of multi-view semantic distribution, a view library is designed for each cluster. Up to 100 single-view features can be stored and sorted by matching score. During querying, the text query is encoded by a visual-language text encoder. For each feature in the library, a set of canonical text embeddings is used as a negative reference to calculate the canonical normalized relevance score. The final clustering score can be the average of the top 5 single-view relevance scores.

[0061] In the relation graph generation stage, relation embeddings are generated for each cluster pair composed of subject clusters and object clusters. Cluster centers are first mapped to bounded regions through scene shrinking and normalized, then input into a 12-level multi-resolution hash grid encoder to obtain absolute position encoding. Simultaneously, the displacement vectors and Euclidean distance between the two centers are calculated. Relative geometric features are obtained through a two-layer multilayer perceptron with layer normalization, and the two are concatenated to form a geometric representation. Semantic features are mapped to role-aware embeddings through independent subject and object projection heads. Geometric and semantic branches are adaptively weighted through a gating fusion mechanism and refined by three residual multilayer perceptron modules with learnable dimensional scaling. To characterize the directionality of spatial relations, a dual-path relation pair cross-attention module is introduced. The forward path allows subject embeddings to focus on object embeddings, while the backward path allows object embeddings to focus on subject embeddings. The two outputs are residual-gated and incorporated into the relation representation, ultimately resulting in a 512-dimensional relation embedding aligned with the pre-trained text embedding space.

[0062] Enumerating all cluster pairs during inference is too costly, so a two-stage strategy can be adopted. First, a lightweight pair scoring network quickly evaluates all cluster pairs based on semantic and geometric features, retaining the top 50 candidates. Then, a complete relation modeling module is run only on the candidate pairs. The resulting relation embeddings are matched with a predefined relation vocabulary using cosine similarity to construct the final 3D scene graph.

[0063] Loss function part: During model training, the loss function can be set as follows: Adaptive Neighborhood Consistency Loss: in, and Instance features representing adjacent Gaussians, It is the set of edges of the K-nearest neighbor graph, which is constructed independently within each cluster. It is a Gaussian kernel edge weight. This is the smoothing bandwidth, which can be set to 0.05. It is a node The weighted degree is the sum of the weights of all its neighboring edges.

[0064] The purpose of this loss function is to smooth the Gaussian features within clusters while maintaining sharp boundaries between clusters, thereby eliminating boundary feature contamination caused by alpha mixing. Degree normalization is introduced to alleviate the dominant role of height-numbered nodes in smoothing. The K-nearest neighbor graph is reconstructed periodically every 500 iterations to adapt to the continuously evolving feature space during training.

[0065] Cosine Alignment Loss: in, Represents the relationship embedding of predictions. This represents the target embedding obtained by encoding the natural language relation description using a frozen text encoder. and These represent the predicted embedding of randomly sampled negative samples and the unrelated target embedding, respectively, and the positive sample weights. A value of 2.0 is acceptable, with negative sample weights. 0.05 is acceptable.

[0066] The purpose of this loss function is to align the predicted relation embeddings with the relation text embeddings, so that relation predictions can be made by matching the cosine similarity with the relation vocabulary.

[0067] Total loss during the graph generation phase: in This corresponds to four auxiliary losses. The symmetric InfoNCE contrastive loss within the batch is used, with other samples in the batch as negative samples. The temperature coefficient can be set to 0.07 to enhance the discriminativeness of relation embedding. As a diversity regularization term, it prevents pattern collapse by constraining the mean cosine similarity between different predictions. For the pairwise confidence loss, an auxiliary head is trained to predict whether a valid relation exists for a given cluster pair. For object classification loss, supervised prediction of subject and object categories from backbone features. Various weights... The values ​​can be 0.1, 5.0, 0.5, and 0.5 respectively, with the diversity regularization term given a larger weight because its numerical scale is much smaller than other terms; otherwise, its gradient would be dominated by the primary objective. The total loss uses cosine alignment loss as the primary objective, combined with contrastive learning and multiple auxiliary supervisions, to jointly learn an embedding space that is discriminative of spatial relationships.

[0068] Effect demonstration: like Figure 2 As shown, the method performs well in real-world robot deployment scenarios. The first column is the input RGB image; the second column is a semantic clustering process diagram, where the method accurately decomposes the scene into semantically meaningful regions; the third column is the open-vocabulary object retrieval results, successfully retrieving individual object instances such as tennis balls, eggplants, snack bags, and carrots; the fourth column is the relation prediction results, generating accurate and focused relation predictions for a given subject and predicate. These results demonstrate the effectiveness of the method in real-world unstructured environments.

[0069] On the other hand, embodiments of this application provide an open vocabulary 3D scene graph generation system for scene understanding, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the open vocabulary 3D scene graph generation method for scene understanding of any of the above embodiments.

[0070] This application also provides a computer storage medium storing a computer program, which is executed to implement the open vocabulary 3D scene graph generation method for scene understanding described in any of the above embodiments.

[0071] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating 3D scene graphs using an open vocabulary for scene understanding, characterized in that, include: S10, Obtain the RGB image; S20. Reconstruct a three-dimensional Gaussian representation of the scene based on the RGB images with multiple viewpoints including camera pose; S30. Construct multiple view selectors for a scene based on 3D Gaussian representation; S40. Based on the SAM model and RGB image, obtain the SAM mask and perform clustering; S50. Based on the clustering results and the scenario represented by three-dimensional Gaussian, a local K-nearest neighbor graph is independently constructed for each cluster; where neighbors are determined by a joint location-feature metric. S60. Smooth the K-nearest neighbor graph; S70. Based on the smoothed image, Gaussian codebook encoding is performed to obtain the root cluster node and leaf cluster node; S80. Generate a 3D scene graph based on the root cluster node and leaf cluster node.

2. The open-vocabulary 3D scene graph generation method for scene understanding according to claim 1, characterized in that, Smoothing the K-nearest neighbor graph includes graph Laplace smoothing with normalization applied to the K-nearest neighbor graph.

3. The open-vocabulary 3D scene graph generation method for scene understanding according to claim 1, characterized in that, After constructing multiple view selectors, CLIP image features are extracted at three spatial scales for each SAM mask in each view. The background is suppressed by compact cropping and is close to the mask area. The bounding box is expanded by local cropping to capture the local context. The scope is further expanded by global cropping to include scene-level information. The CLIP image features at the three scales are weighted and fused. The fused single-view semantic features are associated with the corresponding leaf clusters to construct the multi-view feature library of the leaf clusters.

4. The open-vocabulary 3D scene graph generation method for scene understanding according to any one of claims 1-3, characterized in that, Generating a 3D scene graph based on root and leaf cluster nodes includes the following steps: S801. Based on the root cluster node and leaf cluster node, and using the perspective feature library obtained by multiple perspective selectors, extract the subject global features and object global features for each cluster node respectively. S802. Encode the location of the main cluster center using the first hash grid encoder; encode the location of the object cluster center using the second hash grid encoder; S803. Obtain the relative positions of the global features of the first subject and the global features of the first object; S804. The global features of the first subject and the global features of the first object, as well as the relative positions of the global features of the first subject and the global features of the first object, are concatenated. S805. Obtain the semantic features of the subject and object from the perspective feature library, and then input them into the subject projection head and object projection head respectively to obtain the subject embedding and object embedding of the role perception. S806, the results of splicing output based on S804, and the results of output based on S805 are merged; S807 generates a 3D scene map based on the relation decoder.

5. The open-vocabulary 3D scene graph generation method for scene understanding according to claim 4, characterized in that, After obtaining the relative positions of the first subject global features and the first object global features, before concatenating the first subject global features and the first object global features, and before obtaining the relative positions of the first subject global features and the first object global features, the following steps are also included: S8031. Normalize the relative position data of the global features of the first subject and the global features of the first object.

6. The open-vocabulary 3D scene graph generation method for scene understanding according to claim 4, characterized in that, Based on the S804 stitching output, and after the fusion of the main projection head and the object projection head, before generating the 3D scene map based on the relationship decoder, the following steps are also included: K10, based on the dual-path relationship cross-attention mechanism, performs bidirectional interactive processing on the subject embedding and object embedding of role perception. The forward path makes the subject embedding pay attention to the object embedding, and the reverse path makes the object embedding pay attention to the subject embedding. K20, based on the fusion output and the result of bidirectional interactive processing of the subject embedding and object embedding of role perception based on the dual-path relationship cross-attention mechanism, is incorporated into the relation representation through residual gating to obtain the relation embedding aligned with the pre-trained text embedding space.

7. The open-vocabulary 3D scene graph generation method for scene understanding according to claim 4, characterized in that, The first hash grid encoder and the second hash grid encoder are hash grid encoders with the same shared resolution.

8. An open-vocabulary 3D scene graph generation system for scene understanding, characterized in that, The method includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the open-vocabulary 3D scene graph generation method for scene understanding as described in any one of claims 1-7.

9. A computer storage medium, characterized in that, It stores a computer program that is executed to implement the open vocabulary 3D scene graph generation method for scene understanding as described in any one of claims 1-7.

Images