A 3D Gaussian-based three-dimensional scene segmentation and interaction method

Abstract

This invention discloses a 3D scene segmentation and interaction method based on 3D Gaussian, comprising: Step 1, instance discovery: inputting 3D Gaussian scene data, dividing the 3D Gaussian points into structurally coherent instance-level Gaussian groups to obtain refined instances; Step 2, instance and scene semantic assignment: selecting representative viewpoint images of refined instances and inputting them into a visual-language model to generate semantic description labels for the instances; filtering instance pairs in the scene through spatial geometric relationships, and clarifying the spatial relationship description of instance pairs through a large model, constructing a static scene graph that integrates geometric proximity relationships and instance semantic relationships, forming a structured description of all instances; Step 3, natural language-driven instance localization and interaction: receiving user input commands, combining the instance semantic description, the static scene graph, and the real-time viewpoint direction relationship during the query to perform multi-dimensional matching, determining the target instance ID, and executing interactive operations.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and embodied intelligence, and more particularly to a 3D scene segmentation and interaction method based on 3D Gaussian. Background Technology

[0002] With the development of 3D reconstruction technology and artificial intelligence, how to achieve a deep understanding of 3D scenes and natural language interaction has become a core issue in fields such as robot navigation, augmented reality, and digital twins.

[0003] Currently, 3D scene representation has evolved from traditional point clouds and meshes to neural implicit / explicit representations, such as neural radiation fields and 3D Gaussian splashing. Among them, 3DGS has become the mainstream solution for 3D reconstruction due to its superior rendering quality and real-time inference speed. However, the original 3DGS scene only contains spatial geometry and color information, lacking semantic instance understanding, which prevents users from interacting with specific objects in the scene through natural language.

[0004] Existing technologies, when attempting to address the aforementioned problems, typically suffer from the following limitations: Firstly, instance segmentation is inaccurate. Some solutions attempt to project 2D segmentation results into 3D space, but due to the lack of strict geometric constraints in 3DGS, this often leads to blurred instance edges, severe background noise interference, or misclassification of semantically similar but spatially separated objects. Secondly, spatial relationship modeling is lacking. Existing scene segmentation understanding largely remains at the level of isolated object recognition, lacking modeling of high-level semantic relationships such as physical, structural, and functional relationships between instances. Especially when dealing with viewpoint-related directional descriptions, existing methods often cannot adapt to dynamic camera pose changes. Furthermore, due to the lack of a hierarchical semantic graph structure, existing systems struggle to support complex spatial semantic commands, resulting in insufficient accuracy and robustness in interactions.

[0005] Therefore, how to realize a system that can automatically discover high-quality instances without manual annotation, build stable semantic relationship chains, and support natural language interaction under dynamic perspectives is a technical problem that urgently needs to be solved in the field of 3D scene understanding. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a high-precision, viewpoint-independent, and highly interpretable 3D scene segmentation and interaction method to address the shortcomings of the existing technology. It obtains reliable instance units through soft assignment clustering with semantic-geometric joint optimization, constructs a geometrically invariant scene graph, and combines LLM structured reasoning and CLIP cross-modal matching to achieve accurate and robust instance localization and interaction driven by user commands.

[0007] To address the aforementioned technical problems, this invention discloses a 3D scene segmentation and interaction method based on 3D Gaussian, characterized by the following steps:

[0008] Step 1, Instance Discovery: Input 3D Gaussian scene data, divide the 3D Gaussian points into structurally coherent instance-level Gaussian groups, and perform quality assessment, denoising and merging processing on the segmentation results to obtain structurally coherent and accurate refined instances.

[0009] Step 2, Semantic Assignment of Instances and Scenes: Select representative perspective images of refined instances and input them into the visual-language model to generate semantic description labels for the instances; filter instance pairs in the scene through spatial geometric relationships, and clarify the spatial relationship description of instance pairs through a large model to construct a static scene graph that integrates geometric proximity relationships and instance semantic relationships, forming a structured description of all instances.

[0010] Step 3, Natural Language-Driven Instance Localization and Interaction: Receive user input commands, combine the instance semantic description, static scene graph, and real-time viewpoint direction relationship during query to perform multi-dimensional matching, determine the target instance ID, and execute interactive operations.

[0011] Specifically, each Gaussian point in the 3D Gaussian scene data in step 1 contains a 3D spatial location and a semantic feature vector. The method for obtaining this data is as follows:

[0012] Step 1), use the original 3D Gaussian to reconstruct the visual scene.

[0013] Step 2) The input image is processed by the SAM model to obtain a two-dimensional segmentation map, and the two-dimensional segmentation map is used as the target for comparative learning on 3D Gaussian data.

[0014] Step 2) requires adding eigenvalues ​​to the original Gaussian data and adding an eigenvalue rendering function, which replaces the color values ​​in the original visual rendering formula with eigenvalues.

[0015] Furthermore, the clustering instance discovery in step 1 includes the following steps:

[0016] Step 1-1: Use the K-means algorithm as the basic method for clustering to initialize 256 instance prototype vectors, i.e. instance prototype features;

[0017] Steps 1-2: The similarity between the feature vector of Gaussian elements and the prototype features of instances is used as a semantic matching term. A geometric penalty term is calculated by combining the spatial location of Gaussian points with the distance between the geometric center of the instance. Only Gaussian points that exceed a certain range are penalized to obtain better edge cutting results.

[0018] Steps 1-3 integrate semantic matching terms and set penalty values ​​as a clustering formula. A soft-assignment method is used to calculate the contribution of Gaussian points to each cluster instance, mitigating the impact of boundary Gaussian points and noisy data on clustering. Iterative updates continue until convergence, and hard instance segmentation is performed using the maximum membership principle to obtain the final instance results.

[0019] Steps 1-4 evaluate the quality of each instance obtained in Steps 1-3. On one hand, the spatial variance and number of points in the instance point cloud are calculated to filter out noisy instances caused by poor Gaussian reconstruction quality, merging them into surrounding instances with high similarity. On the other hand, highly similar and neighboring instances in the scene are merged to mitigate the over-segmentation problem caused by the large number of initial K-means clustering points, resulting in the final refined instances. After this, the storage format of scene semantic features changes; Gaussian points only need to store their instance IDs, and the semantic feature vectors corresponding to the instance IDs are saved through a codebook.

[0020] Furthermore, the specific steps for assigning instance and scene semantics in step 2 include:

[0021] Step 2-1, generating a detailed description of a single instance: Perform feature rendering on all viewpoints, and obtain a visibility map by calculating the pixel area occupied by all instances under each viewpoint. Select the viewpoint images corresponding to the top k largest visible areas as representative viewpoint images of the instance;

[0022] Step 2-2: Input representative perspectives into the visual-language model to obtain structured descriptions of instances, containing only adjectives and nouns. The latter are used to locate the essence of the instance, while the former gives it uniqueness. Calculate the mean of the structured descriptions from k perspectives to find the semantic centroid, and use the description closest to the centroid as the final semantic description label.

[0023] Steps 2-3: Filter instance pairs with potential geometric relationships using the geometric location information of the instances (only generate edges for two instances whose distance is less than a threshold, i.e., model the spatial relationship between them); use the same selection method in 2-2 to calculate the area where the two instances coexist, find the optimal viewpoint, and further clarify the spatial relationship of the instances by generating a relationship description through a large model. This description should not contain relative orientation descriptions such as front, back, left, right, etc.

[0024] In summary, the structured description in step 2 includes a unique instance ID, semantic value, natural semantic description, center point coordinates, instance scale, and relationships between instances. During rendering, the semantic value is queried using the instance ID in the Gaussian data, and the input is used to obtain the semantic graph.

[0025] Furthermore, the natural language-driven instance localization and interaction in step 3 specifically includes the following steps:

[0026] Step 3-1: For the natural language command input by the user, call the large model to perform reasoning and parsing, and divide the user's query command into four parts: query target, reference target, positional relationship, and relative relationship; if there is none, then it is considered as none.

[0027] Step 3-2: First, filter the query targets. To cover all queries, the targets need to be divided into three levels: conceptual queries, fuzzy queries, and exact queries. The query level is determined through large-scale model reasoning. Conceptual queries are those lacking clear definitions, such as "green" or "cute." Fuzzy queries are defined as those containing only the query object but lacking detailed descriptions, such as "apple" or "chair." Exact queries are those containing detailed descriptions, such as "red pillow." Different levels mean using different relevance thresholds for filtering.

[0028] The process of filtering the query target in step 3-2 includes: calculating the cosine similarity between the description of the query target and the description of the instance, and filtering out instances with a similarity greater than a threshold for the next layer of matching.

[0029] Step 3-3: For the objects filtered in 3-2, continue to query whether they contain a specified reference relationship, that is, whether they have a corresponding positional relationship with the reference target. The relevance threshold is also used for calculation. Finally, if a relative position exists, project the instance center from that viewpoint onto the current camera coordinate system and determine whether relationships exist in the front-back, left-right, and other directions.

[0030] Specifically, the visual-language model used in step 2 and the model used for inference in step 3 are both multimodal deep neural network models based on the Transformer autoregressive encoder-decoder. They can be deployed locally or call remote APIs. Reference series include GPT, Claude, Gemini, and Qwen-VL. Step 2 uses the model to describe the input and output of images, while step 3 uses the model for inference, filtering and selecting user language commands to understand user intent and accurately locate instances.

[0031] Furthermore, in step 3, during the filtering of target instances, all description pairings are similarity calculated by mapping the language to the vector space using CLIP. The large model here only serves as a tool for invocation and logical reasoning.

[0032] Beneficial Effects: The method provided by this invention offers a way to understand virtual 3D scenes through segmentation and semantic construction of 3D Gaussian scenes. It not only accurately segments instances within the scene but also endows these instances with natural language-level descriptions for open semantic queries. Furthermore, by combining a multimodal large model for structured parsing of language instructions, it improves the ability to accurately respond to interactive commands in complex 3D scenes. Attached Figure Description

[0033] Figure 1 The flowchart illustrates a 3D scene segmentation and interaction method based on 3D Gaussian provided by this invention.

[0034] Figure 2 This is a flowchart of the clustering instance segmentation training process provided by the present invention.

[0035] Figure 3 A flowchart of the language instruction interaction method provided by the present invention.

[0036] Figure 4 A scene diagram of one embodiment of the present invention.

[0037] Figure 5 This is a diagram showing the segmentation result in an embodiment of the present invention.

[0038] Figure 6 This is a comparison diagram of different query statements in an embodiment provided by the present invention. Detailed Implementation

[0039] This invention provides a 3D scene segmentation and interaction method based on 3D Gaussian, comprising the following steps:

[0040] Step 1, instance discovery: Input 3D Gaussian scene data. Each Gaussian point contains a 3D spatial location and semantic feature vector. A two-stage clustering algorithm with semantic dominance and geometric consistency penalty is adopted to divide the 3D Gaussian points into structurally coherent instance-level Gaussian groups. The segmentation results are then subjected to quality evaluation, denoising and merging to obtain structurally coherent and accurate refined instances.

[0041] Step 2: Instance and Scene Semantic Assignment. A representative viewpoint image of an instance is selected and input into the visual-language model to generate a semantic description label for that instance. Instance pairs in the scene are filtered through spatial geometric relationships, and the spatial relationship description of instance pairs is clarified through a larger model. A static scene graph integrating geometric proximity relationships and instance semantic relationships is constructed, forming a structured description of all instances.

[0042] Step 3: Natural language-driven instance localization and interaction. Receive user input commands, combine the instance semantic description, static scene graph, and real-time viewpoint direction relationship during query to perform multi-dimensional matching, determine the target instance ID, and execute interactive operations.

[0043] This embodiment selects the classic scene Playroom from the Deep Blending dataset (the Playroom dataset is a dataset of images from various perspectives within a room) to describe in detail a specific implementation of the present invention. For example... Figure 4As shown, the scene contains dense small objects and strong occlusion, which is suitable for verifying the effectiveness of the present invention in complex interactive scenarios.

[0044] The acquisition of the 3D Gaussian scene data includes:

[0045] Step 1) Reconstruct the visual scene using the existing original 3D Gaussian technique;

[0046] Step 2) Process the Playroom dataset images using the SAM (Segment Anything Model) model to obtain the two-dimensional segmentation map of the dataset, and use the two-dimensional segmentation map as the target to perform comparative learning on 3D Gaussian data.

[0047] Step 2) requires adding semantic feature values ​​to the data obtained from visual scene reconstruction of the original Gaussian model, and adding a feature value rendering function, which replaces the color values ​​in the original visual rendering formula with feature values.

[0048] Each Gaussian point in the 3D Gaussian scene data contains a 3D spatial location and a semantic feature vector;

[0049] After the basic 3D reconstruction and comparative learning described above, the 3D Gaussian scene data for this scenario should include the original Gaussian data and a newly added 16-bit float32 semantic feature vector. For example... Figure 2 As shown, the Gaussian data in step 1 consists of Gaussian primitives and their corresponding point-level features. Next, clustering is used to extract instances.

[0050] Step 1-1: First, randomly initialize 256 instances in the feature space as instance prototypes. The feature vector of the instance prototype is the vector value of the initialized instance. The instance prototype is the initialized instance.

[0051] Steps 1-2: The cosine similarity between the Gaussian feature vector and the instance prototype features is used as the semantic matching term. Simultaneously, a geometric penalty term is calculated by combining the spatial location of the Gaussian point with the center location of the instance prototype. Specifically, the formula is as follows:

[0052]

[0053] in Let i be the eigenvector of the i-th Gaussian element. Let be the feature vector of the prototype of the k-th instance. Let be the coordinates of the i-th Gaussian point, and Let the coordinates be the center of the k-th instance. This is the adaptive boundary radius of the instance, which is also generated during the learning process. These are hyperparameters used to adjust the effect of the geometric penalty term. Final result. This is the clustering score of Gaussian point i for instance k.

[0054] Steps 1-3: During clustering learning, a two-stage method is used for instance assignment: In the learning phase, a soft assignment metric is used, which calculates the probability (contribution) that each Gaussian point belongs to a particular instance. The update calculation for instances is based on contribution. The specific formula is as follows:

[0055]

[0056] in This is a temperature adjustment coefficient used to adjust the sharpness of the assignment. After training, the assignment is performed directly based on the maximum probability, i.e., the argmax value is taken. The indices i and k represent the i-th Gaussian unit and the k-th instance, respectively, and K is the total number of instances. This represents the joint fraction calculated in steps 1-2.

[0057] Steps 1-4: After learning, the instances from this stage need further optimization. All instances need to be assessed for two issues: whether they are noise points and whether they have been over-segmented. For the former, Gaussian point clouds often exhibit a discrete appearance and a small number of instances. Therefore, Gaussian instances with variance greater than the variance threshold and fewer instances than the number threshold are merged into the nearest surrounding Gaussian instances. In this example, the number threshold is 500, and the variance threshold is 0.4 times the mean variance of all instances. Similarly, for the latter, it is also determined whether there are instances with a similarity higher than the threshold of 0.9, and these are then merged. Figure 2 The entire process of step 2 is demonstrated. Meanwhile... Figure 5 A segmentation rendering effect from one perspective is provided. After this, the storage format for scene semantic features changes to: Gaussian points only need to store their instance IDs, and the semantic feature vectors corresponding to the instance IDs are saved through the codebook.

[0058] The soft-assignment training method in step 1 yields smoother instance segmentation boundaries, which aligns with the inherent lack of instance discriminativeness in 3D Gaussian rendering and effectively addresses the sensitivity of K-means initialization. Furthermore, the clustering formula incorporating geometric information complements the inherent spatial information of the 3D scene with the trained semantic information, effectively resolving the potential edge blending problem inherent in contrastive learning. The instance-forming codebook significantly reduces storage costs and allows instances to naturally form a unified whole without requiring manual boundary adjustments.

[0059] Step 2: Use a large language model to perform semantic labeling on instances in the scene.

[0060] Step 2-1: First, generate a detailed description of a single instance. Specifically, obtain a visibility map by calculating the pixel area occupied by all instances in each viewpoint. For each instance, select its top 5 most visible viewpoints, and gray out the remaining pixels;

[0061] Step 2-2: Send the processed image to the large model for description. Here, the large model needs to generate an adjective + noun combination to describe only the instance itself. For the results from the five viewpoints, convert them into feature vectors using CLIP (Contrastive Language–Image Pre-training). Calculate the mean of the five vectors, which is the semantic centroid, and select the result closest to this mean based on semantic similarity as the final semantic description of the instance. For example... Figure 4 The red cushion on the right side of the image is specifically described as a "red bean bag." "Red" is its adjective characteristic, while "bean bag" indicates its item type.

[0062] Steps 2-3: After obtaining the instance descriptions, a basic understanding of the scene needs to be generated, i.e., the relationships between instances need to be generated. First, basic graph filtering is performed based on the geometric positional relationships of the instances themselves, generating edges only for instance pairs whose distance is below a threshold, thus modeling a spatial relationship between them. Then, following the method in step 2-2, the optimal perspective for each instance pair is generated and fed into a large language model to generate a spatial relationship description without relative positional relationships. Similarly... Figure 4 In the example, the relationship between the instance "red bean bag" and the instance "blackcarpet" is "On", while the relationship between the instance "red carpet" and the instance "Cabinet" is "Near".

[0063] Unlike common CLIP-based description generation methods, the scene semantics generation results combined with a large model have a more regular structure and interpretability. It is more stable in threshold calculations at different levels and can form complete query logic. Furthermore, the introduction of a scene graph connects instances throughout the scene, enriching the query methods.

[0064] Step 3 uses a large language model to perform semantic queries in conjunction with the entire process described above. The query process is as follows: Figure 3 As shown.

[0065] Step 3-1: First, use a large language model to filter the input query statement. Filter out the four parts: query target, reference target, positional relationship, and relative relationship. If none are found, return "None". For example, in the query statement "pick up thechair near table", the query target is clearly "chair", while the reference target is "table". Their positional relationship is "near". The statement does not contain relative relationships such as "before", "after", "left", or "right", so the relative relationship is "None".

[0066] Step 3-2: For query targets, to cover different levels of query strength, three levels are established: conceptual query, fuzzy query, and precise query. This level determination is also based on a large model. For example, when querying "bag," no specific type of "bag" is specified, so it's classified as a fuzzy query, and the threshold is adjusted to 0.85. Querying "red bean bag," however, is a precise query, specifying the object's characteristics. The relevance threshold is changed to 0.95, while for a conceptual query it's 0.75. After obtaining the specific query target and relevance threshold, all instances in the scene are matched. This involves calculating the cosine similarity between the query target description and the instance descriptions, filtering out instances with similarity greater than the threshold for the next level of matching. For example... Figure 6 The second line provides a comparison of results with different search precision. When searching for "red carpet," the cosine similarity of "black carpet" in the scene is 0.9023, which is filtered out by the precise search threshold of 0.95, while the completely identical "red bean bag" is selected. Similarly, when searching for "carpet," a fuzzy search with a threshold of 0.85, both "red carpet" and "yellow carpet" have a similarity of 0.8590, which allows them to be selected, consistent with the search logic.

[0067] Step 3-3: When a reference relationship exists, filter instances that meet the reference requirements in the scene graph. Furthermore, if a relative positional relationship exists, project the center point coordinates of the instance pair onto the camera coordinate system for judgment. Figure 6 The first line presents a comparison of query results with positional relationships. When searching for "chair", all chair instances are filtered out following steps 3-1 and 3-2. However, when searching for "chair on the table left", step 3-1, based on the large model parsing, determines the relative relationship as "left" with the table as the reference object. Therefore, the spatial coordinates of these three instances are mapped to the current viewpoint to determine if the position is "left" (i.e., if the x-axis coordinate is in the negative direction of the reference instance). This then filters out the instances on the left.

[0068] Through the above steps, a complete closed-loop process from scene segmentation to semantic description generation and instance querying was completed. Gaussian features were elevated from continuous point-level data to discrete codebooks and then to scene graphs with realistic semantics. This completed the semantic reconstruction process of the 3D scene. Items in the 3D scene possess realistic natural descriptive semantics and instance attribution, enabling the selection of instances from the scene by their IDs and precise item selection using natural language. This lays the groundwork for subsequent Gaussian scene editing and future embodied intelligence-based real-world scene interaction and applications.

[0069] This invention provides a 3D scene segmentation and interaction method based on 3D Gaussian. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A 3D scene segmentation and interaction method based on 3D Gaussian, characterized in that, Includes the following steps: Step 1, Instance Discovery: Input 3D Gaussian scene data, divide the 3D Gaussian points into structurally coherent instance-level Gaussian groups, and perform quality assessment, noise reduction and merging processing on the division results to obtain refined instances; Step 2, Semantic Assignment of Instances and Scenes: Select representative perspective images of refined instances and input them into the visual-language model to generate semantic description labels for the instances; filter instance pairs in the scene through spatial geometric relationships, and clarify the spatial relationship description of instance pairs through a large model to construct a static scene graph that integrates geometric proximity relationships and instance semantic relationships, forming a structured description of all instances. Step 3, Natural Language-Driven Instance Localization and Interaction: Receive user input commands, combine the instance semantic description, static scene graph, and real-time viewpoint direction relationship during query to perform multi-dimensional matching, determine the target instance ID, and execute interactive operations.

2. The 3D scene segmentation and interaction method based on 3D Gaussian as described in claim 1, characterized in that, Each Gaussian point in the three-dimensional Gaussian scene data in step 1 contains a three-dimensional spatial location and a semantic feature vector.

3. The 3D scene segmentation and interaction method based on 3D Gaussian as described in claim 2, characterized in that, The instance-level Gaussian grouping in step 1 is performed using a two-stage clustering algorithm that is semantically dominant and introduces a geometric consistency penalty. Specifically, it includes the following steps: Step 1-1: Randomly initialize n instances in the feature space as instance prototypes; Steps 1-2: The similarity between the feature vector of Gaussian elements and the prototype features of instances is used as a semantic matching term. A geometric penalty term is calculated by combining the spatial location of Gaussian points with the distance between the geometric center of instances, and Gaussian points that exceed a certain range are penalized. Steps 1-3: Integrate semantic matching terms and geometric penalty terms as clustering formulas, calculate the contribution of Gaussian points to each cluster instance using soft assignment, iterate and update until convergence, and perform hard instance segmentation using the maximum membership principle to obtain the final instance result. Steps 1-4 evaluate the quality of each instance obtained in Steps 1-3: On the one hand, calculate the spatial variance and number of instance point clouds, filter out noisy instances caused by the quality of Gaussian reconstruction, and merge them into surrounding instances with high similarity. On the other hand, merge instances with high similarity and proximity in the scene to alleviate the oversegmentation problem caused by the number of K-means cluster initializations and obtain the final refined instances. After this, the storage form of scene semantic features is transformed into Gaussian points only need to store their instance IDs, and the semantic feature vectors corresponding to the instance IDs are saved through the codebook.

4. The 3D scene segmentation and interaction method based on 3D Gaussian as described in claim 1, characterized in that, The structured description in step 2 specifically includes a unique instance ID, semantic value, natural semantic description, center point coordinates, instance scale, and relationships between instances.

5. The 3D scene segmentation and interaction method based on 3D Gaussian as described in claim 1, characterized in that, Step 2, which assigns semantic meaning to instances and scenarios, specifically includes the following steps: Step 2-1, generating a specific description of a single instance: Perform feature rendering on all viewpoints, calculate the pixel area occupied by all instances under each viewpoint to obtain a visibility map, and select the viewpoint images corresponding to the top k largest visible areas as representative viewpoint images of the instance. Step 2-2: Input the representative viewpoint image into the visual-language model to obtain the structured description of the instance. Calculate the mean of the structured descriptions under k viewpoints to find the semantic centroid. Take the description closest to the centroid as the final semantic description label. Steps 2-3: Construct instance pairs with potential geometric relationships using the geometric location information of the instances; using the same method as in 2-1, calculate the common pixel area of ​​each instance pair, find the optimal viewpoint, and further clarify the spatial relationships of the instances by generating a relationship description through a large model.

6. The 3D scene segmentation and interaction method based on 3D Gaussian as described in claim 5, characterized in that, The instance pairs for constructing potential geometric relationships described in steps 2-3 specifically involve generating an edge for two instances whose distance is less than a threshold, i.e., modeling the existence of a spatial relationship between them.

7. The 3D scene segmentation and interaction method based on 3D Gaussian as described in claim 1, characterized in that, Step 3, natural language-driven instance localization and interaction, specifically includes the following steps: Step 3-1: For the natural language command input by the user, call the large model to perform reasoning and parsing, and divide the user's query command into four parts: query target, reference target, positional relationship, and relative relationship; if there is none, then it is considered as none. Step 3-2: First, filter the query targets. To cover the query, the query targets are divided into three levels: concept query, fuzzy query, and precise query. The query level is determined by large model reasoning, and different relevance thresholds are used for instance filtering at different levels. Step 3-3: For the instance objects filtered in 3-2, continue to query whether there is a corresponding positional relationship with the reference target. Finally, if there is a relative position, project the center of the filtered instance onto the current camera coordinate system to further determine whether the directional relationship exists.

8. The 3D scene segmentation and interaction method based on 3D Gaussian as described in claim 7, characterized in that, The process of filtering the query target in step 3-2 includes: calculating the cosine similarity between the description of the query target and the description of the instance, and filtering out instances with a similarity greater than a threshold for the next layer of matching.

9. A 3D scene segmentation and interaction method based on 3D Gaussian as described in claim 5, characterized in that, The structured description of the instance described in step 2-2 contains only adjectives and nouns.

10. A 3D scene segmentation and interaction method based on 3D Gaussian as described in claim 1, characterized in that, The visual-language model used in step 2 and the large model used for reasoning in step 3 are both multimodal deep neural network models based on the Transformer autoregressive encoder-decoder.

Images