Open vocabulary 3d object query method based on semantic guided geometric reasoning

Abstract

This invention discloses an open-vocabulary 3D object query method based on semantic-guided geometric reasoning. First, a representative viewpoint selection step is designed, utilizing a viewpoint entropy-based method to select representative views, significantly optimizing the performance bottleneck of subsequent multimodal large language models. Second, an interactive module is designed to inject sparse semantics into multi-level geometric features, effectively integrating sparse semantic features into geometric features with different spatial granularities. Next, a 3D scene response is performed, decoding and generating 3D Gaussian parameters and corresponding relevance scores for the scene. Finally, based on a relevance score threshold, a set of 3D Gaussian parameters most relevant to the query term is selected, achieving accurate 3D object query without explicit feature alignment.

Description

Technical Field

[0001] This invention relates to an open-vocabulary 3D object query method and system based on semantic-guided geometric reasoning, belonging to the field of 3D scene understanding and artificial intelligence technology, and particularly to a 3D spatial target localization task that utilizes a multimodal large language model to guide geometric feature extraction. Background Technology

[0002] Open-vocabulary 3D query technology has recently attracted widespread attention, aiming to locate objects in 3D scenes based on arbitrary text prompts. Existing methods, such as Large-Scale Spatial Model (LSM), Unified 3D Reconstruction and Understanding (Uni3R), and 3D Language Gaussian Splashing (LangSplat), mostly adopt a feature alignment paradigm, that is, by rendering 3D feature maps and explicitly aligning them with the dense features generated by pre-trained 2D segmentation models, such as CLIP, SAM, or LSeg, in the semantic space.

[0003] However, research has revealed significant limitations to this paradigm: First, the 2D segmentation model lacks explicit awareness of 3D structure; forced alignment compels 3D features to approximate the 2D semantic space, thus suppressing the original fine-grained geometric signals in the 3D representation, leading to ambiguous semantic responses and decreased segmentation quality. Second, when processing multi-view scenes, independently performing large-scale model inference for each viewpoint generates severe computational redundancy. Therefore, how to efficiently and accurately handle complex open-lexical queries while preserving the original geometric fidelity of the 3D scene remains a pressing challenge in the fields of spatial intelligence and robot perception. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an open-vocabulary 3D object query method based on semantic-guided geometric reasoning. This method guides 3D geometric reasoning through high-level abstract semantic features, achieving accurate 3D object querying without explicit feature alignment.

[0005] An open-lexical 3D object query method based on semantic-guided geometric reasoning includes the following steps:

[0006] For poseless multi-view image inputs, representative views are first selected.

[0007] Based on representative views and input queries, sparse semantic features are obtained using a multimodal visual large language model.

[0008] Then, multi-level geometric features are extracted. Sparse semantic features are injected into the multi-level geometric features through an interactive module to obtain semantically enhanced geometric features.

[0009] The above features are used to decode and generate 3D Gaussian parameters of the scene and a relevance score of each 3D Gaussian parameter to the query term. A set of 3D Gaussian parameters most relevant to the query term is then selected based on the relevance score threshold.

[0010] Finally, the query model is trained: During the training process, the correlation score attribute of the Gaussian parameter is rendered into a 2D image by using differentiable rendering technology to calculate the loss, and then the parameters of the update module are trained through backpropagation.

[0011] Furthermore, the specific filtering process is as follows:

[0012] All poseless multi-view image inputs in the scene are traversed, and a method based on view entropy is used for filtering. Specifically, the Shannon entropy calculated based on the image gray intensity distribution is used for all poseless multi-view image inputs, and the view with the smallest entropy value is selected as the representative view.

[0013] Furthermore, the multimodal large language model adopts the LISA model; based on the input text, high-level abstract semantic features are extracted on representative views as sparse semantic features for all poseless multi-view image inputs.

[0014] Furthermore, a pre-trained AnySplat model is used as the 3D reconstruction model to capture cross-view consistency from poseless multi-view image inputs, thereby generating multi-level geometric features.

[0015] Furthermore, the interaction module adopts a multi-level semantic injection and feature fusion strategy. By using sparse semantic features, semantic injection is performed at each level of the multi-level geometric features to finally obtain semantically enhanced geometric features.

[0016] Furthermore, the specific three-dimensional scene response is as follows:

[0017] A fully decoupled attribute decoding and multi-level semantic fusion strategy is adopted. Multi-level geometric features are input into a geometric decoder and an appearance decoder to decode the geometric and appearance attributes of the 3D Gaussian parameters. Semantically enhanced geometric features are then input into a semantic decoder to decode the relevance score of the 3D Gaussian parameters to the query term. The geometric attributes of the Gaussian parameters are the Gaussian center position, and the appearance attributes are the Gaussian scaling, rotation, color, and opacity attributes.

[0018] Furthermore, to train the model, a training strategy is adopted that projects 3D objects onto 2D images before calculating the loss. Specifically, based on the decoded 3D Gaussian parameters, differentiable rendering techniques are used to calculate the relevance score of the 3D Gaussian parameters to the query term. As features, the rendering yields the 2D segmentation prediction result, i.e., the target segmentation mask. :

[0019]

[0020] in, The target mask obtained from rendering represents the 3D query prediction result based on the query text. The opacity attribute represents the i-th Gaussian parameter. This represents the total number of all Gaussian parameters.

[0021] The network parameters are jointly optimized using binary cross-entropy loss and Dice coefficient-based loss, and the two losses are given different weights to balance them.

[0022] Secondly, embodiments of this application provide an electronic device, including a processor and a memory;

[0023] The memory is used to store computer programs.

[0024] When the processor executes the program stored in the memory, it implements any of the open vocabulary three-dimensional object query methods described in this application.

[0025] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the open-vocabulary three-dimensional object query methods described in this application.

[0026] Fourthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the open vocabulary 3D object query methods described in this application.

[0027] The beneficial effects of this invention are as follows:

[0028] To enhance the model's ability to understand implicit human commands, this invention proposes an open-vocabulary 3D object query method based on semantically guided geometric reasoning for the novel task of open-vocabulary 3D reasoning. This method uses a feedforward approach to output an accurate 3D target mask based on a given implicit query text. Evaluation on the open-vocabulary 3D reasoning dataset re-annotated in this invention demonstrates superior performance in handling this type of task, achieving state-of-the-art results in the field. The proposed method can advance the frontier development of open-vocabulary 3D querying and is expected to find applications in a wide range of fields such as spatial intelligence and robotics. Attached Figure Description

[0029] Figure 1 This is a comparison between the embodiments of the present invention and conventional methods.

[0030] Figure 2 This is a complete flowchart of the method according to an embodiment of the present invention.

[0031] Figure 3 The results of testing the embodiments of the present invention and other existing methods on simple query text are shown.

[0032] Figure 4 The results of testing the embodiments of the present invention and other existing methods on complex query text are shown. Detailed Implementation

[0033] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] This invention addresses the aforementioned challenges through the following core components:

[0035] First, a representative viewpoint selection step was designed, utilizing a viewpoint entropy-based method to select representative views, significantly optimizing the performance bottleneck of the subsequent multimodal large language model. Second, an interactive module was designed to inject sparse semantics into multi-level geometric features, effectively integrating sparse semantic features into geometric features with different spatial granularities. Next, a 3D scene response was performed, decoding and generating 3D Gaussian parameters of the scene and the relevance score of each 3D Gaussian parameter to the query term. Finally, a set of 3D Gaussian parameters most relevant to the query term was selected based on a relevance score threshold.

[0036] like Figure 2 As shown, the open-vocabulary 3D object query method based on semantic-guided geometric reasoning has the following specific steps:

[0037] Step 1: Selection of representative perspectives:

[0038] For poseless multi-view image input, a view entropy-based method is used to select representative views. By identifying information-rich and structurally clear viewpoints, high-quality, low-noise input signals are provided for subsequent sparse semantic reasoning, while reducing computational redundancy.

[0039] Multimodal visual large language model (MLLM) can locate corresponding targets in various views based on implicit query text. However, using all existing views and relying on MLLM for implicit information reasoning leads to a significant computational burden. Given the existence of view similarity, semantic redundancy, and noise interference between different views in the same scene, this invention proposes that only a small set of representative views is needed to meet the sparse semantic reasoning requirements of MLLM. This reduces computational overhead while providing stable and informative signals for the reasoning process, improving reasoning robustness. Specifically, this involves traversing all poseless multi-view image inputs in the scene and using a view entropy-based method for filtering, as shown in the following formula:

[0040]

[0041] in, This represents the Shannon entropy calculated based on the image's grayscale intensity distribution. This represents a poseless, multi-view image input. express A specific image within the dataset. By calculating the view entropy of all poseless multi-view image inputs, the view with the smallest entropy value is selected. As a representative view, from a physical perspective, views with lower entropy values ​​typically exhibit a clearer structural layout and fewer blurry or cluttered areas. This effectively reduces semantic ambiguity in MLLM when handling complex 3D scenes and implicit queries, ensuring the accuracy of inference.

[0042] Step 2, Sparse semantic feature extraction:

[0043] Based on representative views and input queries, this invention utilizes a multimodal visual large language model (MLLM) for joint visual-language reasoning to obtain high-level abstract semantic features as sparse semantic features. Unlike traditional two-dimensional mask acquisition, this invention directly extracts high-level abstract semantic features from the deep layers of the model as sparse semantic features for all poseless multi-view image inputs, achieving deep semantic understanding of the target object.

[0044] The multimodal large language model adopts the xinlai / LISA-13B-llama2-v1 model; based on the input text. In representative view This invention extracts high-level abstract semantic features as sparse semantic features for all poseless multi-view image inputs. To obtain a high-level semantic representation of the scene and eliminate the geometric loss caused by explicit feature alignment, this invention proposes extracting semantic cues with a high level of abstraction, rather than traditional two-dimensional pixel-level masks. Specifically, semantic features are obtained through joint vision-language reasoning.

[0045]

[0046] in, This represents a multimodal large language model that performs cross-modal reasoning, i.e., by implicitly querying the text. and representative views As input, the model's deep semantic understanding capabilities are used to map natural language descriptions to the semantic space; This corresponds to the hidden layer related to segmentation in the last layer of the LISA model, namely the high-level abstract semantic features.

[0047] This semantic feature encodes deep semantic information of the selected view under open-vocabulary conditions. Compared to traditional pixel-level dense features, this high-level semantic abstraction offers greater compactness and robustness. By utilizing semantic features as guiding signals, this process can provide accurate semantic anchors for subsequent 3D spatial reasoning without drastically suppressing native 3D geometric features, thus ensuring high accuracy in 3D structure perception even in complex scenes and under implicit query conditions.

[0048] Step 3: Multi-level geometric feature extraction:

[0049] By utilizing a pre-trained 3D reconstruction model, multi-level geometric features for scene reconstruction are extracted from poseless, multi-view image inputs. These geometric features exhibit cross-view consistency and are able to capture the fine spatial structure of the scene.

[0050] This invention employs a pre-trained AnySplat model as the 3D reconstruction model, fully utilizing the pre-trained weights of the AnySplat model to accurately capture cross-view consistency from poseless multi-view image input, thereby generating high-quality, spatially faithful multi-level geometric features. The specific implementation process is as follows:

[0051]

[0052] in, Represents the geometric backbone network of AnySplat, used for inputting pose-free multi-view images. Direct extraction of multi-level geometric features .in Representative at the level The following multi-level geometric features, among which The number of poseless, multi-view image inputs. L represents the dimension of the geometric features, and L represents the total number of layers in the output of the geometric backbone network.

[0053] Unlike traditional methods that rely on explicit camera pose mapping, this invention extracts multi-level features through a geometric backbone network, which spontaneously captures the spatial topological relationships of the scene. These multi-level geometric features constitute a natural matrix for semantic propagation, ensuring that semantic guidance signals maintain spatial continuity throughout the 3D scene. This extraction method based on native geometric features effectively avoids the suppression of 3D geometric details by traditional feature alignment paradigms, providing structured support for the subsequent generation of accurate 3D query responses.

[0054] Step 4: Multi-level semantic injection and feature fusion:

[0055] The sparse semantic features obtained in step (2) are injected into the multi-level geometric features in step (3) through the interactive module to obtain semantically enhanced geometric features. A lightweight injection operator with non-shared parameters is used to enable the geometric features to have semantic perception capabilities at different scales.

[0056] This invention employs a multi-level semantic injection and feature fusion strategy to effectively integrate extracted high-level semantic features into geometric features with different spatial granularities, thereby enhancing the system's robustness in understanding complex 3D structures. Since the geometric features are organized in a multi-level structure, each level encodes the geometric information of the scene at different levels of abstraction. Therefore, by performing semantic injection at each level of the multi-level geometric features, the geometric features acquire semantic perception capabilities at different scales. The specific implementation process is as follows:

[0057]

[0058] in, It is a sparse semantic feature. hierarchical The multi-level geometric features below. This represents a lightweight interactive module implemented through a bidirectional cross-attention mechanism. The module employs a non-shared parameter design across different levels to adapt to the geometric feature distribution at different resolutions.

[0059] Through the interactive module Sparse semantic features are injected into multi-level geometric features to form semantically enhanced geometric features. This process achieves deep coupling between semantic cues and geometric details while maintaining the integrity of the underlying 3D structure. Ultimately, these semantically empowered geometric features can provide precise guidance for the assignment of semantic attributes to each geometric primitive in 3D space through consistent propagation across views.

[0060] Step 5: 3D Scene Response and Result Generation

[0061] The system utilizes multi-level geometric features to decode and generate 3D Gaussian parameters for the scene. Simultaneously, it uses semantically enhanced geometric features to decode the relevance score of each 3D Gaussian parameter to the query term. Based on the relevance score threshold, it selects the set of 3D Gaussian parameters most relevant to the query term, thereby achieving spatial alignment for accurate 3D target localization and segmentation to complete the 3D object query.

[0062] This invention employs a fully decoupled attribute decoding and multi-level semantic fusion strategy. Multi-level geometric features are input into a geometry decoder and an appearance decoder to decode the geometric and appearance attributes of 3D Gaussian parameters. The semantically enhanced geometric features are then input into a semantic decoder to decode the relevance score of the 3D Gaussian parameters to the query term. In this embodiment, the geometry decoder, appearance decoder, and semantic decoder are all DPT-structured decoders. The specific implementation process is as follows:

[0063]

[0064]

[0065]

[0066] Among them, Represents a geometry decoder. Represents the appearance decoder. Represents a semantic decoder. This indicates the location of the Gaussian center, i.e., a geometric property, where i represents the i-th 3D Gaussian parameter; The appearance properties of the Gaussian are represented by the scaling, rotation, color, and opacity properties of the Gaussian parameters, respectively. This represents the relevance score of the 3D Gaussian parameters to the query term.

[0067] By filtering based on the relevance score threshold, a set of 3D Gaussian parameters most relevant to the query term is obtained, thereby achieving precise 3D target localization and segmentation with spatial alignment to complete the 3D object query.

[0068] Step 6, Training and Assessment:

[0069] A query model consisting of a multimodal large-scale model, a 3D reconstruction model, an interaction module, a geometric decoder, an appearance decoder, and a semantic decoder is trained. During training, the multimodal visual large-scale language model and the 3D reconstruction model are frozen, while only the interaction module and the semantic decoder are optimized. Using differentiable rendering techniques, the relevance score attribute with Gaussian parameters is rendered as a 2D image, and the Bce and Dice losses are calculated with the ground truth values. The module parameters are then updated via backpropagation during training. Finally, the IoU and bIoU metrics are used to evaluate the model's performance.

[0070] Specifically, to train the model, a training strategy is adopted that projects 3D objects onto 2D images before calculating the loss. Specifically, this is based on the 3D Gaussian parameters before filtering according to a relevance score threshold. Differentiable rendering technology is used to obtain the relevance score of the query term using 3D Gaussian parameters. As features, the rendering yields the 2D segmentation prediction result, i.e., the target segmentation mask. :

[0071]

[0072] in, The target mask obtained for rendering represents the result based on the query text. The three-dimensional query prediction results.

[0073] This invention employs binary cross-entropy loss (BCE) and Dice coefficient-based loss (DICE) to jointly optimize network parameters, aiming to supervise the model in generating accurate open-vocabulary 3D query prediction results. The specific implementation process is as follows:

[0074]

[0075] in, Represents the truth value of the target mask; and These are the weight coefficients for the two loss functions, Represents the binary cross-entropy loss function. This represents the loss based on the Dice coefficient. In the example, our... and All values ​​are 0.5. This objective function aims to guide the model to effectively utilize propagated semantic cues and multi-level geometric representations. Through joint supervision of pixel-level prediction and global mask structure, the model learns how to leverage semantic guidance to fully exploit geometric information, thereby achieving high-precision open-vocabulary 3D query responses under complex natural language queries.

[0076] This invention evaluates open-vocabulary 3D inference performance on two benchmark datasets:

[0077] (1) ScanNet: It contains rich geometric structures and spatial relationships in indoor scenes. This invention adds complex combinatorial natural language query annotations to it. Compared with traditional category labels, this dataset emphasizes semantically rich and context-dependent pronouns and is designed to evaluate the ability to query open lexical terms under long sentence logic in 3D scenes.

[0078] (2) Hypersim: This dataset encompasses highly accurate synthetic indoor scenes with fine geometric details. Building upon existing target-level annotations, this invention introduces implicit query text with combinatorial and functional descriptions. This dataset rigorously tests the model's performance in matching abstract semantics with precise 3D spatial structures under complex lighting and high-fidelity geometric environments.

[0079] For evaluation, this invention re-annotated the ScanNet and Hypersim datasets, introducing a complex, combinatorial natural language query benchmark. The contributions of this invention include: (1) proposing a semantically guided geometric reasoning paradigm that does not require explicit feature alignment, effectively preserving the native geometric structure of 3D scenes; (2) proposing the LangVGGT framework, which achieves efficient and scalable open-vocabulary query by decoupling sparse semantic reasoning from dense geometric propagation; and (3) establishing a new 3D query benchmark containing complex semantic descriptions, with experiments demonstrating that this method has superior performance in handling complex instructions and preserving geometric details.

[0080] The specific configuration is as follows: This embodiment uses AnySplat to initialize the geometric feature backbone network and the geometric prediction head. For semantic understanding, LISA (LISA-13B-LLaMA2-v1) is used as a multimodal large language model to parse query text and locate targets in representative views. Multi-layer geometric features are fused with semantic cues through a bidirectional cross-attention mechanism, and a DPT-style decoder head is used to output 3D open-vocabulary query results.

[0081]

[0082] Table 1. Quantitative comparison results of open vocabulary queries for simple query terms on ScanNet and Hypersim datasets.

[0083] Under the target view, this method outperforms all benchmark methods for simple open-vocabulary text queries. Experimental results clearly demonstrate that this method can effectively transfer semantic information to the 3D domain. Compared to Dr.Splat, which requires optimization for specific scenes, this method completes scene reconstruction and open-vocabulary queries within a unified feedforward framework, avoiding scene-by-scene training and adaptation. Although LSM and Uni3r also support feedforward inference, their 3D understanding capabilities highly depend on feature alignment strategies, which are limited by the representational capabilities of the underlying 2D model LSeg. While this strong coupling contributes to semantic consistency, it limits the effective utilization of the rich, fine-grained geometric details provided by the geometric backbone network. Furthermore, native 2D segmentation methods such as LSeg and LISA often produce inconsistent predictions across views when applied to 3D scenes, leading to fragmented representations and making it difficult to achieve coherent 3D scene understanding.

[0084]

[0085] Table 2. Quantitative comparison results of open-vocabulary queries for complex query terms on ScanNet and Hypersim datasets.

[0086] As shown in Table 2, due to the increased complexity and combinatoriality of language, the performance of all evaluation methods declined when faced with complex open-vocabulary text queries. Despite these challenges, our method consistently outperformed all comparable methods on both datasets. On the ScanNet dataset, our method achieved a source-view IoU of 55.9% and a BIoU of 32.4%, significantly outperforming feedforward models such as Uni3r (IoU 30.0%, BIoU 12.8%). On the Hypersim dataset, our method also performed exceptionally well, achieving a source-view IoU of 45.9%. Notably, although our model used only simple text supervision during training, it effectively generalized to complex text queries during inference, demonstrating the strong robustness and generalization ability of this invention when handling complex natural language instructions.

[0087] Figure 3 This paper presents a comparative visualization of 3D inference under simple open-vocabulary text queries. In simple real-world scenarios, most benchmark methods such as Dr. Splat, LSM, Uni3R, and LISA can reasonably locate the target region. However, Dr. Splat, limited by scene-by-scene optimization, cannot complete reconstruction and querying within a unified feedforward framework like this method; while LSM and Uni3R, due to their over-reliance on feature alignment of the underlying 2D model LSeg, limit the effective utilization of fine-grained geometric information. In contrast, this method can effectively transform semantic information into a 3D representation, maintaining fine-grained spatial consistency while generating clearer segmentation mask boundaries.

[0088] Figure 4 This paper presents a comparative visualization of 3D inference results under complex and combinatorial natural language queries. As the complexity of the text description increases, the robustness differences between different methods become significantly amplified. Many benchmark methods exhibit blurred and incoherent segmentation masks and a large number of false positives within the target region when handling such queries, reflecting the sensitivity of CLIP-based visual-text alignment methods to linguistic combinatoriality and ambiguity. In stark contrast, our proposed method maintains spatially coherent and consistent predictions under complex queries, with compact masks and a significant reduction in false activations. Although our model is trained under simple text supervision, it demonstrates excellent generalization ability during inference, accurately understanding complex query intents.

[0089] This application also provides an electronic device, including a processor and a memory.

[0090] The memory is used to store computer programs.

[0091] When the processor executes a program stored in the memory, it implements any of the methods described in this application.

[0092] In one possible implementation, the electronic device of this application embodiment further includes a communication interface and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0093] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc.

[0094] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0095] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0096] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0097] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements any of the methods described in this application.

[0098] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the methods described in this application.

[0099] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0100] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] The various embodiments in this specification are described in a related manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other.

[0102] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An open-lexical 3D object query method based on semantic-guided geometric reasoning, characterized in that: Includes the following steps: For poseless multi-view image input, representative views are first selected; Based on representative views and input queries, sparse semantic features are obtained using a multimodal visual large language model; Then, multi-level geometric features are extracted; The sparse semantic features are injected into multi-level geometric features through the interactive module to obtain semantically enhanced geometric features; The above features are used to decode and generate 3D Gaussian parameters of the scene and the relevance score of each 3D Gaussian parameter to the query term; the set of 3D Gaussian parameters most relevant to the query term is obtained by filtering according to the relevance score threshold. Finally, the query model is trained: During the training process, the correlation score attribute of the Gaussian parameter is rendered into a 2D image using differentiable rendering technology to calculate the loss, and then the parameters of the update module are trained through backpropagation.

2. The open-vocabulary 3D object query method based on semantic-guided geometric reasoning according to claim 1, characterized in that, The specific filtering process is as follows: All poseless multi-view image inputs in the scene are traversed, and a method based on view entropy is used for filtering. Specifically, the Shannon entropy calculated based on the image gray intensity distribution is obtained for all poseless multi-view image inputs, and the view with the smallest entropy value is selected as the representative view.

3. The open-vocabulary 3D object query method based on semantic-guided geometric reasoning according to claim 2, characterized in that, The multimodal large language model adopts the LISA model; based on the input text, high-level abstract semantic features are extracted on representative views as sparse semantic features for all poseless multi-view image inputs.

4. The open-vocabulary 3D object query method based on semantic-guided geometric reasoning according to claim 1, characterized in that, A pre-trained AnySplat model is used as the 3D reconstruction model to capture cross-view consistency from poseless multi-view image input, thereby generating multi-level geometric features.

5. The open-vocabulary 3D object query method based on semantic-guided geometric reasoning according to claim 4, characterized in that, The interaction module adopts a multi-level semantic injection and feature fusion strategy. By using sparse semantic features, semantic injection is performed at each level of the multi-level geometric features to finally obtain semantically enhanced geometric features.

6. The open-vocabulary 3D object query method based on semantic-guided geometric reasoning according to claim 5, characterized in that, The specific 3D scene response is as follows: A fully decoupled attribute decoding and multi-level semantic fusion strategy is adopted. Multi-level geometric features are input into the geometric decoder and appearance decoder to decode the geometric and appearance attributes of the 3D Gaussian parameters. The semantically enhanced geometric features are input into the semantic decoder to decode the relevance score of the 3D Gaussian parameters to the query term. The geometric attributes of the Gaussian parameters are the Gaussian center position, and the appearance attributes of the Gaussian parameters are the scaling, rotation, color and opacity attributes of the Gaussian.

7. The open-vocabulary 3D object query method based on semantic-guided geometric reasoning according to claim 6, characterized in that, To train the model, a training strategy is adopted that projects 3D objects onto 2D images before calculating the loss. Specifically, based on the decoded 3D Gaussian parameters, a differentiable rendering technique is used to calculate the relevance score of the 3D Gaussian parameters to the query term. As features, the rendering yields the 2D segmentation prediction result, i.e., the target segmentation mask. : in, The target mask obtained from rendering represents the 3D query prediction result based on the query text. The opacity attribute represents the i-th Gaussian parameter. Represents the total number of all Gaussian parameters; The network parameters are jointly optimized using binary cross-entropy loss and Dice coefficient-based loss, and the two losses are given different weights to balance them.

8. An electronic device, characterized in that, Including processor and memory; The memory is used to store computer programs; When the processor executes the program stored in the memory, it implements the open vocabulary three-dimensional object query method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the open vocabulary three-dimensional object query method according to any one of claims 1-7.

10. A computer program product containing instructions, characterized in that, When it is run on a computer, it causes the computer to perform the open vocabulary three-dimensional object query method according to any one of claims 1-7.

Images