3D open vocabulary detection method based on cross-modal distillation and query guide sampling

Abstract

The invention provides a 3D open vocabulary detection method based on cross-modal distillation and query guide sampling, and the method comprises the steps: employing a pre-trained 2D open vocabulary detection model as a teacher network, and migrating vision-language knowledge to a 3D open vocabulary detection model as a student network through multi-level knowledge distillation; the method comprises the following steps: firstly, carrying out preprocessing and voxelization on an input point cloud, and extracting multi-scale voxel features; performing forward calculation through a teacher model to obtain 2D features, a bounding box and semantic information; realizing cross-modal knowledge migration by means of a triple distillation mechanism of feature hierarchy, semantic confidence and bounding box alignment; further, semantic and geometric priori are fused through a query guiding priori sampling mechanism, a mixed priori graph is generated, feature sampling is guided, and the position of a query point and a feature context are optimized in combination with deformable convolution; finally, 3D detection and visual semantic positioning results are output through hierarchical feature decoding, and the reasoning efficiency is remarkably improved while high detection precision is kept.

Description

A 3D Open Vocabulary Detection Method Based on Cross-Modal Distillation and Query-Guided Sampling Technical Field

[0001] This invention relates to the field of 3D computer vision, and more specifically to 3D open vocabulary detection technology. Background Technology

[0002] In fields such as robotics, autonomous driving, and augmented reality, 3D object detection technology plays a crucial role, enabling systems to understand and interact with the physical environment. An open vocabulary refers to a model's ability to detect and recognize objects of categories never seen during training. With increasingly complex application scenarios, the demand for 3D detection capabilities with open vocabulary has significantly increased. This means the system should be able to identify and locate object categories not present in the training set, allowing the task model to learn a more general visual concept that can be associated with language. This allows the task model to query and detect objects using any input text (i.e., an open vocabulary), even if the task model may have never seen the 3D point cloud data of the object corresponding to that text during training.

[0003] Existing 3D detection technologies are mainly divided into three categories: predefined category-based detection, pseudo-label-based methods, and large language model-based methods. Predefined category-based detection methods (such as VoteNet and GroupFree) are limited by the category set used during training and cannot handle unseen categories. Pseudo-label-based methods (such as OV-3DET and ImOV3D) map 2D detection results to 3D space, but their supervision signals suffer from noise and geometric inaccuracies, limiting detection accuracy. Large language model-based methods offer flexibility but are computationally inefficient and struggle to meet real-time requirements.

[0004] In the field of 3D open vocabulary detection, existing technologies face two major challenges: first, the quality of knowledge transfer. Existing methods often only perform label-level supervision or ignore geometric context when transferring knowledge from 2D pre-trained models to 3D space, resulting in insufficient knowledge transfer; second, computational efficiency. Existing methods typically need to process the dense point cloud of the entire scene when fusing text with 3D scenes, leading to high computational costs and slow inference speed. Specifically, while two-stage methods (such as EDA and ScanNet detection frameworks) have high accuracy, their large inference latency makes them unsuitable for real-time applications; single-stage methods (such as 3D-SPS and BUTD-DETR) have high computational efficiency but insufficient accuracy, especially in complex tasks such as 3D intent guidance and 3D vision guidance. Existing knowledge distillation methods lack structured supervision and cannot effectively support 3D bounding box detection; while direct feature alignment methods (such as CoDA and INHA) often ignore the rich diversity of linguistic expressions, such as synonyms, descriptions of different granularities, or contextual descriptions, limiting the system's generalization ability in open vocabulary scenarios. Meanwhile, most existing methods require a large amount of labeled 3D-text data for training, and the collection and annotation of such data is costly, which severely restricts the practical application of the technology.

[0005] Therefore, there is an urgent need for a 3D open vocabulary detection method that can simultaneously take into account accuracy, efficiency, and generalization ability, in order to solve the problems of the trade-off between accuracy and speed, insufficient knowledge transfer quality, and poor environmental adaptability in existing technologies. Summary of the Invention

[0006] The technical problem to be solved by this invention is to address the shortcomings of existing 3D open vocabulary detection methods, such as difficulty in balancing accuracy and real-time performance, low efficiency and high noise in 2D to 3D knowledge transfer, and insufficient generalization ability in complex language environments. This invention proposes a 3D open vocabulary detection method based on cross-modal distillation and query-guided sampling to achieve high-precision and high-efficiency 3D visual-language localization.

[0007] The technical solution adopted by this invention to solve the above problems is a 3D open vocabulary detection method based on cross-modal distillation and query-guided sampling. It utilizes a pre-trained 2D open vocabulary detection model as a teacher model, transfers structured visual-linguistic knowledge to a 3D student model through multi-level knowledge distillation, and combines a coarse-to-fine query-guided prior sampling mechanism to achieve efficient and accurate open vocabulary detection and localization of 3D point clouds without requiring 2D image input. Its features include the following steps:

[0008] Step 1: Cross-modal teacher-student distillation: Using a pre-trained 2D open vocabulary detection model (such as GroundingDINO) as a fixed teacher model, the 3D detection results are projected onto the 2D image plane through a camera projection model. The 3D student model is jointly distilled from three levels: features, semantics, and bounding boxes to construct a direct language-voxel semantic mapping relationship.

[0009] Step 2: Query-Guided Prior Sampling (QGPS) Step: Visual features and text embeddings are fused through a lightweight coarse-grained inference core to generate two priors: a semantic heatmap and a geometric objectivity score. After upsampling and fusion, a hybrid prior map is formed to guide efficient sampling of high-resolution voxel features, focusing on language-related regions.

[0010] Step 3: Hierarchical Feature Refinement and Decoding Step: The initial query points obtained by sampling are dynamically offset and adjusted using a deformable convolution module, high-level semantic context information is injected, and the final feature is decoded through a bidirectional encoder to output the detection and localization results.

[0011] Step 4: Two-stage training strategy: First, pre-train on a large RGB-D dataset (such as SUN RGB-D) using distillation loss to build basic open vocabulary capabilities; then fine-tune on downstream task datasets using detection loss to adapt to specific task requirements.

[0012] This invention ensures high-fidelity knowledge transfer from 2D to 3D by designing a multi-level (feature / semantic / localization) distillation loss function; the proposed QGPS method effectively overcomes the challenges of linguistic noise and geometric omissions through a complementary mechanism of semantic and geometric priors; the overall system adopts a single-stage, pure 3D reasoning architecture, which achieves end-to-end real-time detection while abandoning the 2D teacher model.

[0013] The beneficial effects of this invention are:

[0014] (1) It breaks through the dilemma of balancing accuracy and speed, and achieves real-time inference while maintaining high accuracy. Experimental results show that the zero-shot detection accuracy on the ScanNet dataset reaches 25.68 mAP@0.25 (surpassing the previous best level by +6.28), and after fine-tuning on the ScanRefer and Intent3D datasets, it reaches 56.81% Acc@0.25 (+0.36) and 61.48% Acc@0.25 (+3.14) respectively, while the pure model inference speed reaches 13.27 FPS.

[0015] (2) An efficient and robust knowledge transfer pathway was constructed. A semantic bridge from text to voxels was directly established through cross-modal distillation, avoiding the problems of high noise and weak generalization of traditional methods based on pseudo-labels or feature alignment.

[0016] (3) It has strong generalization and practicality capabilities. Its pure 3D reasoning process does not rely on 2D image input, which significantly improves the practicality and deployment flexibility of the system and effectively solves the problem of detecting and locating unknown categories of objects in complex language environments. Attached Figure Description

[0017] Figure 1 is a schematic diagram of the implementation method.

[0018] Figure 2 shows the heatmaps of the two priors obtained from the model. Blue represents the geometric prior, and red represents the semantic prior. Detailed Implementation

[0019] To clearly describe the proposed solution, the following terms and concepts will be explained.

[0020] Point cloud: A set of spatial points acquired by a 3D sensor. Each point cloud contains coordinates (x, y, z) and optional features. Optional features for visual semantic point clouds include color.

[0021] Voxel: A 3D pixel unit formed by meshing point clouds, which is the basic unit for processing 3D detection models.

[0022] Text Query: The embedding vector corresponding to the text description, used to guide 3D detection.

[0023] Distillation: In this invention, it refers to the process of transferring knowledge from a pre-trained 2D teacher model to a 3D student model.

[0024] I. Implementation Preparation

[0025] Implementing this invention requires deploying the corresponding software environment on a hardware platform with computing power, and preparing a 3D point cloud and multi-view image dataset that meets the requirements.

[0026] The following hardware and software environment is required to implement this method:

[0027] Hardware: At least one high-performance GPU, such as an NVIDIA RTX 2080 or higher, to support real-time processing of large-scale point cloud data. A minimum of 4GB of video memory is recommended to meet the memory requirements for sparse 3D convolution and cross-modal distillation computations.

[0028] Software: Requires a Python 3.8+ environment and the installation of core libraries such as PyTorch 1.10+, Torch, and mmdetection3D. Additionally, the HuggingFace Transformers library needs to be integrated to support text embedding processing, and the MinkowskiEngine utility library is required for sparse tensor operations.

[0029] Data: A multimodal dataset is required, including:

[0030] (1) Pre-training stage: SUN RGB-D dataset, which provides about 10,000 RGB-D scenes and their corresponding 3D bounding box annotations. It is necessary to ensure that the data contains pairwise matching point clouds and multi / single view images;

[0031] (2) Fine-tuning and evaluation stage: Indoor scene dataset ScanNet, natural language guided detection dataset ScanRefer, intent-driven localization dataset Intent3D, etc., the data must be preprocessed into a compatible point cloud format, such as PLY or bin file, and accompanied by multi-view images and text descriptions.

[0032] The dataset used in the fine-tuning and evaluation phase is any multimodal dataset compatible with the point cloud format.

[0033] II. Implementation Steps of the Invention

[0034] The overall process of this method is based on two core modules: cross-modal teacher-student distillation and query-guided prior sampling. Its system architecture is shown in Figure 1.

[0035] Step 1: Data Preprocessing and Voxelization

[0036] Step 1-1: Perform statistical filtering to denoise the original point cloud (remove outliers) and sample the farthest point at FPS, keeping the number of points below approximately 50,000 to balance detail preservation and computational efficiency.

[0037] Steps 1-2: Voxelize the point cloud. Set the resolution of the fine-level voxels to 8cm to capture local geometric features such as object edges; set the resolution of the coarse-level voxels to 16cm to extract global contextual information; after voxelization, construct a feature pyramid using a sparse 3D convolutional backbone. The sparse 3D convolutional backbone uses ResNet under the Minkowski Engine framework.

[0038] Steps 1-3: Use CLIP, aligned with the 2D image, as the text embedding model to convert the input text into text embeddings. .

[0039] Step 2: Forward computation and feature extraction of the teacher model

[0040] Step 2-1: Use the pre-trained Grounding DINO as the teacher model, whose input is multi-view 2D images and text prompts;

[0041] Step 2-2: For each input containing image I and text prompt T, calculate the output of the teacher model using the following formula:

[0042] ;

[0043] in, For dense 2D feature maps, the spatial resolution of the dense 2D feature maps is downsampled to 1 / 16. For the predicted 2D bounding box, This represents the class confidence score, which is based on the class probability normalized by softmax.

[0044] Step 3: Student Model 3D Feature Encoding

[0045] Step 3-1: The 3D student model uses a sparse ResNet-34 as the backbone network, inputs the voxelized point cloud, and extracts 3D voxel features at both fine and coarse scales:

[0046] ;

[0047] ;

[0048] in, For fine-level 3D voxel features, For coarse-level 3D voxel features, For feature dimension, These represent the effective number of voxels at the fine and coarse levels, respectively.

[0049] Step 4: Cross-modal teacher-student distillation

[0050] Transferring knowledge from the teacher model to the student model involves three levels of distillation:

[0051] Step 4-1: Characteristic-level distillation

[0052] A multi-view geometric consistency projection algorithm is employed to establish a precise 2D-3D correspondence through the camera parameter matrix. A depth perception enhancement module is then used to enhance the teacher's 2D features. Back projection onto 3D space to form a 3D feature mesh .

[0053] ;

[0054] in, For projection function, The weighted coefficients are based on depth consistency and view visibility. This represents the total number of point clouds. Indicates the first Point cloud.

[0055] Align student 3D features using Mean Squared Error Loss (MSE) Features of teacher projection Obtain characteristic level distillation loss :

[0056] .

[0057] Step 4-2: Semantic Confidence Distillation

[0058] Using a variant of Focal Loss, we align the student's predicted 3D category confidence with the teacher's 2D soft target to obtain a semantic confidence distillation loss. :

[0059] ;

[0060] in, Prediction results for student models With the teacher model output target The probability of consistency As a balance factor, To focus parameters, This represents the total number of predicted categories for the teacher model. for The k-th 2D bounding box belongs to the k-th bounding box. Confidence scores for each category The k-th 3D bounding box output for the student model belongs to the k-th bounding box. Confidence scores for each category.

[0061] Step 4-3: Bounding box alignment distillation

[0062] Based on the spatial location information of the 3D detection box, the 3D bounding box predicted by the student is projected using a camera projection model. Projecting onto a 2D image plane yields a pseudo-2D bounding box. :

[0063] ;

[0064] in, Output for student model ;

[0065] Align the projection frame using the mean absolute error L1 loss. High-confidence 2D bounding box with teachers Obtain the bounding box alignment distillation loss:

[0066] ;

[0067] in, The total number of 3D bounding boxes;

[0068] Step 4-4: Total Distillation Loss

[0069] Calculate total distillation loss Weighted sum of the three:

[0070] ;

[0071] in, , and These are the weight parameters for feature level, semantic confidence, and bounding box alignment, respectively.

[0072] Step 5: Query Guided Prior Sampling QGPS

[0073] Implemented by the QGPS module in the 3D open vocabulary detection model. Located after the 3D backbone network, it is the core module for processing visual and textual data. The QGPS module employs a coarse-to-fine strategy to efficiently filter 3D regions relevant to the text.

[0074] Step 5-1: Coarse-grained prior generation mechanism

[0075] Mimicking the human cognitive process of "from whole to part," this approach first generates a global, generalized guiding framework—a coarse-grained prior—and then uses deep features to quickly determine the location of potential target regions, guiding the subsequent generation of high-quality, highly consistent detailed content. This includes:

[0076] Step 5-1-1: Multi-scale feature fusion enhancement

[0077] A feature pyramid fusion strategy is adopted to achieve effective integration of multi-resolution features through mathematical transformation:

[0078] Optional, (1) Feature fusion:

[0079] Let the fine-level 3D voxel features be The rough-level 3D voxel features are The original coarse features The generation process is as follows:

[0080]

[0081] in The mathematical expression for the sparse convolution downsampling operation is:

[0082]

[0083] in, This represents a 3D convolution operation. This indicates a max pooling operation.

[0084] Preferably, (2) feature fusion enhanced by attention weighting mechanism:

[0085] Enhancement and improvement of feature fusion are achieved by introducing gated attention weights. The coarse features whose contribution is enhanced by dynamic adjustment are obtained. :

[0086]

[0087]

[0088] in, For the Sigmoid function, This represents element-wise multiplication. For the gated weight matrix, This is the gated bias term.

[0089] Step 5-1-2: Cross-modal fusion of bidirectional encoders

[0090] The bidirectional encoder achieves deep interaction between visual and linguistic features through the Transformer architecture:

[0091] (1) Feature projection transformation:

[0092] Computational visual feature projection : ;

[0093] Calculate text feature projection : ;

[0094] in, For layer normalization operation, The input query text, The projection weight matrix is ​​the visual feature. For visual feature projection bias term; The text feature projection weight matrix, This is the text feature projection bias term. Optional. Can be replaced with the original coarse features .

[0095] (2) Cross-attention calculation:

[0096] Compute cross-modal context of the attention layer output from vision to text. :

[0097] ;

[0098] in, For normalization operations, , and These represent the query, key, and value at the visual-to-text attention layer. , , and The query, key, and value weight matrices are divided into visual and text attention layers.

[0099] Calculate the cross-modal context of the text-to-visual attention layer output. :

[0100] ;

[0101] in, , and These represent the query, key, and value at the visual-to-text attention layer. , , and It is divided into a query, key, and value weight matrix that corresponds to the text-to-visual attention layer.

[0102] (3) Cross-modal fusion feature generation:

[0103] Cross-modal context and After addition, the results are normalized. The cross-modal fusion features are obtained by using coarse-grained features to represent potential target regions, and these cross-modal fusion features are used as coarse prior features. :

[0104] .

[0105] Step 5-1-3: Parallel Dual Prior Generation

[0106] (1) Semantic prior generation

[0107] Semantic heatmap calculation based on cosine similarity:

[0108] Similarity matrix calculate:

[0109] ;

[0110] in, for The first in A visual feature, To query the text for the first A text token, Represents the dot product. Indicates the modulus length. , Represents the total number of visual features. Indicates the total number of text tokens.

[0111] Attention-weighted aggregation yields semantic heatmaps :

[0112]

[0113]

[0114] For the first The contribution of each visual feature To query the text for the first A text token, , Aggregate weights for attention weighting. For multi-layer sensing operations, It is an exponential function.

[0115] The final result This represents the semantic similarity between each point and the corresponding object, i.e., the semantic matching degree. This semantic matching degree is used as the semantic prior. .

[0116] Semantic prior is a heatmap representing where things resemble the text description. It is obtained by calculating the cosine similarity between 3D visual features and text features. Simply put, it identifies the 3D regions where visual features best match the semantics of the text. It focuses more on the degree of matching between appearance and semantics.

[0117] (2) Geometric prior generation

[0118] Mathematical modeling of the geometry-guided predictor:

[0119] Geometric feature extraction :

[0120] ;

[0121] in, For the first Geometric features of a visual feature For the first Point density of a visual feature For the first The curvature of a visual feature For the first The normal vector of a visual feature. For the first Depth information of visual features.

[0122] Objectivity score prediction:

[0123] ;

[0124] in, For the first Geometric objectivity score of visual features For the Sigmoid function, and These are the object-specific score weight matrix and the bias term, respectively.

[0125] Geometric loss function design:

[0126] ;

[0127] in For the first Binary geometric mask labels for visual features.

[0128] The objectivity scores corresponding to all visual features constitute the geometric prior. .Right now, for The objectivity score for each point is the probability that the object indicated by the text might exist at each point. The geometric prior is a heatmap representing where a complete object might be. Based on the underlying geometric properties of the point cloud (such as point density, curvature, normal vector, etc.), the probability that each location is an object and whether it has the geometric characteristics of a complete object is predicted. It focuses more on geometry and structure. In Figure 2, (a) describes a desk: There is a window with green curtains, and to the left of the window with green curtains is a desk. The desk is the item we are looking for. (b) describes a door: The door is located to the right of the shelving. It is also located to the left of the couch. (c) describes a garbage bin: I plan to dispose of used papers and beverage cups during the meeting. (d) Describing the table: I want to serve meals on it after cooking.

[0129] Step 5-2: Hybrid Prior Fusion and Query Sampling

[0130] Step 5-2-1: Based on coarse prior features The two priors obtained and Upsampling to fine coordinates And merged into a hybrid prior S:

[0131] ;

[0132] in, For sparse tensor interpolation, This is the Sigmoid function.

[0133] Step 5-2-2: Based on the hybrid prior S, initialize the query point set using a hybrid sampling strategy. The example defaults to selecting 2000 points in the point cloud for querying. This strategy primarily uses Top-K sampling to select 90% of the points, ensuring the selection of the most critical points; it is supplemented by 5% farthest point sampling (FPS) to enhance the spatial coverage of the point set and avoid clustering; at the same time, 5% random sampling is introduced to maintain a certain degree of exploratoryness and prevent the omission of potentially important points due to over-reliance on specific rules. By default, 2000 points are selected as query points in the point cloud. This number is an empirical value that balances computational resource consumption and the risk of omission: increasing the number of points may improve model performance, but will significantly increase computational overhead; reducing the number of points may lead to missed detections due to insufficient information. Therefore, in practical applications, this parameter can be flexibly adjusted according to the resource constraints and accuracy requirements of the specific scenario.

[0134] The QGPS module incorporates semantic priors based on semantic matching scores. Geometric priors of the geometric probability of an object A 3D spatial probability map is used to intelligently and efficiently guide the model to view (sample) the location where it is most likely to find the target.

[0135] Step 5-3: Deformable fine sampling

[0136] Use deformable blocks By learning dynamic offsets and correcting the initial query point position to focus on more informative features, a more refined query feature set is obtained. as follows:

[0137]

[0138] Specifically, let the initial set of query points be... The initial query point set corresponds to the following features: , The initial total number of query points, fine-level 3D voxel features .

[0139] Offset prediction formula:

[0140]

[0141] in The offset prediction network is implemented using a multilayer perceptron.

[0142]

[0143] in, For activation function, Features corresponding to the initial query point set. and These are the activation weights and bias terms, respectively. and These are the multi-layer perceptual weights and bias terms, respectively.

[0144] Deformable convolution operations:

[0145] Corrected query point location The calculation is as follows:

[0146]

[0147] Feature resampling yields refined query features :

[0148]

[0149] in For the first The weights that can be learned per sampling grid For the first Each sampling grid offset, This represents the total number of sampling grids. The example uses... That is, for each of the 2000 initial sampling points, 8 biases are applied to correct the original sampling, because the original sampling may be inaccurate. It's a hyperparameter.

[0150] Step 5-4: Context Fusion and Final Prediction

[0151] Step 5-4-1: Rough prior features Upsampling , and refined query features Element-by-element addition is performed, followed by context fusion to obtain the fused features. :

[0152]

[0153] Step 5-4-2: Merge the features Enter the bidirectional encoder again Finally, cross-modal inference is performed with the text embedding T to obtain the predicted features. :

[0154]

[0155] Step 5-4-3: The prediction head of the 3D open vocabulary detection model is fed into the model to obtain the final 3D bounding box. With prediction category confidence score .

[0156] Step 6. Two-stage training strategy

[0157] This method employs a two-stage training strategy to train the 3D open vocabulary detection model, balancing generality and task specificity.

[0158] The 3D open vocabulary detection model takes voxelized point clouds and text queries as input and outputs 3D bounding boxes and category prediction results corresponding to the text query target.

[0159] Step 6-1 Pre-training phase:

[0160] Dataset: Based on the SUN RGB-D dataset.

[0161] Training objective: The 3D open vocabulary detection model, acting as a student model, learns general visual-semantic priors from the 2D open vocabulary detection model, which acts as a teacher model, through cross-modal distillation. It then transfers structured visual-linguistic knowledge to the 3D open vocabulary detection model through multi-level knowledge distillation.

[0162] Pre-training loss function : ;

[0163] in, For the positioning loss, a hybrid loss of mean absolute error L1 loss and 3D distance intersection-union (3D DIoU) loss is adopted. To compare the alignment loss, we use the dynamically weighted cross-entropy loss (Focal Loss) method. , and These are the weighting coefficients for positioning loss, contrast alignment loss, and total distillation loss, respectively.

[0164] Step 6-2 Fine-tuning stage:

[0165] Datasets: Based on downstream task datasets, such as ScanRefer and Intent3D.

[0166] Training objective: To adapt the 3D open vocabulary detection model to localization tasks, including visual localization and intent localization.

[0167] Fine-tuning the loss function : ;

[0168] in, This is the closed-set classification loss, used to optimize the model's ability to recognize objects of known categories in the downstream task dataset. It is calculated using dynamically weighted cross-entropy loss (Focal Loss). The loss is geometric. , , and These are the weighting coefficients for the localization loss, closed-set classification loss, and geometric loss, respectively. Here, the closed-set classification loss is used. Replaced total distillation loss .

[0169] Step 7. Inference process of 3D open vocabulary detection model

[0170] Step 7-1: During inference, only point cloud and text query need to be input into the 3D open vocabulary detection model; image data is not required.

[0171] Step 7-2: Preprocess the point cloud and voxelize it.

[0172] Step 7-3: Extract 3D voxel features using the 3D backbone network of the 3D open vocabulary detection model.

[0173] Step 7-4: The QGPS module of the 3D open vocabulary detection model generates and samples mixed priors based on the text query, and finally outputs the predicted 3D bounding box and category prediction corresponding to the text query target.

[0174] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention.

Claims

1. A 3D open vocabulary detection method based on cross-modal distillation and query-guided sampling, characterized in that, It includes a training phase and an inference phase: The training phase includes the following steps: Pre-training phase: Based on a multimodal dataset, knowledge from a pre-trained 2D open vocabulary detection model (serving as the teacher model) is transferred to a 3D open vocabulary detection model (serving as the student model) through cross-modal teacher-student distillation. This distillation includes feature-level distillation, semantic confidence distillation, and bounding box alignment distillation. The model parameters are then optimized using a pre-trained loss function, which includes localization loss, contrast-alignment loss, and total distillation loss. Fine-tuning phase: Based on a downstream task dataset, the model is task-adapted using a fine-tuning loss function. This fine-tuning loss function, based on the pre-training loss function, retains the localization loss and contrast-alignment loss, removes the total distillation loss, and introduces closed-set classification loss and geometric loss. In the fine-tuning stage, after extracting 3D voxel features, a step is performed to generate a hybrid prior based on semantic and geometric priors, using the 3D voxel features and text query. This step adopts a coarse-to-fine strategy, including: generating a semantic prior representing semantic matching degree and a geometric prior representing the geometric probability of the object based on coarse prior features; upsampling the semantic and geometric priors to fine-level coordinates and fusing them into a hybrid prior; initializing the query point set based on the hybrid prior; performing deformable fine sampling on the initialized query point set to obtain fine query features; in the inference stage: the trained 3D open vocabulary detection model receives the input point cloud and text query without requiring 2D image input, and outputs the 3D bounding box and class confidence corresponding to the text query target.

2. The method as described in claim 1, characterized in that, Feature-level distillation involves back-projecting the 2D features of the teacher model into a 3D space using multi-view geometric consistency projection to form the teacher's projected features in a 3D feature grid. Mean squared error loss is then used to align the 3D student features with the teacher's projected features. Semantic confidence distillation uses dynamically weighted cross-entropy loss to align the student's predicted 3D class confidence with the teacher's 2D soft target. Bounding box alignment distillation involves projecting the student's 3D bounding box onto a 2D image plane using a camera projection model to obtain a pseudo-2D bounding box. L1 loss is then used to align the pseudo-2D bounding box with the teacher's 2D bounding box.

3. The method as described in claim 1, characterized in that, Mixed Priors The calculation method is as follows: in, For semantic priors, As a geometric prior, For fine coordinates, For sparse tensor interpolation, This is the Sigmoid function.

4. The method as described in claim 1, characterized in that, Semantic prior is constructed based on the cosine similarity between visual features and text tokens to create a semantic heatmap; geometric prior is generated by extracting geometric features from point clouds and predicting them through object-related scores.

5. The method as described in claim 1, characterized in that, The closed-set classification loss is used to optimize the model's ability to recognize objects of known categories in the downstream task dataset. It is calculated using dynamically weighted cross-entropy loss (Focal Loss); the geometric loss function... The calculation is as follows: in For the first The binary geometric mask label of a visual feature is obtained through the actual object bounding box. If the point is inside the bounding box, the binary geometric mask label of the visual feature is 1; otherwise, it is 0. Rough prior features The Middle Geometric objectivity score of visual features This represents the total number of visual features.

6. The method as described in claim 1, characterized in that, Obtain rough prior features The specific steps include: 1) Generating the original coarse features ： in This indicates a sparse convolution downsampling operation. For detailed 3D voxel features, This is a rough 3D voxel feature; and After the voxelized point cloud is input into the student model, the backbone network of the student model extracts 3D voxel features at both fine and coarse scales; 2) By introducing gating attention weights The coarse features whose contribution is enhanced by dynamic adjustment are obtained. ： in, For the Sigmoid function, This represents element-wise multiplication. For the gated weight matrix, 3) Obtain the visual feature projection through feature projection transformation, which is the gated bias term; and text feature projection ； ； ;in, For layer normalization operation, The input query text, The projection weight matrix is ​​the visual feature. For visual feature projection bias term; The text feature projection weight matrix, 4) Calculate the cross-modal context of the attention layer output from vision to text, which is the text feature projection bias term; Cross-modal context of text-to-visual attention layer output ： ;in, For normalization operations, 、 and These represent the query, key, and value at the visual-to-text attention layer, respectively. ， 、 and The query, key, and value weight matrix is ​​divided into visual-to-text attention layers. ;in, 、 and These represent the query, key, and value at the visual-to-text attention layer, respectively. ， 、 and The query is divided into a text-to-visual attention layer, and the weight matrix corresponds to the key and value; 5) cross-modal context and After addition, the results are normalized. The cross-modal fusion features are obtained and used as coarse prior features. ： 。 7. The method as described in claim 6, characterized in that, The training phase also includes preprocessing and voxelization of the input point cloud, including: performing statistical filtering and farthest point sampling on the point cloud; setting fine-level voxel resolution and coarse-level voxel resolution; constructing a feature pyramid using a sparse 3D convolutional backbone to extract fine-level and coarse-level voxel features.

8. The method as described in claim 1, characterized in that, Deformable fine sampling specifically involves: predicting the offset of the query point position and correcting the initial query point position; then resampling the corrected query point using deformable convolution to obtain fine query features.

9. The method as described in claim 1, characterized in that, The query point set initialization based on hybrid priors is specifically achieved by using a hybrid sampling strategy to initialize the query point set. The hybrid sampling strategy primarily uses Top-K sampling to select 90% of the query points, supplemented by farthest point sampling FPS to select 5% of the query points, and also introduces random sampling to select 5% of the query points.

Images