A monocular unsupervised open vocabulary indoor object detection method based on three-dimensional Gauss

Abstract

The application discloses a monocular unsupervised open vocabulary indoor object detection method based on three-dimensional Gauss, which comprises the following steps: firstly, generating a two-dimensional candidate box by using a monocular image, combining monocular depth prediction and gradient-based smoothing completion to obtain a high-quality dense depth map, and projecting the two-dimensional candidate box to a three-dimensional space to generate a pseudo label; secondly, explicitly parameterizing three-dimensional features into a Gaussian set, and directly predicting three-dimensional detection information and semantic features by using an anchor-free detection head; finally, performing similarity calculation on the semantic features and text embedding to realize open vocabulary classification, and combining Gaussian rendering calculation with point cloud-text-image feature comparison loss to optimize the network. The application can detect objects in an indoor environment with high precision under an open category condition without artificial three-dimensional labeling, and can significantly reduce the cost and improve the generalization ability and adaptability.

Description

Technical Field

[0001] This invention relates to three-dimensional target detection technology, and in particular to a monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian, which is applicable to general target perception and recognition in various indoor scenes such as residences, offices, and classrooms. Background Technology

[0002] In indoor environments, such as residences, office buildings, hotels, public places, and various building interiors, accurate object detection and identification are crucial for tasks such as indoor robot navigation, automated inventory management, spatial digital reconstruction, augmented reality applications, and smart home systems. Key objects in indoor environments (such as furniture, appliances, structural components, and decorative objects) often exhibit diverse categories, large scale variations, complex spatial layouts, severe occlusion, and significant lighting changes. These targets are typically densely distributed and irregularly arranged, and are also subject to interference from factors such as occlusion, reflections, shadows, repetitive textures, and cluttered backgrounds. Therefore, timely and accurate detection and identification of these indoor targets is one of the core challenges faced by indoor scene understanding and intelligent perception systems.

[0003] In practical applications, 3D object detection in indoor scenes is extremely complex. Traditional 3D detection methods typically rely on large amounts of manually labeled data for model training and feature extraction. However, obtaining large-scale, high-quality 3D labeled data is very difficult. Indoor environments contain a rich variety of objects with significant differences in shape and material. Coupled with common issues like object occlusion, uneven lighting, and cluttered backgrounds, 3D annotation requires substantial manual input and is prone to mislabeling or inaccuracies. When annotating indoor targets, annotators not only need to understand the appearance and structure of multiple object types but also must handle highly complex indoor spatial layouts, significantly increasing the difficulty and cost of data annotation. For these reasons, traditional high-precision 3D detection methods are often difficult to apply directly in indoor scenes and are subject to significant limitations.

[0004] In recent years, monocular 3D object detection technology has seen rapid development in fields such as robotics, AR / VR, indoor scene reconstruction, and smart homes. Compared to multi-view or LiDAR solutions, monocular methods have advantages such as low hardware cost and convenient deployment, but they still suffer from insufficient accuracy in depth estimation and 3D geometry reconstruction. Most existing methods are based on closed vocabularies, which can only identify fixed categories appearing in the training set and cannot adapt to the needs of rich and frequently updated object types in indoor environments.

[0005] Open vocabulary detection methods utilize visual-linguistic joint embedding to map image features to a semantic space shared with text descriptions, achieving detection and recognition of any category by calculating similarity with category text embeddings. However, directly applying open vocabulary methods to monocular 3D detection faces three major challenges: (1) the variety of indoor targets and the lack of large-scale 3D labeled data make it difficult for 3D detection models to obtain accurate geometric supervision; (2) monocular depth prediction is prone to noise and holes under complex lighting, occlusion, and reflective surface conditions, affecting 3D positioning accuracy; (3) how to simultaneously optimize 3D geometric prediction and semantic matching under unsupervised conditions to balance positional accuracy and category generalization ability. Therefore, there is an urgent need for a technical solution that can generate stable and high-quality 3D pseudo-labels in the absence of manual 3D annotation, improve category generalization ability by combining open vocabulary mechanisms, and improve monocular detection accuracy by using explicit 3D Gaussian representation. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian to address the shortcomings of the existing technology.

[0007] To address the aforementioned technical problems, this invention discloses a monocular unsupervised open-vocabulary indoor target detection method based on three-dimensional Gaussian, comprising the following steps:

[0008] Step 1: Collect real indoor scene images as a training set. Generate 3D pseudo-labels for the training set through 2D candidate box detection, monocular depth prediction, gradient-based depth smoothing completion, and 3D projection. The categories of indoor objects in Step 1 are determined according to the requirements of the embodiment.

[0009] Step 2: Construct a Gaussian-based monocular open vocabulary 3D detection network, initialize the Gaussian set, extract multimodal features and perform cross-modal fusion, iteratively update the Gaussian set, and use an anchorless detection head to predict the 3D detection box set and semantic features;

[0010] Step 3: Train a Gaussian-based monocular open vocabulary 3D detection network using pseudo-labels of real scene images, match and classify the predicted semantic features with the text embeddings of the open vocabulary, and optimize the network with reconstruction contrast loss.

[0011] Step 4: Use the trained 3D detection network to perform target detection.

[0012] The generation of pseudo-labels for the training set described in step 1 includes the following steps:

[0013] Step 1-1: The collected real-world scene images I (RGB image set) and the input prompt words (where the prompt words specify the range of categories to be detected) are processed. Run a pre-trained 2D detector with an open vocabulary and freeze its parameters to obtain a set of 2D detection boxes: ,in To detect the pixel coordinates of the top-left corner of the bounding box, These are the width and height of the detection frame, respectively. For category labeling or category confidence, To determine the number of detection boxes, a threshold strategy is used to filter them if and only if the centrality score is [value missing]. Retain the two-dimensional detection frame at the time ;

[0014] Steps 1-2: The monocular depth estimation module is used to perform monocular depth prediction on the real scene image to obtain an initial depth map D(x,y), and the predicted initial depth map is corrected using a depth correction algorithm to generate a dense depth map.

[0015] Steps 1-3 combine the two-dimensional candidate boxes with the dense depth map, generate a three-dimensional point cloud through camera model projection, and cluster them to form three-dimensional pseudo-labels (such as three-dimensional bounding boxes or target voxel sets).

[0016] In steps 1-2, the monocular depth estimation module includes a depth estimation module and a depth correction algorithm, which are used to provide a dense and robust depth map required for subsequent 3D inference.

[0017] Step 1-2-1: The depth estimation module uses a pre-trained depth estimation network and freezes its parameters to generate an initial depth map D, representing the absolute physical distance from the pixel to the camera. ;in Image pixel coordinates, This corresponds to the depth value;

[0018] Step 1-2-2: The initial depth map D is modified by a depth correction algorithm to suppress outliers and edge errors, resulting in a corrected dense depth map.

[0019] Step 1-2-2 specifically includes:

[0020] (1) Object edge region recognition: Calculate the horizontal gradient on the initial depth map. with vertical gradient And based on this, the gradient magnitude is obtained. ,like ,in If the threshold value is set to the edge, then that location is marked as the area to be corrected.

[0021] (2) Outlier Identification: Percentile-based progressive probing is used to estimate the maximum / minimum acceptable depth thresholds for the scene, and pixels exceeding the maximum depth threshold or falling below the minimum depth threshold are included in the area to be corrected. Preferably, the correction starts from a preset high percentile (e.g., ...). Gradually reduce the depth to approximate the upper and lower bounds of the real scene;

[0022] (3) Image edge region setting: Set the width or height of the outer edge of the initial depth map. As a fixed region to be corrected, to cover the unstable estimate of the field of view boundary;

[0023] (4) Depth value regeneration: A K-D tree (KDTree) is built using pixels that are not marked as areas to be corrected (valid points). For each pixel to be corrected, the nearest valid neighbor point is searched, and its depth value is assigned or guided interpolation is performed to obtain the corrected dense depth map. .

[0024] Steps 1-3 include:

[0025] Step 1-3-1, Camera Projection Model: Set the intrinsic parameter matrix The main point is focal length is , For pixels and its depth 3D points corresponding to the camera coordinate system for:

[0026]

[0027]

[0028]

[0029] Step 1-3-2, Detection box pixel selection: For the two-dimensional detection box The pixel set is

[0030] ;

[0031] Based on this, the corresponding three-dimensional point set is obtained:

[0032]

[0033] Step 1-3-3: To suppress background and noise points, density clustering (such as DBSCAN) is used. -means) and statistical filtering are used for 3D point cloud clustering and segmentation;

[0034] Steps 1-3-4: Generate 3D pseudo-labels: based on clustering and segmented 3D point sets. Fit 3D target pseudo-labels; find the minimum bounding box:

[0035]

[0036] Alternatively, output a parametric box containing center, size, and orientation:

[0037]

[0038] The center point of the detection frame is determined by The estimated size is determined by the point cloud extent or category prior, and the orientation angle... The categories are estimated by principal component analysis or equivalent methods. The true value of the corresponding two-dimensional detection box Obtain and record the set of all 3D detection boxes. for .

[0039] The monocular open vocabulary 3D detection network described in step 2 consists of a 2D side, a 3D side, deformable cross attention, 3D interaction, and a regression head;

[0040] Two-dimensional side extracts multi-scale two-dimensional semantic features from two-dimensional encoding. In the deformable cross-attention module and Gaussian center parameters The predicted depth D obtained from the monocular depth estimation module in step 1-2-1 is fused together.

[0041] The 3D side extracts 3D semantic features from a 3D convolutional backbone network, and integrates these features with the Gaussian center parameters in the 3D interaction module. The fusion process involves a four-stage structure of sub-manifold sparse convolutional blocks and stride sparse convolutions (each stage has 2-3 residual blocks), and a three-dimensional neck network that uses transposed sparse convolution upsampling and is fused with lateral connections.

[0042] Deformable cross-attention: with Gaussian center parameter Projection point obtained by camera projection Using this as a reference point, in multi-scale two-dimensional semantic features Multi-scale deformable sampling and linear fusion are performed to obtain Gaussian features with two-dimensional side encoding. ;

[0043] 3D Interaction: Gaussian Center Parameters Voxelization and The Gaussian features of the three-dimensional side encoding are obtained by aggregating the neighborhood three-dimensional context through multi-layer sparse convolution / residual blocks. ;

[0044] Regression Head: Gaussian features encoded in two-dimensional sides Gaussian features with 3D side encoding Fusion into aggregated Gaussian features And output the Gaussian prediction parameter increments through multiple MLP layers. .

[0045] Step 2 includes:

[0046] Step 2-1 (Gaussian Initialization): Initialize N 3D Gaussian seeds within the current camera's view frustum using a uniform or hierarchical strategy to obtain the initial Gaussian set. ,in Central position Covariance (or equivalent metric) ), For rotation, For opacity / weight, For semantic embedding;

[0047] Step 2-2 (Multimodal Feature Extraction): The 2D encoder extracts multi-scale 2D semantic features from the input indoor real-world scene image. And based on the camera's internal and external parameters Compared with the initial depth map in step 1-2-1 Backprojection / voxalization to a 3D sparse voxel space; feature extraction and top-down / bottom-up fusion of a 3D convolutional backbone network and a 3D neck network to obtain multi-scale 3D features. ;

[0048] Steps 2-3 (Cross-modal feature integration): The 3D convolutional backbone network uses the Gaussian center parameters of the current iteration. For the query, deformable cross-attention is used to extract multi-scale two-dimensional semantic features. Mid-sample two-dimensional semantic cues, and simultaneously Voxelization and multi-scale 3D features Sparse 3D convolutional interactions are performed on the surface and fused with depth priors to form aggregated feature vectors for each Gaussian. Used for iterative prediction of Gaussian parameters;

[0049] Step 2-4 (Gaussian Iteration Update): Steps 2-2 and 2-3 are iterated R times to obtain a convergent Gaussian set. Incrementing the Gaussian prediction parameters through parallel regression branches And based on this, perform a Gaussian parameter update:

[0050] ; or ;

[0051] ; ;

[0052] in, Represents quaternion combinations, It is Sigmoid. The center position Covariance (or equivalent standard) ), rotation Opacity / Weight Semantic embedding Gaussian prediction parameter increments;

[0053] Steps 2-5 (Anchor-free 3D detection): At the non-empty voxel position at the Gaussian center, an anchor-free 3D detection head is used for prediction, and a set of 3D detection boxes is output.

[0054] The Gaussian parameters and regression parameterization in steps 2-4 satisfy the following:

[0055] a) Central location Regression was performed using residuals.

[0056] b) Covariance For positive definiteness, diagonal parameterization (regression to logarithmic scaling and exponential mapping) or Cholesky parameterization (regression to the lower triangular matrix L and setting...) (and the diagonal elements are positive by an exponential function constraint).

[0057] c) Rotation Use unit quaternions or 6D orthogonal parameterization and normalize at the output;

[0058] d) Opacity / Weight Constrained by the Sigmoid function;

[0059] e) Semantic embedding Aggregated feature vectors after self-fusion of linear / MLP (Multilayer Perceptron) Mapped for use in open vocabulary alignment. The semantic embedding vector dimension;

[0060] f) Covariance during initialization phase Rotation during the initialization phase, based on the set scale and isotropic / anisotropic prior assignments. Initialize to a unit rotation or an approximation aligned with the camera's optical axis.

[0061] The frameless detection head in steps 2-5 includes:

[0062] Multi-scale three-branch structure with shared weights: 3D features at each scale Several sparse convolutional layers, normalization layers, and activation layers are stacked on top of each other, and then the layers branch into regression, classification, and centrality branches.

[0063] The regression branch: predicts the center of the 3D bounding box. ,size Orientation ,in Using continuous parameterization ;

[0064] The classification branch: outputs semantic features. The category / cue embedding obtained by the text encoder with frozen parameters Calculate (normalized) cosine similarity to achieve open vocabulary classification;

[0065] The centrality branch outputs the centrality confidence score, which is multiplied by the classification score to form a candidate score.

[0066] Post-processing: Based on the score, perform 3D NMS (Non-maximum suppression algorithm) or rotate NMS to output the final detection box.

[0067] Step 3 includes:

[0068] A Gaussian-based monocular open-vocabulary 3D detection network is trained using pseudo-labels from real-world images, and the total training loss is calculated. The detection loss, reconstruction loss, and multimodal contrast loss are jointly optimized, and the overall objective function is:

[0069]

[0070] in, To match the total classification loss, To rebuild the losses, For multimodal contrast loss, the three terms correspond to weights. >0;

[0071] The total matching classification loss includes open vocabulary classification loss, centrality loss, and 3D bounding box regression loss: for positive samples Summing and normalizing yields the total matching and classification loss:

[0072]

[0073] in, For open vocabulary classification loss, For 3D bounding box regression loss, The centrality loss is used, where Pos is the set of all positive candidate locations / candidate boxes.

[0074] The open vocabulary classification loss For each positive sample candidate location / candidate box i∈Pos generated by the anchorless 3D detector, the semantic features will be predicted. With open vocabulary text embedding Calculate cosine similarity and with As the category probability, where The temperature coefficient (controlling the smoothness of the softmax function) is used to determine the true class of the i-th positive sample. The open vocabulary classification loss is: ; where positive sample candidate positions / number of candidate boxes The open vocabulary essentially constitutes a large set of object-type nouns, covering most common physical objects, including but not limited to tables, chairs, and benches. It defaults to a dictionary covering a large number of indoor object categories, but can also be specified by input prompts. This is to implement the semantic features of each noun in the vocabulary. To ensure effective alignment between terms, this invention employs a text encoder to embed all terms in the vocabulary. Specifically, the text encoder maps each term to a corresponding text embedding vector. For example, "table" is mapped to its corresponding text embedding, and so is "chair," and then similarity is calculated, followed by the calculation of the loss function;

[0075] The three-dimensional box regression loss The parameters of the regression branch output prediction box are: Get the complete set of 3D detection boxes Zhongyu The best matching 3D detection box Calculate the loss as a pseudo-label;

[0076] If AABB (axis-aligned box) is used, and IoU loss is applied, let ;

[0077] If using OBB (Rotated Frame), then use rotation. loss: To alleviate the ambiguity of orientation-width / height interchange, orientation... Combined with continuous Möbius parameterization. During training, predictions are made directly from the regression branch. and Mutual restoration

[0078] The centrality loss The centrality branch outputs the centrality score. Its truth value Taking the extended form of 3D FCOS, let the distance from the candidate position to the six faces of the truth box be... ,but

[0079]

[0080] The centrality loss uses binary cross-entropy ( ):

[0081]

[0082] Reconstruction loss based on 3D Gaussian , For weights, optionally, for and Applying regularization to stabilize rendering (such as scale prior and sparse prior), this regularization is incorporated into .in, For the feature loss function, For depth loss function;

[0083] The reconstruction loss By simultaneously constraining image semantic features and depth through 3D Gaussian differentiable rendering, the geometric and semantic consistency of Gaussian parameters can be directly supervised.

[0084] The final Gaussian set obtained through iteration is With camera internal and external parameters Will The pixel center projected onto the image plane And projected by Jacobi Two-dimensional covariance ;

[0085] Define screen space Gaussian kernel For each pixel u, sort π(u) by depth from near to far, and let the base opacity coefficient be... Foreground transmittance Where m≺j represents the Gaussian kernel π(u) in screen space preceding j (the basic opacity parameter of the j-th Gaussian, controlling its visibility on the pixel. Foreground transmittance: represents the degree of occlusion of pixel u by all Gaussians preceding the j-th Gaussian), then feature rendering... With depth rendering for:

[0086] .

[0087] in for At the current viewpoint depth coordinates, let the 2D encoder be at the pixel... The target features are Target depth is It can be derived from monocular depth network / sensor depth and from visibility mask. Given the constraints, the feature loss function and the depth loss function are defined as follows:

[0088] .

[0089] in For the set of valid pixels, Punish Charbonnier .

[0090] The computational image-3D-text multimodal contrast loss By aligning the information noise contrast estimation (InfoNCE) along the three paths of text, image, and 3D, the capabilities of the open vocabulary are significantly enhanced.

[0091] For each predicted 3D detection box (The center is given by the regression branch) ,size Orientation ), define three modal embeddings and perform multi-view alignment:

[0092] (1) 3D bounding box embedding: Gaussian features within the convergent bounding box are used to obtain the 3D bounding box embedding. (Pool functions such as average / weighted attention), and normalization. ;

[0093] (2) Image patch embedding: embedding the 3D detection box Projecting onto the image plane yields a 2D region. On the two-dimensional feature map, RoIAlign / patch pooling is used to obtain... and normalization ;

[0094] (3) Text embedding: Candidate open vocabulary text set Normalization ;

[0095] Set the temperature hyperparameters for 3D-text alignment and image-3D alignment respectively. , Define the InfoNCE contrast loss:

[0096] a) The 3D-text alignment loss is:

[0097] ,

[0098] b) The image-3D alignment loss is:

[0099] ,

[0100] Thus, the multimodal contrast loss is obtained:

[0101] .

[0102] in For weights ( In the embodiments given in this invention, the value is 0, but some embodiments recommend a value of 1. Adjustments need to be made according to the specific embodiments. In the embodiments given in this invention, the value is 1). For the first A set of candidate texts for each sample (including positive and sampled negative texts).

[0103] Step 4 includes: performing three-dimensional nonmaximum suppression on the three-dimensional detection results, and outputting the final detection results as the product of class probability and centrality.

[0104] Beneficial effects:

[0105] 1. This invention proposes a method for generating 3D pseudo-labels that combines 2D candidate box detection, monocular depth prediction, and gradient-based depth smoothing completion under the condition of no manual 3D annotation. It can automatically generate 3D supervision signals with near-human annotation accuracy in real scene images, providing high-quality data support for the training of 3D detection networks and effectively improving detection accuracy and robustness.

[0106] 2. This invention employs explicit parametric modeling of 3D Gaussian features and performs anchor-free 3D detection directly at the Gaussian center, predicting bounding boxes and orientation information. This method characterizes the geometric shape and uncertainties of the target, avoids the hyperparameter dependence of anchor-frame matching, and improves the accuracy of geometric reconstruction and adaptability to complex indoor environments.

[0107] 3. This invention introduces an open vocabulary classification mechanism, which matches the detected semantic features with the open vocabulary text embedding, and combines Gaussian rendering reconstruction loss, feature contrast loss and open vocabulary detection loss for joint optimization, thereby enhancing the category generalization and semantic discrimination capabilities, enabling the system to adapt to various indoor target detection and maintain high-precision recognition. Attached Figure Description

[0108] Figure 1 This is a flowchart of the present invention.

[0109] Figure 2 This is a schematic diagram of the network structure of the present invention.

[0110] Figure 3 This is a test example diagram of the present invention.

[0111] Figure 4 This is a diagram illustrating the three-dimensional Gaussian transformation process of the present invention. Detailed Implementation

[0112] This method involves 3D target detection and indoor scene understanding technologies, and is particularly suitable for general target perception and recognition in complex indoor environments. Addressing the challenges of diverse target categories, complex spatial layouts, significant scale and pose variations in indoor scenes, and the limitation of 3D labeled data to fixed categories, this method proposes a solution that combines pseudo-label generation, a 3D Gaussian detection network, open vocabulary classification, and Gaussian reconstruction with comparative optimization.

[0113] A monocular unsupervised open-vocabulary indoor object detection method based on 3D Gaussian is proposed. Its core lies in using arbitrary 2D images as auxiliary information to enhance the annotation information (position-level annotation, i.e., object category and center point coordinates) of weakly supervised 3D object detection, thereby improving detection accuracy. Figure 1 As shown, it includes the following steps:

[0114] Step 1: Collect real indoor scene images as the training set. Generate 3D pseudo-labels for the training set through 2D candidate box detection, monocular depth prediction, gradient-based depth smoothing completion, and 3D projection. This provides reliable supervision without manual 3D annotation, reduces costs, and alleviates scale / noise bias.

[0115] Step 2: Construct a Gaussian-based monocular open vocabulary 3D detection network, initialize the Gaussian set, extract multimodal features and perform cross-modal fusion, iteratively update the Gaussian set, and use an anchorless detection head to predict the 3D detection box set and semantic features;

[0116] The image and depth features are projected / voxed into a sparse 3D raster, and multi-scale features are extracted through multi-stage sparse 3D convolution. Gaussian parameters (including center, covariance / scale, rotation, opacity, and semantic vector) are regressed at the centers of non-zero voxels to form a Gaussian set. An anchorless bounding box detector is then used at the center of the Gaussian set to output 3D bounding boxes, semantic features, and centrality. Explicit 3D Gaussian achieves simultaneous geometric and semantic representation, weakens anchor box priors, and adapts to different sizes and poses.

[0117] Step 3: Train a Gaussian-based monocular open-vocabulary 3D detection network using pseudo-labels from real-world images. Match and classify the predicted semantic features with the text embeddings of the open vocabulary, and optimize the network using reconstruction contrastive loss. The evolution of the Gaussian vector during training is described in [link to training documentation]. Figure 4 As shown;

[0118] Step 4: Perform target detection using the trained 3D detection network. The final detection results are shown below. Figure 3As shown. During the network prediction phase, the input consists of a single RGB image, input text (optional; if text is specified, it is used; otherwise, a long text containing numerous indoor item categories is used as the default input), and camera intrinsics. The network forward-engineers candidate 3D bounding boxes, category similarity, and centrality. 3D non-maximum suppression is applied to the 3D detection results, and the final detection result is output as the product of category probability and centrality. Zero-shot expansion can be achieved by supplementing the text vocabulary. Overall, the system is user-friendly for monocular deployment, robust in scoring, and expandable in categories.

[0119] Step 4 is as follows: First, input a single RGB image, input text, and the corresponding camera intrinsic matrix K; then, through forward propagation of the network, output a set of 3D detection boxes, including the center coordinates, size, orientation angle, semantic feature vector, and centrality score of each box; second, perform 3D non-maximum suppression (3D NMS) or rotation NMS on the output detection boxes to remove redundant boxes with high overlap; further, multiply the class probability of each box (calculated through semantic features and text embedding) by the centrality score to obtain the final score, and output the detection results in sorted order by score; the text vocabulary can be dynamically expanded according to actual needs to achieve zero-shot class recognition and improve the system's generalization ability on unknown classes.

[0120] The entire reasoning process does not rely on depth sensors or manual annotation; high-precision 3D detection can be completed using only a monocular image, demonstrating good practicality and scalability.

[0121] The generation of pseudo-labels for the training set described in step 1 includes the following steps:

[0122] Step 1-1: The collected real-world scene images I (RGB image set) and the input prompt words (where the prompt words specify the range of categories to be detected) are processed. Run a pre-trained 2D detector with an open vocabulary (the 2D detector used in this embodiment is the Grounded SAM model), and freeze its parameters to obtain a set of two-dimensional detection boxes: ,in To detect the pixel coordinates of the top-left corner of the bounding box, These are the width and height of the detection frame, respectively. For category labeling or category confidence, To determine the number of detection boxes, a threshold strategy is used to filter them if and only if the centrality score is [value missing]. Retain the two-dimensional detection frame at the time ;

[0123] Steps 1-2: The monocular depth estimation module performs monocular depth prediction on the real scene image to obtain an initial depth map D, and the predicted initial depth map is corrected using a depth correction algorithm to generate a dense depth map.

[0124] Steps 1-3 combine the two-dimensional candidate boxes with the dense depth map, generate a three-dimensional point cloud through camera model projection, and cluster them to form three-dimensional pseudo-labels (such as three-dimensional bounding boxes or target voxel sets).

[0125] In steps 1-2, the monocular depth estimation module includes a depth estimation module and a depth correction algorithm, which are used to provide a dense and robust depth map required for subsequent 3D inference.

[0126] Step 1-2-1: The depth estimation module uses a pre-trained depth estimation network (in this embodiment, the Depth Anything v2 model is used) and freezes its parameters to generate an initial depth map D, representing the absolute physical distance from the pixel to the camera. ;in Image pixel coordinates, This corresponds to the depth value;

[0127] Step 1-2-2: The initial depth map D is modified by a depth correction algorithm to suppress outliers and edge errors, resulting in a corrected dense depth map.

[0128] Step 1-2-2 specifically includes:

[0129] (1) Object edge region recognition: Calculate the horizontal gradient on the initial depth map. with vertical gradient And based on this, the gradient magnitude is obtained. ,like ,in If the threshold value is set to the edge, then that location is marked as the area to be corrected.

[0130] (2) Outlier Identification: Percentile-based progressive probing is used to estimate the maximum / minimum acceptable depth thresholds for the scene, and pixels exceeding the maximum depth threshold or falling below the minimum depth threshold are included in the area to be corrected. Preferably, the correction starts from a preset high percentile (e.g., ...). Gradually reduce the depth to approximate the upper and lower bounds of the real scene;

[0131] (3) Image edge region setting: Set the width or height of the outer edge of the initial depth map. As a fixed region to be corrected, to cover the unstable estimate of the field of view boundary;

[0132] (4) Depth value regeneration: A K-D tree (KDTree) is built using pixels that are not marked as areas to be corrected (valid points). For each pixel to be corrected, the nearest valid neighbor point is searched, and its depth value is assigned or guided interpolation is performed to obtain the corrected dense depth map. .

[0133] Steps 1-3 include:

[0134] Step 1-3-1, Camera Projection Model: Set the intrinsic parameter matrix The main point is focal length is , For pixels and its depth 3D points corresponding to the camera coordinate system for:

[0135]

[0136]

[0137]

[0138] Step 1-3-2, Detection box pixel selection: For the two-dimensional detection box The pixel set is

[0139] ;

[0140] Based on this, the corresponding three-dimensional point set is obtained:

[0141]

[0142] Step 1-3-3: To suppress background and noise points, density clustering (such as DBSCAN) is used. -means) and statistical filtering are used for 3D point cloud clustering and segmentation;

[0143] Steps 1-3-4: Generate 3D pseudo-labels: based on clustering and segmented 3D point sets. Fit 3D target pseudo-labels; find the minimum bounding box:

[0144]

[0145] Alternatively, output a parametric box containing center, size, and orientation:

[0146]

[0147] The center point of the detection frame is determined by The estimated size is determined by the point cloud extent or category prior, and the orientation angle... The categories are estimated by principal component analysis or equivalent methods. From the corresponding two-dimensional detection box Obtain and record the set of all 3D detection boxes. for .

[0148] This step automatically generates 3D point clouds and corresponding labels without requiring extensive manual annotation. Through effective noise reduction, the resulting pseudo-labels are cleaner and more reliable, making them suitable for use as high-quality supervisory signals. This approach addresses the technical problems of traditional methods, such as reliance on manual annotation, high costs, and susceptibility to subjective bias. It overcomes the limitations of existing methods in label quality and scale, thus achieving low-cost and high-precision training data construction.

[0149] The monocular open vocabulary 3D detection network described in step 2 consists of a 2D side, a 3D side, deformable cross attention, 3D interaction, and a regression head;

[0150] Two-dimensional side extracts multi-scale two-dimensional semantic features from two-dimensional encoding. In the deformable cross-attention module and Gaussian center parameters The predicted depth D obtained from the monocular depth estimation module in step 1-2-1 is fused together.

[0151] The 3D side extracts 3D semantic features from a 3D convolutional backbone network, and integrates these features with the Gaussian center parameters in the 3D interaction module. The fusion process involves a four-stage structure of sub-manifold sparse convolutional blocks and stride sparse convolutions (each stage has 2-3 residual blocks), and a three-dimensional neck network that uses transposed sparse convolution upsampling and is fused with lateral connections.

[0152] Deformable cross-attention: with Gaussian center parameter Projection point obtained by camera projection Using this as a reference point, in multi-scale two-dimensional semantic features Multi-scale deformable sampling and linear fusion are performed to obtain Gaussian features with two-dimensional side encoding. ;

[0153] 3D Interaction: Gaussian Center Parameters Voxelization and The Gaussian features of the three-dimensional side encoding are obtained by aggregating the neighborhood three-dimensional context through multi-layer sparse convolution / residual blocks. ;

[0154] Regression Head: Gaussian features encoded in two-dimensional sides Gaussian features with 3D side encoding Fusion into aggregated Gaussian features And output the Gaussian prediction parameter increments through multiple MLP layers. .

[0155] Step 2 includes:

[0156] Step 2-1 (Gaussian Initialization): Initialize N 3D Gaussian seeds within the current camera's view frustum using a uniform or hierarchical strategy to obtain the initial Gaussian set. ,in As the center position, Covariance (or equivalent metric) ), For rotation, For opacity / weight, For semantic embedding;

[0157] Step 2-2 (Multimodal Feature Extraction): A two-dimensional encoder (using CLIP in this embodiment) extracts multi-scale two-dimensional semantic features from the input indoor real-world scene image. And based on the camera's internal and external parameters Compared with the initial depth map in step 1-2-1 Backprojection / voxalization to a 3D sparse voxel space; feature extraction and top-down / bottom-up fusion of a 3D convolutional backbone network and a 3D neck network to obtain multi-scale 3D features. ;

[0158] Steps 2-3 (Cross-modal feature integration): The 3D convolutional backbone network uses the Gaussian center parameters of the current iteration. For the query, deformable cross-attention is used to extract multi-scale two-dimensional semantic features. Mid-sample two-dimensional semantic cues, and simultaneously Voxelization and multi-scale 3D features Sparse 3D convolutional interactions are performed on the surface and fused with depth priors to form aggregated feature vectors for each Gaussian. Used for iterative prediction of Gaussian parameters;

[0159] Step 2-4 (Gaussian Iteration Update): Steps 2-2 and 2-3 are iterated R times to obtain a convergent Gaussian set. Incrementing the Gaussian prediction parameters through parallel regression branches And based on this, perform a Gaussian parameter update:

[0160] ; or ;

[0161] ; ;

[0162] in, Represents quaternion combinations, It is Sigmoid. The center position Covariance (or equivalent standard) ), rotation Opacity / Weight Semantic embedding Gaussian prediction parameter increments;

[0163] Steps 2-5 (Anchor-free 3D detection): At the non-empty voxel position at the Gaussian center, an anchor-free 3D detection head is used for prediction, and a set of 3D detection boxes is output.

[0164] The Gaussian parameters and regression parameterization in steps 2-4 satisfy the following:

[0165] a) Central location Regression was performed using residuals.

[0166] b) Covariance For positive definiteness, diagonal parameterization (regression to logarithmic scaling and exponential mapping) or Cholesky parameterization (regression to the lower triangular matrix L and setting...) (and the diagonal elements are positive by an exponential function constraint).

[0167] c) Rotation Use unit quaternions or 6D orthogonal parameterization and normalize at the output;

[0168] d) Opacity / Weight Constrained by the Sigmoid function;

[0169] e) Semantic embedding Aggregated feature vectors after self-fusion of linear / MLP (Multilayer Perceptron) Obtained through mapping and used for open vocabulary alignment;

[0170] f) Covariance during initialization phase Rotation during the initialization phase, based on the set scale and isotropic / anisotropic prior assignments. Initialize to a unit rotation or an approximation aligned with the camera's optical axis.

[0171] The frameless detection head in steps 2-5 includes:

[0172] Multi-scale three-branch structure with shared weights: 3D features at each scale Several sparse convolutional layers, normalization layers, and activation layers (sparse) are stacked on top of each other. The process consists of a convolutional layer, a normalization layer, and an activation layer, which then branch into regression, classification, and centrality branches.

[0173] The regression branch: predicts the center of the 3D bounding box. ,size Orientation ,in Using continuous parameterization ;

[0174] The classification branch: outputs semantic features. The category / cue embedding obtained by the text encoder with frozen parameters Calculate (normalized) cosine similarity to achieve open vocabulary classification;

[0175] The centrality branch outputs the centrality confidence score, which is multiplied by the classification score to form a candidate score.

[0176] Post-processing: Based on the score, perform 3D NMS (Non-maximum suppression algorithm) or rotate NMS to output the final detection box.

[0177] This step introduces an iterative generation mechanism of 3D Gaussian in the scene reconstruction and object detection process, making the generated 3D representation more closely resemble the real scene. Compared with detection methods that do not employ 3D Gaussian, this step significantly improves the accuracy and stability of detection. This technology solves the problem of insufficient scene realism in traditional 3D reconstruction and overcomes the practical difficulty of declining detection quality in complex scenes, thereby achieving higher quality 3D scene reconstruction and object detection results.

[0178] Step 3 includes:

[0179] A Gaussian-based monocular open-vocabulary 3D detection network is trained using pseudo-labels from real-world images, and the total training loss is calculated. ,

[0180] The detection loss, reconstruction loss, and multimodal contrast loss are jointly optimized, and the overall objective function is:

[0181]

[0182] in, To match the total classification loss, To rebuild the losses, For multimodal contrast loss, the three terms correspond to weights. >0;

[0183] The total matching classification loss includes open vocabulary classification loss, centrality loss, and 3D bounding box regression loss: for positive samples Summing and normalizing yields the total matching and classification loss:

[0184]

[0185] in, For open vocabulary classification loss, For 3D bounding box regression loss, The centrality loss is used, where Pos is the set of all positive candidate locations / candidate boxes.

[0186] The open vocabulary classification loss For each positive sample candidate location / candidate box i∈Pos generated by the anchorless 3D detector, the semantic features will be predicted. With open vocabulary text embedding Calculate cosine similarity and with As the category probability, where Let be the temperature coefficient, and let be the true value category of the i-th positive sample. The open vocabulary classification loss is: ; where positive sample candidate positions / number of candidate boxes ;

[0187] The three-dimensional box regression loss The parameters of the regression branch output prediction box are: Get the complete set of 3D detection boxes Zhongyu The best matching 3D detection box The loss is calculated using pseudo-labels.

[0188] If using AABB (axis-aligned box), use The loss made ;

[0189] If using OBB (Rotated Frame), then use rotation. loss: To alleviate the ambiguity of orientation-width / height interchange, orientation... Combined with continuous Möbius parameterization. During training, predictions are made directly from the regression branch. and Mutual restoration

[0190] The centrality loss The centrality branch outputs the centrality score. Its truth value Taking the extended form of 3D FCOS, let the distance from the candidate position to the six faces of the truth box be... ,but

[0191]

[0192] The centrality loss uses binary cross-entropy ( ):

[0193]

[0194] Reconstruction loss based on 3D Gaussian , For weights, optionally, for and Applying regularization to stabilize rendering (such as scale prior and sparse prior), this regularization is incorporated into .in, For the feature loss function, For depth loss function;

[0195] The reconstruction loss By simultaneously constraining image semantic features and depth through 3D Gaussian differentiable rendering, the geometric and semantic consistency of Gaussian parameters can be directly supervised.

[0196] The final Gaussian set obtained through iteration is With camera internal and external parameters Will The pixel center projected onto the image plane And projected by Jacobi Two-dimensional covariance ;

[0197] Define screen space Gaussian kernel For each pixel u, sort π(u) by depth from near to far, and let the base opacity coefficient be... Foreground transmittance Where m≺j represents the Gaussian kernel π(u) in screen space preceding j, then feature rendering... With depth rendering for:

[0198] .

[0199] in for At the current viewpoint depth coordinates, let the 2D encoder be at the pixel... The target features are Target depth is It can be derived from monocular depth network / sensor depth and from visibility mask. Given the constraints, the feature loss function and the depth loss function are defined as follows:

[0200] .

[0201] in For the set of valid pixels, Punish Charbonnier .

[0202] The computational image-3D-text multimodal contrast loss By aligning the information noise contrast estimation (InfoNCE) along the three paths of text, image, and 3D, the capabilities of the open vocabulary are significantly enhanced.

[0203] For each predicted 3D detection box (The center is given by the regression branch) ,size Orientation ), define three modal embeddings and perform multi-view alignment:

[0204] (1) 3D bounding box embedding: Gaussian features within the convergent bounding box are used to obtain the 3D bounding box embedding. (Pool functions such as average / weighted attention), and normalization. ;

[0205] (2) Image patch embedding: embedding the 3D detection box Projecting onto the image plane yields a 2D region. On the two-dimensional feature map, RoIAlign / patch pooling is used to obtain... and normalization ;

[0206] (3) Text embedding: Candidate open vocabulary text set Normalization ;

[0207] Set the temperature hyperparameters for 3D-text alignment and image-3D alignment respectively. , Define the InfoNCE contrast loss:

[0208] a) The 3D-text alignment loss is:

[0209] ,

[0210] b) The image-3D alignment loss is:

[0211] ,

[0212] Thus, the multimodal contrast loss is obtained:

[0213] .

[0214] in For weights ( In this embodiment, the value is 0. In this embodiment, the value is 1). For the first A set of candidate texts for each sample (including positive and sampled negative texts).

[0215] During model training, this step achieves joint optimization of information from different modalities by designing a loss function that integrates multimodal features. This method effectively improves the robustness of the system and supports prediction with open vocabularies, making it more widely applicable compared to traditional methods with closed vocabularies. This addresses the limitations of existing methods in cross-modal fusion and open-vocabulary prediction, overcomes the system's susceptibility to single-modal constraints in complex tasks, and achieves more robust and universally applicable prediction results.

[0216] Step 4 includes: performing three-dimensional nonmaximum suppression on the three-dimensional detection results, and outputting the final detection results as the product of class probability and centrality.

[0217] This invention provides a monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A monocular unsupervised open-vocabulary indoor target detection method based on 3D Gaussian, characterized in that, Includes the following steps: Step 1: Collect real indoor scene images as training set, and generate 3D pseudo-labels for the training set through 2D candidate box detection, monocular depth prediction, gradient-based depth smoothing completion, and 3D projection. Step 2: Construct a Gaussian-based monocular open vocabulary 3D detection network, initialize the Gaussian set, extract multimodal features and perform cross-modal fusion, iteratively update the Gaussian set, and use an anchorless detection head to predict the 3D detection box set and semantic features; Step 3: Train a Gaussian-based monocular open vocabulary 3D detection network using pseudo-labels of real scene images, match and classify the predicted semantic features with the text embeddings of the open vocabulary, and optimize the network with reconstruction contrast loss. Step 4: Use the trained 3D detection network to perform target detection.

2. The monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian as described in claim 1, characterized in that, The generation of pseudo-labels for the training set described in step 1 includes the following steps: Step 1-1: Run a pre-trained 2D detector with an open vocabulary on the collected real scene images and input prompt words, freeze its parameters to obtain a set of two-dimensional detection boxes, and use a threshold strategy to filter the detection boxes. Steps 1-2: Perform monocular depth prediction on the real scene image using the monocular depth estimation module to obtain an initial depth map, and use a depth correction algorithm to correct the predicted initial depth map to generate a dense depth map. Steps 1-3 combine the two-dimensional candidate boxes with the dense depth map, generate a three-dimensional point cloud through camera model projection, and cluster them to form three-dimensional pseudo-labels.

3. The monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian as described in claim 2, characterized in that, In steps 1-2, the monocular depth estimation module includes a depth estimation module and a depth correction algorithm; Step 1-2-1: The depth estimation module uses a pre-trained depth estimation network and freezes its parameters to generate an initial depth map D, which represents the absolute physical distance from the pixel to the camera. Step 1-2-2: The initial depth map D is modified by a depth correction algorithm to suppress outliers and edge errors, resulting in a corrected dense depth map.

4. The monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian as described in claim 3, characterized in that, Step 1-2-2 specifically includes: Step 1-2-2-1: Perform object edge region recognition, outlier recognition, and image edge region setting to determine the area to be corrected; Step 1-2-2-2, Depth value regeneration: Build a K-dimensional tree with pixels that are not marked as areas to be corrected, and search for the nearest valid neighbor for each pixel to be corrected. Assign its depth value or guide interpolation to obtain the corrected dense depth map.

5. The monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian as described in claim 2, characterized in that, Steps 1-3 include: Step 1-3-1: Project the camera model to generate a 3D point cloud; Step 1-3-2: Select a set of pixels based on the two-dimensional detection box, and obtain the corresponding three-dimensional point set accordingly. ; Step 1-3-3: Density clustering and statistical filtering are used for 3D point cloud clustering and segmentation; Steps 1-3-4: Based on clustering and segmentation of the 3D point set Fit 3D target pseudo-labels.

6. The monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian as described in claim 5, characterized in that, The monocular open vocabulary 3D detection network described in step 2 consists of a 2D side, a 3D side, deformable cross attention, 3D interaction, and a regression head; Two-dimensional side extracts multi-scale two-dimensional semantic features from two-dimensional encoding. ; The three-dimensional side extracts three-dimensional semantic features by a three-dimensional convolutional backbone network. The three-dimensional convolutional backbone network adopts a four-stage structure of sub-manifold sparse convolutional blocks and stride sparse convolution. The three-dimensional neck network adopts transposed sparse convolution upsampling and is fused with lateral connections. Deformable cross-attention: with Gaussian center parameter Projection point obtained by camera projection Using this as a reference point, in multi-scale two-dimensional semantic features Multi-scale deformable sampling and linear fusion are performed to obtain Gaussian features with two-dimensional side encoding. ; 3D Interaction: Gaussian Center Parameters Voxelization and multi-scale 3D features The Gaussian features of the three-dimensional side encoding are obtained by aggregating the neighborhood three-dimensional context through multi-layer sparse convolution / residual blocks. ; Regression Head: Gaussian features encoded in two-dimensional sides Gaussian features with 3D side encoding Fusion into aggregated Gaussian features The Gaussian prediction parameter increments are output through multiple MLP layers.

7. The monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian as described in claim 6, characterized in that, Step 2 includes: Step 2-1: Initialize N 3D Gaussian seeds within the current camera's view frustum to obtain the initial Gaussian set. ; Step 2-2: The 2D encoder extracts multi-scale 2D semantic features from the input indoor real-world scene image. And based on the camera's internal and external parameters Compared with the initial depth map in step 1-2-1 Backprojection / voxalization to a 3D sparse voxel space; feature extraction and top-down / bottom-up fusion of a 3D convolutional backbone network and a 3D neck network to obtain multi-scale 3D features. ; Steps 2-3: The 3D convolutional backbone network uses the Gaussian center parameters of the current iteration. For the query, deformable cross-attention is used to extract multi-scale two-dimensional semantic features. Mid-sample two-dimensional semantic cues, and simultaneously Voxelization and multi-scale 3D features Sparse 3D convolutional interactions are performed on the surface and fused with depth priors to form aggregated feature vectors for each Gaussian. Used for iterative prediction of Gaussian parameters; Steps 2-4, 2-2, and 2-3 are iterated R times to obtain a convergent Gaussian set. The Gaussian prediction parameters are incremented by a parallel regression branch, and an update of the Gaussian parameters is performed accordingly. Steps 2-5: At the non-empty voxel position at the center of the Gaussian, use an anchorless 3D detection head to make predictions and output a set of 3D detection boxes.

8. The monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian as described in claim 7, characterized in that, The frameless detection head in steps 2-5 includes: Multi-scale three-branch structure with shared weights: 3D features at each scale Several sparse convolutional layers, normalization layers, and activation layers are stacked on top of each other, and then the layers branch into regression, classification, and centrality branches. The regression branch: predicts the center of the 3D bounding box. ,size Orientation ; The classification branch: outputs semantic features. The category / cue embedding obtained by the text encoder with frozen parameters Calculate (normalized) cosine similarity to achieve open vocabulary classification; The centrality branch outputs the centrality confidence score, which is multiplied by the classification score to form a candidate score. Post-processing: Based on the score, perform 3D NMS or rotated NMS to output the final detection box.

9. The monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian as described in claim 7, characterized in that, Step 3 includes: A Gaussian-based monocular open-vocabulary 3D detection network is trained using pseudo-labels from real-world images to predict semantic features. Match and classify text embeddings with an open vocabulary, and calculate the total training loss. .

10. The monocular unsupervised open vocabulary indoor target detection method based on three-dimensional Gaussian as described in claim 9, characterized in that, Step 4 includes: performing three-dimensional nonmaximum suppression on the three-dimensional detection results, and outputting the final detection results as the product of class probability and centrality.

Images