3D scene object detection method based on 3DGS technology
By generating sparse point clouds and Gaussian sets, and combining cross-attention and contrastive loss functions to optimize the 3D object detection method, the problem of object prediction box output under long text descriptions is solved, and the model's localization and understanding capabilities in three-dimensional space are improved.
Patent Information
- Application Number
- CN202511758175.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-02-24
AI Technical Summary
Existing 3D object detection methods based on 3DGS technology struggle to accurately output object prediction boxes under long text descriptions and neglect object position information in 3D scenes, affecting the model's understanding performance.
By acquiring multi-view images and camera parameters, a sparse point cloud is generated and a Gaussian set is initialized. A cross-attention mechanism is used to generate hybrid features. The contrastive loss function is used to optimize the matching between the Gaussian and the text. The IoU and L1 loss functions are constructed for model training, and the two-dimensional bounding box of the target object is output.
It improves the accuracy and generalization ability of the model in target localization in 3D space, especially in complex scenes, and is able to better understand and express the semantic information of 3D scenes.
Smart Images
Figure CN121564701A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a 3D scene object detection method based on 3DGS technology. Background Technology
[0002] 3D object detection based on 3DGS technology aims to use 3DGS as a scene representation to identify and locate target objects in three-dimensional space, typically outputting the object's 3D bounding box and category information. Compared to 2D object detection, it not only needs to identify objects but also estimate their position, size, and orientation in space, thus providing more comprehensive geometric and semantic information. 3D object detection is not only a fundamental step in understanding and parsing the 3D environment but also a prerequisite for many high-level applications. 3D object detection can accurately locate the geometric position and pose of target objects in space, providing richer structured information than 2D detection, which helps to achieve more reliable perception and interaction.
[0003] Existing methods for using 3DGS technology as scene representation to achieve 3D object detection can be broadly classified into two categories: (1) Single-scene optimization methods. These methods typically optimize the 3DGS representation directly on an image of a single scene, learn Gaussian parameters by rendering consistency constraints, and then combine 2D detection or segmentation results to improve semantic information. Candidate regions or bounding boxes are extracted from the Gaussian representation for object detection within the scene. The advantage is that the results are detailed and fit the specific scene, but the generalization ability is insufficient, and each detection requires re-optimization. (2) Multi-scene generalization methods. These methods train the network on large-scale multi-scene data, enabling it to directly predict Gaussian parameters and have cross-scene semantic understanding capabilities, thereby quickly completing 3D representation and detection in new scenes. They are usually combined with visual language models or multimodal supervision to support open vocabulary detection and semantic query. The advantages are strong generalization, high inference efficiency, and suitability for large-scale applications.
[0004] Existing single-scene optimization methods primarily utilize 2D base models to achieve knowledge transfer from 3D scenes. They render a specific viewpoint of a 3D scene into 2D using 3DGS technology, and then use a 2D visual language model to transfer and distill 2D knowledge into the 3D model, thus achieving a transition from 2D to 3D. However, most of these methods focus on object detection in 3D scenes guided by short text prompts, and the models lack the ability to understand complex descriptions in long texts, failing to fully integrate these descriptions to accurately output object prediction boxes. Furthermore, these methods neglect the crucial information of object positions within the 3D scene during training, thus affecting the model's understanding of the 3D scene. Summary of the Invention
[0005] The purpose of this invention is to provide a 3D scene object detection method based on 3DGS technology, which aims to solve or improve at least one of the above-mentioned technical problems.
[0006] To achieve the above objectives, the present invention provides the following solution: A 3D scene object detection method based on 3DGS technology includes: Acquire multi-view images of the 3D scene and the corresponding camera parameters, and obtain the prompt text for the object detection task; Based on multi-view images and camera parameters, a sparse point cloud is generated using a 3D reconstruction tool, and a Gaussian set is initialized. ; Gaussian set In each three-dimensional Gaussian semantic vectors and spatial location Fusion, combining the prompt text, and extracting the textual features of the prompt text. Generate hybrid features through cross-attention mechanism ; Based on hybrid characteristics For each pair of samples, positive and negative samples are constructed. By comparing the loss functions, the distance between semantically relevant Gaussian samples and the prompt text is narrowed, while irrelevant Gaussian samples are pushed away. The optimized mixed features are then output. ; Based on the optimized hybrid characteristics and embedded sequences Calculate each three-dimensional Gaussian The semantic response score of the prompt text is used to generate a semantic response map; The semantic response map is compressed into a one-dimensional feature vector. The two-dimensional bounding box coordinates of the target object are regressed through the prediction head. The annotation tool provides supervision signals to obtain the predicted bounding box and the ground truth bounding box. Based on the predicted bounding box and the true bounding box, construct the IoU loss function and the L1 loss function, combine them with the contrastive loss function to construct the total loss function, and output the gradient signal for backpropagation to update the model; Based on the predicted bounding box parameters, draw a rectangle on the image at the corresponding viewpoint, add prompt text as a semantic label, and generate the detection result.
[0007] Furthermore, based on multi-view images and camera parameters, a sparse point cloud is generated using a 3D reconstruction tool, and a Gaussian set is initialized. ,include: Sparse reconstruction of multi-view images and camera parameters is performed using COLMAP or SfM tools to obtain sparse 3D point clouds. and camera pose; The expression for a sparse 3D point cloud is: In the formula, This represents the spatial location of the i-th sparse point; N is the total number of sparse points. Based on the spatial location of each sparse point Initialize a three-dimensional Gaussian vector with the center as the center. And assign properties to the quadruples to obtain the Gaussian set. ; 3D Gaussian The expression is: In the formula, Indicates the spatial location of the i-th sparse point; Let be the covariance matrix of Gaussian; It is the weighted average of the pixel colors of the i-th sparse point in the images from each viewpoint; The semantic vector assigned to each Gaussian is initially a zero vector.
[0008] Furthermore, the Gaussian set In each three-dimensional Gaussian semantic vectors and spatial location Fusion, combining the prompt text, and extracting the textual features of the prompt text. Generate hybrid features through cross-attention mechanism ,include: Input the prompt text into the pre-trained BERT model to extract the embedding sequence. And calculate text features ; In the formula, J is the number of tokens in the prompt text after word segmentation, i.e., the length of the embedded sequence; This involves performing an arithmetic average of the embedded sequences along the word dimension; d represents the hidden dimension of the BERT model, i.e., the text features. The vector dimension; Embedded in text; For each three-dimensional Gaussian semantic vector and spatial location The fusion process yields a fusion vector, expressed as: In the formula, This is the fusion vector; To convert spatial coordinates Mapping to semantic dimension operations; According to the embedded sequence and fusion vector Calculate cross-attention to obtain mixed features. The expression is: In the formula, For cross-attention mechanism, text embedding For the query, Gaussian fusion vector Calculate the weighted semantic response for the key and value.
[0009] Furthermore, by calculating the mixed features Features of prompt text The dot product similarity is used to construct positive and negative sample pairs. By comparing the loss functions, the distance between semantically related Gaussian signals and prompt text is narrowed, while irrelevant Gaussian signals are pushed away. The optimized mixed features are then output. ,include: Calculate mixed features Features of prompt text The dot product similarity is used to determine positive samples. Three-dimensional Gaussian samples with a dot product similarity exceeding a preset threshold are considered positive samples. This process identifies a subset of Gaussian samples relevant to the prompt text, thus obtaining the positive sample set. ; Based on the positive sample set The negative sample set and the total sample set are obtained by the following expression: In the formula, For negative sample sets; For the total sample set; A contrastive loss function is constructed based on positive and negative sample pairs, and a three-dimensional Gaussian loss function is generated through backpropagation. semantic vectors Output optimized hybrid features The expression is: In the formula, To compare the loss values; Textual features for prompt text; positive sample set Mixed characteristics of neutral samples; This represents the mixed features of all positive and negative sample pairs; This is the temperature coefficient.
[0010] Furthermore, based on the optimized hybrid features and embedded sequences Calculate each three-dimensional Gaussian The semantic response score of the prompt text is used to generate a semantic response map, including: Calculate each three-dimensional Gaussian The semantic response score to the prompt text is expressed as: In the formula, The score represents the semantic response, reflecting the three-dimensional Gaussian response. The semantic relevance of the represented area to the prompt text; Embedded in text; Modify the original 3DGS rendering formula to As a semantic modulation factor, it generates a semantic response map, expressed as: In the formula, Here, H and W represent the height and width of the semantic response map, respectively; S represents the total number of 3D Gaussian vectors rendered for the current pixel v. The semantic response score for the m-th Gaussian; Let be the opacity of the m-th Gaussian; Let be the opacity of the j-th Gaussian.
[0011] Furthermore, the semantic response map is compressed into a one-dimensional feature vector, and the two-dimensional bounding box coordinates of the target object are regressed through the prediction head. Supervision signals are provided using annotation tools to obtain the predicted bounding box and the ground truth bounding box, including: Global average pooling (GAP) is applied to the semantic response map to obtain the feature vector. ; The feature vector z is sequentially input into the prediction head, which consists of two layers of a multilayer perceptron (MLP). The normalized bounding box parameters are output to obtain the predicted bounding box, expressed as: In the formula, For predicting the bounding box; x, y are normalized center coordinates; w, h are normalized width and height; The GroundingDINO prediction model is used to extract the ground bounding boxes corresponding to the prompt text from the original image. , as a monitoring signal.
[0012] Furthermore, based on the predicted and ground truth bounding boxes, an IoU loss function and an L1 loss function are constructed. Combined with the contrastive loss function, a total loss function is built. The output gradient signal is then used for backpropagation to update the model, including: The IoU loss function expression is: In the formula, Loss due to IoU; To predict the bounding box; The numerator is the area of the intersection of the two bounding boxes, and the denominator is the area of the union of the two bounding boxes. The L1 loss function expression is: In the formula, For L1 loss; x, y are the center coordinates of the predicted bounding box; w, h are the width and height of the predicted bounding box; , The center coordinates of the actual bounding box; , The width and height of the actual bounding box; The total loss expression is: In the formula, This represents the total loss value. To compare the loss values; Loss due to IoU; This is the L1 loss.
[0013] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This invention discloses a 3D scene object detection method based on 3DGS technology. The method inputs the position information of a 3D Gaussian object within the scene and introduces a semantic feature vector for each Gaussian object. It then selects a 2D model with strong text understanding capabilities for sentences of similar length and combines it with 3DGS to learn 3D scene semantic knowledge. Finally, it uses the 2D model to generate pseudo-predicted bounding boxes to supervise the model's final output. The method of this invention introduces the interaction between text query and Gaussian representation during the training phase, which improves the ability to locate targets in three-dimensional space.
[0014] The method of this invention introduces Gaussian-text contrastive learning. By constructing a contrastive learning loss, it improves the matching degree between 3D Gaussian and semantic information, thereby enhancing the model's ability to understand 3D scenes. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the original image from a certain perspective in three-dimensional space in this embodiment; Figure 3 This is a schematic diagram showing the original image annotated according to the prompt text in this embodiment. Detailed Implementation
[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0018] The purpose of this invention is to provide a 3D scene object detection method based on 3DGS technology, which aims to solve or improve at least one of the above-mentioned technical problems.
[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0020] like Figure 1 As shown, this invention provides a 3D scene object detection method based on 3DGS technology, including: Step 1: Obtain multi-view images of the 3D scene and the corresponding camera parameters, and obtain the prompt text for the object detection task; Among them, multi-view images require at least three static 3D scene images taken from different perspectives; camera parameters: intrinsic and extrinsic parameters corresponding to each image, used to establish the mapping relationship between image pixels and world coordinates; prompt text, such as "find the chopsticks next to the bowl", expresses the semantic description of the target to be detected in natural language.
[0021] Preprocessing is required for multi-view images and prompt text, including: normalizing the multi-view images to... ; Step 2: Based on multi-view images and camera parameters, generate sparse point clouds using 3D reconstruction tools, and initialize a 3D Gaussian distribution with geometric properties and learnable semantic vectors to form an initial differentiable representation of the 3D scene, including the following steps: Step 21: Use COLMAP or SfM tools to perform sparse reconstruction on multi-view images and camera parameters to obtain sparse 3D point clouds. and camera pose; The expression for a sparse 3D point cloud is: In the formula, This represents the spatial location of the i-th sparse point; N is the total number of sparse points. Step 22, based on the spatial location of each sparse point Initialize a three-dimensional Gaussian vector with the center as the center. And assign properties to the quadruples to obtain the Gaussian set. ; 3D Gaussian The expression is: In the formula, Indicates the spatial location of the i-th sparse point; Given the covariance matrix of a Gaussian, obtain the Gaussian shape based on the distribution of the local neighborhood of the point cloud and initialize it; It is the weighted average of the pixel colors of the i-th sparse point in the images from each viewpoint; The semantic vector assigned to each Gaussian is initially a zero vector; Step 3, Gaussian set In each three-dimensional Gaussian semantic vectors and spatial location Fusion, combining the prompt text, and extracting the textual features of the prompt text. The cross-attention mechanism generates hybrid features that integrate geometric and semantic information. It includes the following steps: Step 31: Input the prompt text into the pre-trained BERT model and extract the embedding sequence. And calculate text features ; In the formula, J is the number of tokens in the prompt text after word segmentation, i.e., the length of the embedded sequence; This involves performing an arithmetic average of the embedded sequences along the word dimension; d represents the hidden dimension of the BERT model, i.e., the text features. The vector dimension; Embedded in text; Step 32, for each three-dimensional Gaussian semantic vector and spatial location The fusion process yields a fusion vector, expressed as: In the formula, This is the fusion vector; To convert spatial coordinates Mapping to semantic dimension operations; Step 33, based on the embedded sequence and fusion vector Calculate cross-attention to obtain mixed features. The expression is: In the formula, For cross-attention mechanism, text embedding For the query, Gaussian fusion vector Calculate the weighted semantic response for the key and value.
[0022] The processing flow for cross-attention is as follows: embed the text and fusion vector Mapping to a unified attention space yields Query, Key, and Value respectively; then, the similarity between Query and each Key is calculated, and the attention weight of each Gaussian point to the text is obtained through the Softmax function; finally, these weights are used to perform a weighted summation of Value to generate text-guided semantic response features.
[0023] Step 4, calculate the mixed features Features of prompt text The dot product similarity is used to construct positive and negative sample pairs. By comparing the loss functions, the distance between semantically related Gaussian signals and prompt text is narrowed, while irrelevant Gaussian signals are pushed away. The optimized mixed features are then output. It includes the following steps: Step 41, Calculate the mixed features Features of prompt text The dot product similarity is used to determine positive samples. Three-dimensional Gaussian samples with a dot product similarity exceeding a preset threshold are considered positive samples. This process identifies a subset of Gaussian samples relevant to the prompt text, thus obtaining the positive sample set. ; Step 42, based on the positive sample set The negative sample set and the total sample set are obtained by the following expression: In the formula, For negative sample sets; For the total sample set; Step 43: Construct a contrastive loss function based on the positive and negative sample pairs, and perform a three-dimensional Gaussian loss function through backpropagation. semantic vectors Output optimized hybrid features The expression is: In the formula, To compare the loss values; Textual features for prompt text; positive sample set Mixed characteristics of neutral samples; This represents the mixed features of all positive and negative sample pairs; This is the temperature coefficient.
[0024] Step 5, based on the optimized hybrid features and embedded sequences Calculate each three-dimensional Gaussian The semantic response score for the prompt text is used to generate a semantic response map, which includes the following steps: Calculate each three-dimensional Gaussian The semantic response score to the prompt text is expressed as: In the formula, The score represents the semantic response, reflecting the three-dimensional Gaussian response. The semantic relevance of the represented area to the prompt text; Embedded in text; Modify the original 3DGS rendering formula to As a semantic modulation factor, it generates a semantic response map, expressed as: In the formula, For the semantic response map, H and W are the height and width of the semantic response map. The higher the pixel value, the more the corresponding position matches the prompt text description; S is the total number of three-dimensional Gaussians rendered for the current pixel v. The semantic response score for the m-th Gaussian; Let be the opacity of the m-th Gaussian; Let be the opacity of the j-th Gaussian.
[0025] In computer graphics and image processing, opacity in the above steps refers to the ability of a pixel or object to allow background color to pass through, which determines the degree of Gaussian contribution to the final generated semantic response map.
[0026] Step 6: Compress the semantic response map into a one-dimensional feature vector, regress the two-dimensional bounding box coordinates of the target object using the prediction head, and use annotation tools to provide supervision signals to obtain the predicted bounding box and the ground truth bounding box. This includes the following steps: Global average pooling (GAP) is applied to the semantic response map to obtain the feature vector. ; The feature vector z is sequentially input into a prediction head consisting of two layers of a Multilayer Perceptron (MLP). The prediction head comprises two parts: the first layer is a fully connected linear layer, used to map the input features to an intermediate semantic space and enhance non-linear expressive power with an activation function; the second layer is also a fully connected layer, which further maps the intermediate features to the final bounding box parameter dimension. After the successive transformations of the two MLP layers, the output vector is normalized to output the normalized bounding box parameters, thus obtaining the predicted bounding box, expressed as: In the formula, For predicting the bounding box; x, y are normalized center coordinates; w, h are normalized width and height; The GroundingDINO prediction model is used to extract the ground bounding boxes corresponding to the prompt text from the original image. , as a monitoring signal.
[0027] The true bounding box in this invention Instead of manual annotation, the data is generated with the assistance of a pre-trained GroundingDINO text-guided object detection model. The prompt text describing the target object and the corresponding original image are input into the GroundingDINO model. Based on its pre-training capabilities on large-scale image and text data, the model automatically locates the target region in the image that corresponds to the semantic meaning of the text and outputs the bounding box coordinates of that region.
[0028] Step 7: Construct the IoU loss function and L1 loss function based on the predicted bounding box and the ground truth bounding box. Combine them with the contrastive loss function to construct the total loss function. Output the gradient signal for backpropagation to update the model, including the following steps: The IoU loss function expression is: In the formula, Loss due to IoU; To predict the bounding box; The numerator is the area of the intersection of the two bounding boxes, and the denominator is the area of the union of the two bounding boxes.
[0029] The IoU loss value reflects the degree of overlap between the predicted bounding box and the ground truth bounding box; the greater the overlap, the closer the IoU is to 1. The loss function is 1 − IoU, which allows the model to continuously optimize the position and size of the predicted bounding box during training to maximize its overlap with the ground truth bounding box.
[0030] The L1 loss function expression is: In the formula, For L1 loss; x, y are the center coordinates of the predicted bounding box; w, h are the width and height of the predicted bounding box; , The center coordinates of the actual bounding box; , The width and height of the actual bounding box; L1 loss measures the deviation between the model output and the target value. The larger the difference, the greater the loss. When the difference is zero, the loss is also zero. It is often used to measure the accuracy of predictions in regression tasks.
[0031] The total loss expression is: In the formula, This represents the total loss value. To compare the loss values; Loss due to IoU; This is the L1 loss.
[0032] The total loss function consists of two parts: one is the contrastive learning loss. The first is used to align text semantics with Gaussian features; the second is IoU loss. This is used to constrain the spatial consistency between the predicted bounding box and the ground truth bounding box. The two are combined through a weighted sum, where... The weight is set to 0.1 to balance the contributions of semantic constraints and spatial constraints, so that the model can learn semantic matching ability during training while ensuring the spatial accuracy of the detection results.
[0033] Step 8: Based on the predicted bounding box parameters, draw a rectangle on the image at the corresponding viewpoint, add prompt text as a semantic label, and generate the detection result.
[0034] The technical solution of this invention introduces Gaussian position information and combines it with semantic feature vectors during the training phase, enabling 3DGS to better learn the three-dimensional semantic knowledge of the scene, and has a better understanding and expression ability of complex scenes compared with traditional methods.
[0035] The technical solution of this invention introduces the interaction between text query and Gaussian representation, and introduces the corresponding components between Gaussian and text as rendering variables, which enhances Gaussian's understanding and matching of text, breaks through the limitation of previous methods that only rely on single-view rendering features, and enables the model to more accurately locate targets in three-dimensional space, which is particularly suitable for tasks that require spatial reasoning.
[0036] The technical solution of this invention introduces Gaussian-text contrastive learning loss, which fully integrates semantic information and 3D information, improves the matching degree between each Gaussian and the corresponding text, and enables the model to accurately understand the Gaussian correspondence described by the user's text, and realize the 3D object detection task through rendering.
[0037] As shown in Table 1, in order to verify the performance of the method of the present invention in object detection compared with Feature-3DGS, LEGaussians, GS-Grouping and GOI, object detection results were performed on four scenes using the LERF-OVS dataset.
[0038] Table 1. Detection results of the LERF-OVS dataset.
[0039] In Table 1 above, ram. is an abbreviation for "RAMEN" scene, representing a tabletop scene containing objects such as ramen bowls, char siu, and eggs; fig. is an abbreviation for "FIGURINES" scene, which typically contains multiple small dolls, toys, and other densely arranged fine-grained objects; tea. is an abbreviation for "TEATIME" scene, simulating a tea table setting, containing items such as teapots, cups, and snacks; kit. is an abbreviation for "KITCHEN" scene, representing a kitchen environment with many types of objects, complex occlusion, and a cluttered background; avg. is the average detection accuracy across the four scenes mentioned above.
[0040] As shown in Table 1, the method of the present invention has achieved the best or near-best performance on the LERF-OVS open vocabulary detection task, especially in complex and dense scenarios, and has good generalization ability and practical application potential.
[0041] The following uses the prompt text "a pair of chopsticks next to the bowl" as an example to illustrate a specific embodiment of the present invention, such as... Figure 2 As shown, when a user inputs the prompt text "a pair of chopsticks next to a bowl," the system first encodes the text using the BERT model to obtain the corresponding semantic embedding; then, it associates the semantics of the text with any original scene image (such as...). Figure 2 The inputs (as shown) are fed into the GroundingDINO text guidance detection model, which automatically generates ground truth bounding boxes corresponding to the semantic meaning of the prompt. After obtaining the ground truth bounding boxes, they are combined with Gaussian fusion features obtained from multi-view image rendering of the scene. Cross-modal alignment of language and 3D Gaussian features is achieved through cross-attention, generating a semantically enhanced Gaussian representation. This feature is then processed by a prediction head consisting of two MLP layers to output a 2D predicted bounding box, which is then optimized using the ground truth bounding box as supervision. Finally, based on the predicted bounding box and visibility calculation, the system backprojects the detection results into a 3D Gaussian space, achieving precise 3D localization of "a pair of chopsticks next to a bowl," as shown. Figure 3 The marked box shown.
[0042] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0043] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A 3D scene object detection method based on 3DGS technology, characterized in that, include: Acquire multi-view images of the 3D scene and the corresponding camera parameters, and obtain the prompt text for the object detection task; Based on multi-view images and camera parameters, a sparse point cloud is generated using a 3D reconstruction tool, and a Gaussian set is initialized. ; Gaussian set In each three-dimensional Gaussian semantic vectors and spatial location Fusion, combining the prompt text, and extracting the textual features of the prompt text. Generate hybrid features through cross-attention mechanism ; By calculating the mixed features Features of prompt text The dot product similarity is used to construct positive and negative sample pairs. By comparing the loss functions, the distance between semantically related Gaussian signals and prompt text is narrowed, while irrelevant Gaussian signals are pushed away. The optimized mixed features are then output. ; Based on the optimized hybrid characteristics and embedded sequences Calculate each three-dimensional Gaussian The semantic response score of the prompt text is used to generate a semantic response map; The semantic response map is compressed into a one-dimensional feature vector. The two-dimensional bounding box coordinates of the target object are regressed through the prediction head. The annotation tool provides supervision signals to obtain the predicted bounding box and the ground truth bounding box. Based on the predicted bounding box and the true bounding box, construct the IoU loss function and the L1 loss function, combine them with the contrastive loss function to construct the total loss function, and output the gradient signal for backpropagation to update the model; Based on the predicted bounding box parameters, draw a rectangle on the image at the corresponding viewpoint, add prompt text as a semantic label, and generate the detection result.
2. The 3D scene object detection method based on 3DGS technology according to claim 1, characterized in that, The process involves generating a sparse point cloud using 3D reconstruction tools based on multi-view images and camera parameters, and initializing a Gaussian set. ,include: Sparse reconstruction of multi-view images and camera parameters is performed using COLMAP or SfM tools to obtain sparse 3D point clouds. and camera pose; The expression for a sparse 3D point cloud is: In the formula, This represents the spatial location of the i-th sparse point; N is the total number of sparse points. Based on the spatial location of each sparse point Initialize a three-dimensional Gaussian vector with the center as the center. And assign properties to the quadruples to obtain the Gaussian set. ; 3D Gaussian The expression is: In the formula, Indicates the spatial location of the i-th sparse point; Let be the covariance matrix of Gaussian; It is the weighted average of the pixel colors of the i-th sparse point in the images from each viewpoint; The semantic vector assigned to each Gaussian is initially a zero vector.
3. The 3D scene object detection method based on 3DGS technology according to claim 1, characterized in that, The Gaussian set In each three-dimensional Gaussian semantic vectors and spatial location Fusion, combining the prompt text, and extracting the textual features of the prompt text. Generate hybrid features through cross-attention mechanism ,include: Input the prompt text into the pre-trained BERT model to extract the embedding sequence. And calculate text features ; In the formula, J represents the number of tokens in the prompt text after word segmentation; This is an arithmetic average operation on the embedded sequences along the word dimension; d is the hidden dimension of the BERT model; Embedded in text; For each three-dimensional Gaussian semantic vector and spatial location The fusion process yields a fusion vector, expressed as: In the formula, This is the fusion vector; To convert spatial coordinates Mapping to semantic dimension operations; According to the embedded sequence and fusion vector Calculate cross-attention to obtain mixed features. The expression is: In the formula, For cross-attention mechanism, text embedding For the query, Gaussian fusion vector Calculate the weighted semantic response for the key and value.
4. The 3D scene object detection method based on 3DGS technology according to claim 1, characterized in that, By calculating the mixed features Features of prompt text The dot product similarity is used to construct positive and negative sample pairs. By comparing the loss functions, the distance between semantically related Gaussian signals and prompt text is narrowed, while irrelevant Gaussian signals are pushed away. The optimized mixed features are then output. ,include: Calculate mixed features Features of prompt text The dot product similarity is used to determine positive samples. Three-dimensional Gaussian samples with a dot product similarity exceeding a preset threshold are considered positive samples. This process identifies a subset of Gaussian samples relevant to the prompt text, thus obtaining the positive sample set. ; Based on the positive sample set The negative sample set and the total sample set are obtained by the following expression: In the formula, For negative sample sets; For the total sample set; A contrastive loss function is constructed based on positive and negative sample pairs, and a three-dimensional Gaussian loss function is generated through backpropagation. semantic vectors Output optimized hybrid features The expression is: In the formula, To compare the loss values; Textual features for prompt text; positive sample set Mixed characteristics of neutral samples; This represents the mixed features of all positive and negative sample pairs; This is the temperature coefficient.
5. The 3D scene object detection method based on 3DGS technology according to claim 1, characterized in that, Based on the optimized hybrid characteristics and embedded sequences Calculate each three-dimensional Gaussian The semantic response score of the prompt text is used to generate a semantic response map, including: Calculate each three-dimensional Gaussian The semantic response score to the prompt text is expressed as: In the formula, The semantic response score reflects the three-dimensional Gaussian response. The semantic relevance of the represented area to the prompt text; Embedded in text; Modify the original 3DGS rendering formula to As a semantic modulation factor, it generates a semantic response map, expressed as: In the formula, Here, H and W represent the height and width of the semantic response map, respectively; S represents the total number of 3D Gaussian vectors rendered for the current pixel v. The semantic response score for the m-th Gaussian; Let be the opacity of the m-th Gaussian; Let be the opacity of the j-th Gaussian.
6. The 3D scene object detection method based on 3DGS technology according to claim 1, characterized in that, The process of compressing the semantic response map into a one-dimensional feature vector, regressing the two-dimensional bounding box coordinates of the target object through a prediction head, and using annotation tools to provide supervision signals to obtain the predicted bounding box and the ground truth bounding box includes: Global average pooling (GAP) is applied to the semantic response map to obtain the feature vector. ; The feature vector z is sequentially input into the prediction head, which consists of two layers of a multilayer perceptron (MLP). The normalized bounding box parameters are output to obtain the predicted bounding box, expressed as: In the formula, For predicting the bounding box; x, y are normalized center coordinates; w, h are normalized width and height; The GroundingDINO prediction model is used to extract the ground bounding boxes corresponding to the prompt text from the original image. , as a monitoring signal.
7. The 3D scene object detection method based on 3DGS technology according to claim 1, characterized in that, The process of constructing an IoU loss function and an L1 loss function based on the predicted bounding box and the ground truth bounding box, combining them with the contrastive loss function to construct a total loss function, and outputting the gradient signal for backpropagation to update the model includes: The IoU loss function expression is: In the formula, Loss due to IoU; To predict the bounding box; The numerator is the area of the intersection of the two bounding boxes, and the denominator is the area of the union of the two bounding boxes. The L1 loss function expression is: In the formula, For L1 loss; x, y are the center coordinates of the predicted bounding box; w, h are the width and height of the predicted bounding box; , The center coordinates of the actual bounding box; , The width and height of the actual bounding box; The total loss expression is: In the formula, This represents the total loss value. To compare the loss values; Loss due to IoU; This is the L1 loss.