Layered distillation method oriented to understanding of three-dimensional scene of open world
By constructing a cross-modal hierarchical distillation model on the ScanNet dataset, the problems of data scarcity and knowledge transfer in the transition from 2D visual language models to 3D scene understanding are solved, improving the generalization ability of 3D scene understanding, especially in semantic segmentation and object detection tasks.
Patent Information
- Application Number
- CN202511923947.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-02-24
AI Technical Summary
Existing technologies face challenges such as data scarcity and insufficient knowledge transfer when transferring two-dimensional visual language models to three-dimensional scene understanding, which limits the generalization ability of three-dimensional scene understanding.
We pre-trained using the widely used ScanNet dataset and constructed a hierarchical distillation model by distilling knowledge across modal scenes, regions, and targets. This model transferred visual semantic knowledge to the 3D model. PointMAE and DINOv2 were used to extract point cloud and image features, and the CLIP encoder was combined to obtain natural language information. A unified multimodal representation space was constructed using image projection head and text projection head, and a distillation loss function was introduced for feature fusion.
It enhances the generalization ability of 3D representation and improves the performance of open-world 3D scene understanding, especially demonstrating competitiveness in semantic segmentation and object detection tasks.
Smart Images
Figure CN121564465A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of 3D scene understanding technology, specifically a hierarchical distillation method for understanding open-world 3D scenes. Background Technology
[0002] Open-world 2D scene understanding: Visual language pre-trained models achieve zero-shot learning capabilities to solve the problem of understanding unknown scenes through contrastive learning strategies on large-scale image-text pairs. Significant work has emerged in this field. OpenSeg integrates the CLIP model into open-vocabulary segmentation tasks. ZegFormer decouples semantic segmentation tasks into class-independent grouping and segmentation-level zero-shot classification tasks. OVSeg develops a two-stage model that includes mask generation based on a base segmentation model and CLIP adaptation using collected mask-class pairs by adjusting mask cues. MaskCLIP introduces a mask self-distillation target into a contrastive visual language framework to obtain a transferable visual model. CAT-Seg proposes spatial and class cost aggregation techniques that enhance the aggregation effect of cosine similarity scores between image and text embeddings, thereby achieving open-vocabulary semantic segmentation.
[0003] Open-world 3D scene understanding. PointLCIP uses rendered depth maps as a bridge to achieve cross-modal alignment between point clouds and text. CLIP2Point introduces a dual contrastive learning framework to jointly optimize a CLIP's image encoder and depth image encoder, enhancing semantic consistency. PointCLIP V2 further integrates multi-view projection and utilizes a large language model to generate text cues, improving zero-shot performance. ULIP and ULIP-2 construct point cloud-image-text triples to achieve cross-modal alignment. However, these works mainly focus on target-level open vocabulary recognition tasks. For 3D semantic segmentation tasks, OpenScene jointly embeds 3D point features into text and image features in the CLIP feature space to achieve zero-shot 3D dense prediction. PLA uses images as a connecting bridge to build associations between 3D data and language, promoting the transfer of knowledge from 2D visual language models to 3D scene understanding models. Bridge3D distills image and text knowledge from the base model into 3D representation learning in a self-supervised manner.
[0004] Specifically, the shortcomings of existing technologies are as follows:
[0005] 1. Unlike two-dimensional open-world scene understanding models, directly applying the two-dimensional visual language model paradigm to three-dimensional scene understanding faces significant challenges. First, large-scale, high-quality point cloud-text pairs are still scarce. Second, collecting and labeling high-quality point cloud data is time-consuming and labor-intensive, limiting generalization capabilities.
[0006] 2. Existing open-world 3D scene understanding methods primarily utilize color or depth images as bridges to build connections between 3D data and language, enabling the transfer of knowledge from 2D visual language models to the 3D domain. However, these methods face the problem of insufficient knowledge transfer, limiting the generalization ability of the learned 3D representations / features. Furthermore, the existing domain differences between 3D data, images, and text further restrict the performance of 3D scene understanding.
[0007] Therefore, a new solution is needed to address the above problems. Summary of the Invention
[0008] The purpose of this invention is to provide a hierarchical distillation method for understanding open-world 3D scenes, in order to solve the technical problems mentioned in the background art.
[0009] To achieve the above objectives, the present invention provides the following technical solution: a hierarchical distillation method for understanding open-world 3D scenes, comprising at least the following steps:
[0010] S1: Pre-training data preparation, using the widely used ScanNet dataset, which contains 1531 scan data from homes, offices, and hotels;
[0011] S2: Perform cross-modal scene-level knowledge distillation and build a scene-level knowledge distillation module, which is used to transfer visual semantic knowledge from the visual language model to the three-dimensional model.
[0012] S3: Perform cross-modal region-level knowledge distillation, and introduce a region-level knowledge distillation strategy to help student models better understand local semantics and details of different regions;
[0013] S4: Perform cross-modal target-level knowledge distillation. The target-level knowledge distillation module is designed to achieve understanding of target-level features, and finally obtain the overall loss function.
[0014] Furthermore, S1 includes at least the following steps:
[0015] The ScanNet dataset provides color images. and depth images This eliminates the need for additional data generation.
[0016] For each scene's color image A visual language model for generating subtitles from images. Generate scene description ;
[0017] For region-level processing, each color image is divided into four small blocks. These regions are input into a visual language model for image-to-capture captions to generate corresponding language descriptions. ;
[0018] In addition, a segmentation model, namely the SAM model, is used. To generate segmentation mask embedding features for each image .
[0019] Furthermore, S2 includes at least the following steps:
[0020] First, the following formula is used to reduce the domain difference between 2D images and 3D point clouds by mapping the depth image to the 3D point cloud space;
[0021]
[0022] Where (u,v) are pixel coordinates, and Z is the depth value; () indicates focal length; Indicates the coordinates of the optical center; () indicates offset;
[0023] For language descriptions, the traditional method of manually setting prompts or templates is no longer relied upon, as this method lacks sufficient semantic description. Instead, generated subtitles are used. Provide a more detailed text description;
[0024] Following these rules, construct scenario-level triplet data ( (I,T) to pre-train 3D models;
[0025] Then, for multimodal feature extraction, PointMAE is used as the 3D encoder. To capture scene-level point cloud features DINOv2 was selected as the image encoder. To extract high-quality two-dimensional visual features Using CLIP's text encoder Obtaining semantic information from natural language ;
[0026] ( )
[0027] (I)
[0028] (T)
[0029] To construct a unified multimodal representation space, an image projection head is used. and text projection head Mapping 3D features to the latent feature space of images and text;
[0030] ( ),
[0031] Finally, the distillation loss function is used as follows:
[0032]
[0033] Make and With the corresponding and Make them as similar as possible.
[0034] Furthermore, S3 includes at least the following steps:
[0035] First, use 2 2. Partitioning strategy;
[0036] Subsequently, the same image and text encoders as those used at the scene level are applied to process the image regions and their corresponding text descriptions, respectively, to obtain the region image features. and text embedding features ;
[0037] ( )
[0038] ( )
[0039] Finally, the obtained scene-level 3D features It is divided into four regions, each region corresponding to an image patch;
[0040] To integrate region-level 3D, visual, and textual features, a region-specific projection head is further introduced. and The segmented three-dimensional features are projected onto the representation space of images and text;
[0041] ( ), ( )
[0042] here, Indicates a partitioning operation;
[0043] The region-level loss function is defined as follows: .
[0044] Furthermore, S4 includes at least the following steps:
[0045] First, the decoder output is introduced as the target-level 3D feature. ;
[0046] , =Decoder(Mask( ))
[0047] Among them, Decoder( ) and Mask () indicates decoding and masking operations;
[0048] Then, the segmentation mask generated by the SAM model is embedded with the feature M and the corresponding text description feature. Combined, the target-level distillation loss function is formalized as:
[0049] ,
[0050] in, Indicates the target projection head, used to... Projected onto the fused representation space; Refers to the fusion operation; Indicates the characteristics after fusion;
[0051] Finally, process the reconstructed features. To generate point clouds ,calculate Input point cloud Chamfer distance between them;
[0052]
[0053] The overall loss function is defined as follows: .
[0054] Compared with the prior art, the beneficial effects of the present invention are:
[0055] This invention constructs a hierarchical distillation model for understanding open-world 3D scenes. The model framework integrates cross-modal scene-level knowledge distillation, cross-modal region-level knowledge distillation, and cross-modal target-level knowledge distillation modules, which transfer diverse and hierarchical visual-semantic knowledge from visual language models to the 3D domain, thereby improving the generalization ability of 3D representation. Attached Figure Description
[0056] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0057] Figure 1 This is a framework diagram of the hierarchical distillation model for understanding open-world 3D scenes in this invention;
[0058] Figure 2 This is a comparison diagram of different region division strategies of the present invention;
[0059] Figure 3 This is a visualization of the open-world 3D semantic segmentation results of this invention on the ScanNet dataset;
[0060] Figure 4 This is a visualization of the open-world 3D semantic segmentation results of this invention on the S3DIS dataset;
[0061] Figure 5 This is a visualization of the 3D object detection results of this invention on the ScanNet v2 dataset. Detailed Implementation
[0062] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0063] Please see Figure 1 This paper showcases the hierarchical distillation pre-training framework for open-world 3D scene understanding proposed in this patent. The framework integrates three distillation modules: cross-modal scene-level distillation, cross-modal region-level distillation, and cross-modal target-level distillation, enabling progressive capture and integration of 3D, visual, and textual features, thereby enhancing the generalization ability of 3D representations.
[0064] A hierarchical distillation method for understanding open-world 3D scenes includes at least the following steps:
[0065] S1: Pre-training data preparation, using the widely used ScanNet dataset, which contains 1531 scan data from homes, offices, and hotels;
[0066] S2: Perform cross-modal scene-level knowledge distillation and build a scene-level knowledge distillation module. The scene-level knowledge distillation module is used to transfer visual semantic knowledge from the visual language model to the 3D model.
[0067] S3: Perform cross-modal region-level knowledge distillation, and introduce a region-level knowledge distillation strategy to help student models better understand local semantics and details of different regions;
[0068] S4: Perform cross-modal target-level knowledge distillation. The target-level knowledge distillation module is designed to achieve understanding of target-level features and finally obtain the overall loss function.
[0069] S1 includes at least the following steps:
[0070] The ScanNet dataset provides color images. and depth images This eliminates the need for additional data generation.
[0071] For each scene's color image A visual language model for generating subtitles from images. Generate scene description ;
[0072] For region-level processing, each color image is divided into four small blocks. These regions are input into a visual language model for image-to-capture captions to generate corresponding language descriptions. ;
[0073] In addition, a segmentation model, namely the SAM model, is used. To generate segmentation mask embedding features for each image .
[0074] S2 includes at least the following steps:
[0075] First, the following formula is used to reduce the domain difference between 2D images and 3D point clouds by mapping the depth image to the 3D point cloud space;
[0076]
[0077] Where (u,v) are pixel coordinates, and Z is the depth value; () indicates focal length; Indicates the coordinates of the optical center; () indicates offset;
[0078] For language descriptions, instead of relying on traditional manually set prompts or templates (which lack sufficient semantic description), generated subtitles are used. Provide a more detailed text description;
[0079] Following these rules, construct scenario-level triplet data ( (I,T) to pre-train 3D models;
[0080] Then, for multimodal feature extraction, PointMAE is used as the 3D encoder. To capture scene-level point cloud features DINOv2 was selected as the image encoder. To extract high-quality two-dimensional visual features Using CLIP's text encoder Obtaining semantic information from natural language ;
[0081] ( )
[0082] (I)
[0083] (T)
[0084] To construct a unified multimodal representation space, an image projection head is used. and text projection head Mapping 3D features to the latent feature space of images and text;
[0085] ( ),
[0086] Finally, the distillation loss function is used as follows:
[0087]
[0088] Make and With the corresponding and Make them as similar as possible.
[0089] S3 includes at least the following steps:
[0090] First, use 2 2. Partitioning Strategy; This configuration offers many advantages. First, compared to finer-grained partitioning, such as 3... 3 or 4 4,2 The segmentation method provides a relatively moderate region size, effectively balancing computational demands and information transfer. This allows for the capture of local details while maintaining global contextual relationships. Secondly, in caption generation based on visual language models, overly fine segmentation can lead to semantically ambiguous region descriptions. For example... Figure 2 As shown, in 4 4. The text description of the image in the lower right corner of the partitioning strategy is not accurate enough, which is not conducive to the learning of region knowledge distillation.
[0091] Subsequently, the same image and text encoders as those used at the scene level are applied to process the image regions and their corresponding text descriptions, respectively, to obtain the region image features. and text embedding features ;
[0092] ( )
[0093] ( )
[0094] Finally, the obtained scene-level 3D features The image is divided into four regions, each corresponding to a single image patch. This ensures the consistency of global information and facilitates the effective transfer of contextual information between different regions.
[0095] To integrate region-level 3D, visual, and textual features, a region-specific projection head is further introduced. and The segmented three-dimensional features are projected onto the representation space of images and text;
[0096] ( ), ( )
[0097] here, Indicates a partitioning operation;
[0098] The region-level loss function is defined as follows: .
[0099] S4 includes at least the following steps:
[0100] In the 3D mask autoencoder framework, the decoder output contains point-level features that encode global shape and local details. This makes the decoder features particularly suitable for target-level representation, especially in fine-grained tasks such as semantic segmentation. Furthermore, point-level decoder features can be aligned with pixel-level features. Therefore, following the Bridge3D approach, this invention first introduces the decoder output as a target-level 3D feature. ;
[0101] , =Decoder(Mask( ))
[0102] Among them, Decoder( ) and Mask () indicates decoding and masking operations;
[0103] Then, the segmentation mask generated by the SAM model is embedded with the feature M and the corresponding text description feature. Combined, the target-level distillation loss function is formalized as:
[0104] ,
[0105] in, Indicates the target projection head, used to... Projected onto the fused representation space; Refers to the fusion operation; Indicates the characteristics after fusion;
[0106] Finally, process the reconstructed features. To generate point clouds ,calculate Input point cloud Chamfer distance between them;
[0107]
[0108] The overall loss function is defined as follows: .
[0109] Based on the above embodiments, the following experimental verification is proposed:
[0110] A. Dataset and Evaluation Metrics
[0111] Datasets: To validate the effectiveness of the method, experiments were conducted on three publicly available datasets. ScanNet and S3DIS were used for open-world 3D semantic segmentation, and ScanNet V2 was used for 3D object detection. Following the PLA strategy, ScanNet's 19 categories were divided into three base / new category strategies: B15 / N4, B12 / N7, and B8 / N11. For S3DIS, the 12 categories were divided into two base / new category strategies: B8 / N4 and B6 / N6, where B and N represent the base and new categories, respectively. This partitioning strategy ensures fair comparison in an open-world setting.
[0112] Evaluation metrics: For 3D semantic segmentation, the mean intersection-union ratio (mIoU) and hIoU are used, with mIoU calculated separately for the base class and the new class. B and mIoU N For 3D target detection, the standard average accuracy (AP) at IoU thresholds of 0.25 and 0.50 is primarily used. 25 and AP 50 .
[0113] B. Implementation Details
[0114] Pre-training: Adjust the original input image and the segmented image patches to 640. The file size is 480 resolution. During the pre-training phase, the AdamW optimizer is used with a learning rate of 5e-4, weight decay of 5e-2, and a batch size of 64. The architecture consists of a 6-layer encoder and a 2-layer decoder.
[0115] Open-world 3D semantic segmentation: For the ScanNet and S3DIS datasets, the AdamW optimizer is mainly used with a learning rate of 1e-6 and a weight decay of 5e-2. The batch size is set to 32 for ScanNet and 16 for S3DIS.
[0116] 3D object detection: 3DETR is used as the point cloud encoder. The framework uses the AdamW optimizer with a learning rate of 5e-4, weight decay of 0.1, and a batch size of 8.
[0117] All experiments were run on three NVIDIA RTX 3090 GPUs.
[0118] C. Experimental Results
[0119] Open-world 3D semantic segmentation. Tables 1 and 2 present the quantitative segmentation results of the method described in this patent on the ScanNet and S3DIS datasets. "N prior" indicates whether new class names were provided during training. These results demonstrate that the method can achieve competitive performance on open-world understanding tasks. Clearly, the method outperforms the PLA method, improving the hIoU metric from 1.1% to 1.7% on the ScanNet dataset and from 3.7% to 4.5% on the S3DIS dataset. Figure 3 and Figure 4 Visualized results are presented, further validating the potential and capability of this method in open-world 3D scene understanding tasks, and demonstrating its ability to predict new categories.
[0120] Table 1. Open-world 3D semantic segmentation results on the ScanNet dataset.
[0121]
[0122] Table 2. Open-world semantic segmentation results on the S3DIS dataset.
[0123]
[0124] 3D object detection. Table 3 presents the experimental results for 3D object detection. Compared with the baseline model 3DETR, this method achieves an accuracy of 65.9%. 25，This represents a 3.8% improvement over the baseline method and a 44.7% improvement in AP. 50 This represents a 6.8% improvement over the baseline method. These findings validate the effectiveness of our proposed method in downstream object detection tasks. Meanwhile, Figure 5 Visualization results of the target detection are presented. These results further demonstrate that the proposed method can provide more granular target detection capabilities and can detect targets that 3DETR cannot detect.
[0125] Table 3. 3D object detection results on the ScanNet v2 dataset
[0126]
[0127] D. Ablation test
[0128] Table 4 provides ablation experiment results to verify the effectiveness of the cross-modal scene-level knowledge distillation, cross-modal region-level knowledge distillation, and cross-modal target-level knowledge distillation modules in this method. It can be seen that each distillation module plays a crucial role in open-world 3D semantic segmentation and 3D target detection.
[0129] Table 4 Ablation experiments for different levels of distillation modules
[0130]
[0131] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
Claims
1. A hierarchical distillation method for understanding open-world 3D scenes, characterized in that: At least the following steps are included: S1: Pre-training data preparation, using the widely used ScanNet dataset, which contains 1531 scan data from homes, offices, and hotels; S2: Perform cross-modal scene-level knowledge distillation and build a scene-level knowledge distillation module, which is used to transfer visual semantic knowledge from the visual language model to the three-dimensional model. S3: Perform cross-modal region-level knowledge distillation, and introduce a region-level knowledge distillation strategy to help student models better understand local semantics and details of different regions; S4: Perform cross-modal target-level knowledge distillation. The target-level knowledge distillation module is designed to achieve understanding of target-level features and finally obtain the overall loss function.
2. The hierarchical distillation method for understanding open-world 3D scenes according to claim 1, characterized in that: S1 includes at least the following steps: The ScanNet dataset provides color images. and depth images This eliminates the need for additional data generation. For each scene's color image A visual language model for generating subtitles from images. Generate scene description ; For region-level processing, each color image is divided into four small blocks. These regions are input into a visual language model for image-to-capture captions to generate corresponding language descriptions. ; In addition, a segmentation model, namely the SAM model, is used. To generate segmentation mask embedding features for each image .
3. The hierarchical distillation method for understanding open-world 3D scenes according to claim 2, characterized in that: S2 includes at least the following steps: First, the following formula is used to reduce the domain difference between 2D images and 3D point clouds by mapping the depth image to the 3D point cloud space; Where (u,v) are pixel coordinates, and Z is the depth value; () indicates focal length; Indicates the coordinates of the optical center; () indicates offset; For language descriptions, the generated subtitles are used. Provide a more detailed text description; Following these rules, construct scenario-level triplet data ( (I,T) to pre-train 3D models; Then, for multimodal feature extraction, PointMAE is used as the 3D encoder. To capture scene-level point cloud features DINOv2 was selected as the image encoder. To extract high-quality two-dimensional visual features Using CLIP's text encoder Obtaining semantic information from natural language ; ( ) (I) (T) To construct a unified multimodal representation space, an image projection head is used. and text projection head Mapping 3D features to the latent feature space of images and text; ( ), Finally, the distillation loss function is used as follows: Make and With the corresponding and Make them as similar as possible.
4. The hierarchical distillation method for understanding open-world 3D scenes according to claim 3, characterized in that: The S3 includes at least the following steps: First, use 2 2. Partitioning strategy; Subsequently, the same image and text encoders as those used at the scene level are applied to process the image regions and their corresponding text descriptions, respectively, to obtain the region image features. and text embedding features ; ( ) ( ) Finally, the obtained scene-level 3D features It is divided into four regions, each region corresponding to an image patch; To integrate region-level 3D, visual, and textual features, a region-specific projection head is further introduced. and The segmented three-dimensional features are projected onto the representation space of images and text; ( ), ( ) here, Indicates a partitioning operation; The region-level loss function is defined as follows: .
5. The hierarchical distillation method for understanding open-world 3D scenes according to claim 4, characterized in that: The S4 includes at least the following steps: First, the decoder output is introduced as the target-level 3D feature. ; , =Decoder(Mask( )) Among them, Decoder( ) and Mask () indicates decoding and masking operations; Then, the segmentation mask generated by the SAM model is embedded with the feature M and the corresponding text description feature. Combined, the target-level distillation loss function is formalized as: , in, Indicates the target projection head, used to... Projected onto the fused representation space; Refers to the fusion operation; Indicates the characteristics after fusion; Finally, process the reconstructed features. To generate point clouds ,calculate Input point cloud Chamfer distance between them; The overall loss function is defined as follows: .