Instance understanding 3D reconstruction model based on instance geometry Transform

Abstract

The invention belongs to the technical field of 3D reconstruction. The invention provides an instance understanding 3D (three-dimensional) reconstruction model based on an instance geometry Transform. According to the embodiment of the invention, a multi-view consistent instance segmentation pipeline is designed, and a large-scale training data set covering a plurality of scenes and a large number of images is constructed. A large Transform model is designed, and a cross-modal fusion module is designed to embed geometric features into instance representation, so that spatial reconstruction and instance understanding are unified in the model for end-to-end training. A scene understanding normal form based on instance perception is provided, and an instance mask is used as a bridge to realize plug-and-play integration with various visual language models and large-scale multi-modal models. According to the method, under the condition that only the image is input, the camera parameters, the depth map, the point diagram and the instance feature map with consistent 3D can be obtained in one-time forward propagation, and space intelligent tasks such as multi-view instance matching, 2D / 3D semantic segmentation and scene question and answer can be efficiently supported.

Description

Technical Field

[0001] This disclosure relates to the field of 3D reconstruction technology, and more particularly to an instance-understanding 3D reconstruction model based on instance geometry Transformer. Background Technology

[0002] In the pursuit of spatial intelligence, constructing a unified representation capable of simultaneously and accurately perceiving the geometric structure and rich semantic content of a scene, much like humans, has become a core objective in computer vision and artificial intelligence. This capability is crucial for downstream applications such as robot navigation and manipulation, augmented reality / virtual reality (AR / VR) interaction, and autonomous driving path planning. The human visual system possesses a natural advantage, processing the geometry and semantic information of the 3D world as interwoven dimensions, thereby achieving a coherent and accurate understanding of complex scenes. However, within the existing artificial intelligence technology system, achieving this unified reconstruction and understanding of both geometry and semantics still faces significant challenges.

[0003] Current mainstream methods for 3D scene understanding are mainly divided into two categories: "decoupled" and "tightly coupled," but both strategies have significant technical bottlenecks. Decoupled methods typically adopt a fragmented processing paradigm, treating 3D geometric reconstruction and high-level semantic understanding as two independent tasks. A typical workflow involves first predicting low-level geometry using multi-view stereo vision (MVS) or image-based 3D reconstruction techniques, followed by semantic annotation using visual language models (VLMs) or 2D segmentation models. This staged processing not only severs the mutually beneficial relationship between low-level geometry and high-level semantics, preventing geometric information from aiding semantic understanding, but also suffers from a serious "error propagation" problem. Errors in the early geometric reconstruction stage are directly transmitted and amplified in subsequent semantic understanding, severely hindering the overall model performance. Furthermore, decoupled methods usually require separate execution of the reconstruction and semantic understanding processes, resulting in high overall computational cost and long processing time.

[0004] To compensate for the aforementioned shortcomings, some studies have attempted to employ a tightly coupled strategy, aligning spatial models with specific VLMs (such as CLIP and LSeg) through feature mapping. However, this strategy introduces new problems. Firstly, there is geometric degradation. Because 3D geometry contains fine-grained high-frequency structural signals, while high-level textual concepts are typically abstract and smooth, forced alignment often leads to overly smooth geometric representations, losing key structural details and disrupting multi-view representation. Figure 1Firstly, there are limitations in scalability. Existing methods typically "lock" the 3D model to a specific VLM architecture, limiting the system's perceptual capacity and lacking flexibility. They cannot be integrated with more powerful next-generation models such as SigLIP and Qwen-VL 2.5 in a "plug-and-play" manner. Furthermore, since most alignment-based VLMs are primarily trained on 2D data, their features often only distinguish semantic categories (e.g., "chair") and cannot identify different instances within the same category. This is particularly problematic when handling 3D instance matching under large perspective changes and complex spatial question answering tasks.

[0005] Besides limitations at the algorithmic architecture level, the scarcity of high-quality training data is also a key factor restricting the development of this field. Existing datasets struggle to provide large-scale, high-quality, and 3D-consistent instance-level mask annotations, making it difficult for models to learn robust instance-level "geometric-semantic" correspondences.

[0006] In summary, achieving a unified "geometry-semantic" scene understanding requires addressing multiple core issues: Algorithmically, it is necessary to overcome the problems of geometry-semantic separation, error propagation, and high time consumption in decoupled methods, while avoiding geometric degradation, limited model scalability, and insufficient instance discrimination capabilities caused by tightly coupled methods; at the data level, high-quality, 3D consistent instance-level annotations are still very scarce, further limiting the model's learning ability.

[0007] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0008] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0009] The purpose of this disclosure is to provide an instance-understanding 3D reconstruction model based on instance geometry Transformer, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.

[0010] According to a first aspect of the present disclosure, an instance-understanding 3D reconstruction model based on instance geometry Transformer is provided, comprising: Example: A 3D reconstruction model for understanding geometric Transformers, characterized by including: The Transformer backbone network, dual downstream decoders, and cross-modal fusion module; among which... The Transformer backbone network includes 24 layers of alternating intra-view self-attention mechanisms and global cross-view attention mechanisms, which are used to fuse local information from a single view with global structure from multiple views and output a unified token representation. The dual downstream decoding heads include a geometry decoding head and an instance decoding head. The geometry decoding head is used to decode the geometric features of the scene from the token representation, and the instance decoding head is used to decode the instance features from the token representation. The geometry decoding head includes a camera parameter predictor, a depth predictor, and a point cloud predictor. The cross-modal fusion module embeds geometric features into instance features through cross-attention and sliding window cross-attention, and uses geometric details to guide the edge refinement of instance features to obtain instance feature maps.

[0011] Furthermore, the total loss function for the 3D reconstruction model in the example understanding is:

[0012] in, Let the camera parameter loss function be... For depth loss function, For point plot loss function, This is the loss function for multi-view contrast.

[0013] Furthermore, the processing procedure of the Transformer backbone network is as follows: Image The input is fed into the Transformer backbone network, where 24 layers of alternating intra-view self-attention and global cross-view attention mechanisms fuse single-view local information with multi-view global structure to process the image. Encoded as a unified token representation Where H is the height of the image, W is the width of the image, M is the number of tokens in each image, and D is the dimension of the token.

[0014] Furthermore, the processing procedure of the geometry decoder is as follows: The token is input into the camera parameter predictor, which uses the camera token to regress intrinsic and extrinsic parameters. The depth predictor and point cloud predictor reconstruct multi-level hierarchical geometric features from a unified token representation through layer-by-layer upsampling and multi-scale feature fusion networks to obtain the first geometric feature, the second geometric feature, and the third geometric feature.

[0015] Furthermore, the processing procedure for the instance decoding head is as follows: The token representation is formatted by LayerNorm and Reshape sequentially, and then input into Conv2d for preliminary feature extraction to obtain basic features. The basic features are simultaneously fed into four parallel branches: the first branch, the second branch, the third branch, and the fourth branch, with each branch processed independently. First branch: First, the feature is upsampled by 4 times through the first ConvTransposed, then it enters the Projects Block for feature mapping; then it is input to the first ConvTransposed and the Projects Block in sequence to further refine the feature. The second branch: First, it performs a 2x upsampling through the second ConvTransposed, and then enters the Projects Block for feature mapping; The third branch: First, perform normal 1x sampling through Identity, then enter the Projects Block for feature mapping; Fourth branch: First, perform 2x downsampling using Conv2d2, then proceed to the Projects Block for feature mapping; The outputs of the four branches are fed into PositionEmbeddingSine to add position encoding, resulting in multi-level instance features; the instance features include the first instance feature, the second instance feature, the third instance feature, and the fourth instance feature.

[0016] Furthermore, the processing procedure of the cross-modal fusion module is as follows: The first instance features, second instance features, third instance features, and fourth instance features are all input into Conv2d to adjust the feature dimensions to enhance local features; The fourth instance feature is input into the Feature Fusion Block for multi-scale feature fusion and resolution restoration through progressive upsampling to obtain the first fused feature; based on cross attention, the third geometric feature is used as the key vector and value vector, and the first fused feature is used as the query vector. The third geometric feature is embedded into the first fused feature to obtain the first embedded feature. The first embedded feature and the third instance feature are input into the Feature Fusion Block for processing to obtain the second fused feature; based on cross attention, the second geometric feature is used as the key vector and value vector, the second fused feature is used as the query vector, and the second geometric feature is embedded into the second fused feature to obtain the second embedded feature; The second embedded feature and the second instance feature are input into the Feature Fusion Block for processing to obtain the third fused feature; based on sliding window cross attention, the first geometric feature is used as the key vector and value vector, the third fused feature is used as the query vector, and the first geometric feature is embedded into the third fused feature to obtain the third embedded feature; The third embedded feature and the first instance feature are input into the Feature Fusion Block for processing to obtain the fourth fused feature; The global association of the fourth fusion feature is enhanced by using a sliding window self-attention method, then upsampled to the image resolution, and finally mapped through a convolutional layer to obtain the final 8-dimensional instance feature map.

[0017] According to a second aspect of the present disclosure, a method for using an instance-understanding 3D reconstruction model based on instance geometry Transformer is provided, comprising: Building multi-view Figure 1 The instance segmentation dataset consists of synthetic data, real-world video capture data, and real-world RGBD capture data with coarse 3D instance annotations. Differential optimization is performed on real-world video capture data and real-world RGBD capture data to obtain instance segmentation masks, which are then combined with synthetic data to construct a training dataset; The instance understanding 3D reconstruction model is trained using the training dataset to obtain a well-trained instance understanding 3D reconstruction model. The instance understanding 3D reconstruction model is applied to downstream spatial intelligence tasks. The instance mask output by the instance understanding 3D reconstruction model is used as an interaction bridge to combine with any visual language model or large multimodal model in a plug-and-play manner to complete multi-view instance matching, open vocabulary semantic segmentation and scene question answering tasks.

[0018] Furthermore, the steps of differentially optimizing real-world video capture data and real-world RGBD capture data to obtain instance segmentation masks, and combining these with synthetic data to construct a training dataset include: For real-world video capture data, the first frame is processed using the instance segmentation pipeline driven by SAM2 to generate an initial instance mask as the base mask information; The mask is propagated along the timeline using an instance segmentation pipeline, and its temporal consistency and integrity are ensured through keyframe iteration and bidirectional propagation to generate multiviews. Figure 1 Consistent instance mask; For real-world RGBD acquisition data, the real-world RGBD acquisition data with coarse 3D instance annotations is projected onto the RGB image to generate an initial 2D instance mask with coarse edges but consistent IDs; The RGB image is processed using the instance segmentation pipeline to obtain a fine mask with fine edges but no assigned ID; Align the fine-grained mask with the initial 2D instance mask, and assign the correct, multi-view to each fine-grained mask. Figure 1 Consistent IDs are used to generate sophisticated instance masks; According to multiple views Figure 1 Consistent instance masks, fine-grained instance masks, and synthetic data constitute the training dataset.

[0019] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: In the embodiments of this disclosure, through the instance understanding 3D reconstruction model based on instance geometry Transformer described above, on the one hand, a multi-view approach is designed. Figure 1 This application constructs a comprehensive instance segmentation pipeline and builds a large-scale training dataset covering multiple scenes and a large number of images. A large-scale Transformer model is constructed, and a cross-modal fusion module is designed to embed geometric features into instance representations, enabling spatial reconstruction and instance understanding to be trained end-to-end within a single model. An instance-aware scene understanding paradigm is proposed, utilizing instance masks as a bridge to achieve plug-and-play integration with various visual language models and large-scale multimodal models. Furthermore, this application can obtain camera parameters, depth maps, point maps, and 3D consistent instance feature maps in a single forward propagation with only image input, efficiently supporting spatial intelligence tasks such as multi-view instance matching, 2D / 3D semantic segmentation, and scene question answering. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0021] Figure 1 A framework diagram of an instance-understanding 3D reconstruction model based on instance geometry Transformer is shown in an exemplary embodiment of this disclosure; Figure 2 This diagram illustrates the structure of an example decoding head in an exemplary embodiment of this disclosure. Figure 3 This diagram illustrates the structure of the Projects Block in the instance decoding header of an exemplary embodiment of this disclosure. Figure 4 This illustration shows a cross-modal fusion module embedding geometric features into instance features in an exemplary embodiment of this disclosure; Figure 5 This diagram illustrates the structure of the cross-modal fusion module in an exemplary embodiment of this disclosure. Figure 6 This diagram illustrates the structure of the Feature Fusion Block in the cross-modal fusion module of an exemplary embodiment of this disclosure. Figure 7 A step diagram illustrating the method of using an instance-understanding 3D reconstruction model based on instance geometry Transformer in an exemplary embodiment of this disclosure; Figure 8 This diagram illustrates the workflow of an instance segmentation pipeline in an exemplary embodiment of this disclosure. Figure 9 A flowchart illustrating the application of space intelligence tasks in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0022] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0023] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0024] This example implementation provides an instance-understanding 3D reconstruction model based on instance geometry Transformer. (Reference) Figure 1 As shown, the instance-understanding 3D reconstruction model based on instance geometry Transformer can include: The Transformer backbone network, dual downstream decoders, and cross-modal fusion module; among which... The Transformer backbone network includes 24 layers of alternating intra-view self-attention mechanisms and global cross-view attention mechanisms, which are used to fuse local information from a single view with global structure from multiple views and output a unified token representation. The dual downstream decoding heads include a geometry decoding head and an instance decoding head. The geometry decoding head is used to decode the geometric features of the scene from the token representation, and the instance decoding head is used to decode the instance features from the token representation. The geometry decoding head includes a camera parameter predictor, a depth predictor, and a point cloud predictor. The cross-modal fusion module embeds geometric features into instance features through cross-attention and sliding window cross-attention, and uses geometric details to guide the edge refinement of instance features to obtain instance feature maps.

[0025] Based on the instance-understanding 3D reconstruction model using the aforementioned instance-based geometry Transformer, on the one hand, a multi-view approach was designed. Figure 1 This application constructs a comprehensive instance segmentation pipeline and builds a large-scale training dataset covering multiple scenes and a large number of images. A large-scale Transformer model is constructed, and a cross-modal fusion module is designed to embed geometric features into instance representations, enabling spatial reconstruction and instance understanding to be trained end-to-end within a single model. An instance-aware scene understanding paradigm is proposed, utilizing instance masks as a bridge to achieve plug-and-play integration with various visual language models and large-scale multimodal models. Furthermore, this application can obtain camera parameters, depth maps, point maps, and 3D consistent instance feature maps in a single forward propagation with only image input, efficiently supporting spatial intelligence tasks such as multi-view instance matching, 2D / 3D semantic segmentation, and scene question answering.

[0026] Below, we will refer to Figures 1 to 6 The various parts of the instance-understanding 3D reconstruction model based on instance geometry Transformer described above in this example implementation will be explained in more detail.

[0027] In one embodiment, to obtain end-to-end geometry and instance understanding capabilities, this application designs and constructs an instance-understanding 3D reconstruction model based on instance geometry Transformer. Establish a unified representation to achieve comprehensive 3D reconstruction and instance-level semantic understanding in a mutually reinforcing manner. Input N Zhang Image I i It can infer camera parameters in a single forward propagation. t i Depth map D i dot map P i and 3D consistent instance-level feature maps S i : (1) The model in this application is a large-scale Transformer architecture with 1.3 billion parameters. The overall framework consists of multiple closely cooperating components that work together to achieve a unified geometry-instance understanding. The overall framework and process are as follows: Figure 1 As shown, firstly, a large-scale unified Transformer based on VGGT is used to extract a unified token representation from multi-view inputs, laying the foundation for subsequent cross-modal fusion. Subsequently, this application designs a dual downstream decoding head containing a cross-modal fusion module, achieving high-quality spatial structure inference and instance-level representation through mutual enhancement of geometric and instance features. Finally, with the help of 3D consistency supervision, the model can learn to construct a 3D consistent instance feature field during training, thereby maintaining robust spatial consistency and instance awareness capabilities in multi-view scenes. This overall framework realizes end-to-end modeling capabilities from 2D images to 3D geometry reconstruction and instance understanding, laying a solid foundation for various spatial intelligence tasks.

[0028] 1. Large-scale unified Transformer. This application follows the design principles of VGGT to construct a large-scale unified Transformer for processing multi-view input images. Encoded as a unified and strong token representation Where H and W represent the image dimensions, M represents the number of tokens per image, and D is the dimension of the token. The model in this application first uses pre-trained DINOv2 to extract patch-level visual tokens from the images. To adapt to any number of viewpoint inputs and maintain equivariance with viewpoint arrangement, this application additionally concatenates a learnable camera token into the token sequence for each viewpoint. Subsequently, the model transforms these image tokens into a unified token representation Ti through a 24-layer attention module consisting of alternating intra-view self-attention and global cross-view attention mechanisms. This process effectively integrates information from a single viewpoint with global information, enabling the model to form a holistic and globally consistent understanding of the 3D scene.

[0029] 2. Dual Downstream Decoding Heads and Cross-Modal Fusion. Based on the unified token representation Ti, this application designs two downstream branches—a geometry decoding head and an instance decoding head—to predict the geometric structure and instance features of the scene, respectively. The geometry decoding head inherits the design of VGGT and consists of three sub-modules: a camera parameter predictor, a depth predictor, and a point cloud predictor. The camera predictor uses viewpoint-related camera token regression to obtain the camera's intrinsic and extrinsic parameters; while the depth and point cloud predictors are based on a DPT structure, using a layer-by-layer upsampling and multi-scale feature fusion network. Reconstructing multi-level hierarchical geometric features from a unified token representation Ti This process ultimately yields the corresponding depth map and point map. Similarly, the instance decoding head... Similarly, using a DPT-based dense prediction structure, the unified token representation Ti is decoded into instance features. : (2) More specifically, the processing procedure of the geometry decoder is as follows: The token is input into the camera parameter predictor, which uses the camera token to regress intrinsic and extrinsic parameters. The depth predictor and point cloud predictor reconstruct multi-level hierarchical geometric features from a unified token representation through layer-by-layer upsampling and multi-scale feature fusion networks to obtain the first geometric feature, the second geometric feature, and the third geometric feature.

[0030] like Figure 2 As shown, the processing procedure for the instance decoding head is as follows: The token representation is formatted by LayerNorm and Reshape sequentially, and then input into Conv2d for preliminary feature extraction to obtain basic features. The basic features are simultaneously fed into four parallel branches: the first branch, the second branch, the third branch, and the fourth branch, with each branch processed independently. First branch: First, the feature is upsampled by 4 times through the first ConvTransposed, then it enters the Projects Block for feature mapping; then it is input to the first ConvTransposed and the Projects Block in sequence to further refine the feature. The second branch: First, it performs a 2x upsampling through the second ConvTransposed, and then enters the Projects Block for feature mapping; The third branch: First, perform normal 1x sampling through Identity, then enter the Projects Block for feature mapping; Fourth branch: First, perform 2x downsampling using Conv2d2, then proceed to the Projects Block for feature mapping; The outputs of the four branches are fed into PositionEmbeddingSine to add position encoding, resulting in multi-level instance features; the instance features include the first instance feature, the second instance feature, the third instance feature, and the fourth instance feature.

[0031] like Figure 3 The diagram shown is a framework diagram of Projects Block.

[0032] To further enhance the model's ability to perceive fine spatial structures, this application proposes a cross-modal fusion module. This module efficiently embeds geometric features into instance features within a local spatial range through cross-attention and sliding window cross-attention, as illustrated in the diagram below. Figure 4 As shown. This enhances the sensitivity of instance features to object boundaries and spatial layout, while avoiding the secondary complexity of global attention. Its formulaic expression is: (3) After fusion, this application maps the refined instance features into a final 8-dimensional instance feature map. .

[0033] More specifically, such as Figure 5 As shown, the processing procedure of the cross-modal fusion module is as follows: The first instance features, second instance features, third instance features, and fourth instance features are all input into Conv2d to adjust the feature dimensions to enhance local features; The fourth instance feature is input into the Feature Fusion Block for multi-scale feature fusion and resolution restoration through progressive upsampling to obtain the first fused feature; based on cross attention, the third geometric feature is used as the key vector and value vector, and the first fused feature is used as the query vector. The third geometric feature is embedded into the first fused feature to obtain the first embedded feature. The first embedded feature and the third instance feature are input into the Feature Fusion Block for processing to obtain the second fused feature; based on cross attention, the second geometric feature is used as the key vector and value vector, the second fused feature is used as the query vector, and the second geometric feature is embedded into the second fused feature to obtain the second embedded feature; The second embedded feature and the second instance feature are input into the Feature Fusion Block for processing to obtain the third fused feature; based on sliding window cross attention, the first geometric feature is used as the key vector and value vector, the third fused feature is used as the query vector, and the first geometric feature is embedded into the third fused feature to obtain the third embedded feature; The third embedded feature and the first instance feature are input into the Feature Fusion Block for processing to obtain the fourth fused feature; The global association of the fourth fusion feature is enhanced by using a sliding window self-attention method, then upsampled to the image resolution, and finally mapped through a convolutional layer to obtain the final 8-dimensional instance feature map.

[0034] like Figure 6 The diagram shown is a framework diagram of the Feature Fusion Block.

[0035] 3. 3D Consistency Supervision. This is to ensure the model learns 3D consistent instance features across different viewpoints from only 2D input. This application introduces multi-view contrast loss. This loss function aims to bring features belonging to a single 3D instance closer together, while distancing features from different instances. Given a set of sampled pixels... Multi-view contrast loss The definition is as follows: (4) in, Indicates the L2 distance between normalized features. For pixels Corresponding instance ID. Coefficient and The weights of the zoom-in and zoom-out terms are controlled separately, and the hyperparameter M is used to adjust the discriminative power between different instances. This loss function constrains instance features based on the geometric structure of the 3D scene, which helps improve the model's generalization ability.

[0036] 4. Overall Loss Function. The model in this application serves as a unified end-to-end framework, trained holistically using a comprehensive multi-task loss function: (5) Among them, the geometric supervision term is the camera parameter loss function. Depth loss function Plot loss function This follows the training paradigm of VGGT. These terms are primarily used to supervise the output of the geometry decoder, ensuring that the model can accurately predict camera parameters, scene depth information, and point maps, thus guaranteeing the accuracy of geometry reconstruction. The multi-view contrast loss... The output of the instance decoding head is supervised to ensure that the generated instance features maintain 3D consistency across different viewpoints. The cross-modal fusion module in this application integrates geometric structure and instance understanding. Through this multi-task joint optimization approach, the model in this application achieves mutual promotion between geometric learning and instance learning, ensuring that the unified output representation possesses both high-precision geometric structure and robust instance consistency.

[0037] Furthermore, this example implementation also provides a method for using an instance-understanding 3D reconstruction model based on instance geometry Transformer. (Reference) Figure 7 As shown, the method of using this instance-understanding 3D reconstruction model based on instance geometry Transformer can include: Building multi-view Figure 1The instance segmentation dataset consists of synthetic data, real-world video capture data, and real-world RGBD capture data with coarse 3D instance annotations. Differential optimization is performed on real-world video capture data and real-world RGBD capture data to obtain instance segmentation masks, which are then combined with synthetic data to construct a training dataset; The instance understanding 3D reconstruction model is trained using the training dataset to obtain a well-trained instance understanding 3D reconstruction model. The instance understanding 3D reconstruction model is applied to downstream spatial intelligence tasks. The instance mask output by the instance understanding 3D reconstruction model is used as an interaction bridge to combine with any visual language model or large multimodal model in a plug-and-play manner to complete multi-view instance matching, open vocabulary semantic segmentation and scene question answering tasks.

[0038] In one embodiment, this application proposes an end-to-end unified solution that simultaneously achieves geometry reconstruction and instance understanding in a single forward propagation. First, to address the data bottleneck, this application proposes a novel multi-view approach. Figure 1 This application describes a high-quality instance segmentation pipeline. This pipeline is optimized for different data sources, generating high-quality, 3D-consistent instance-level masks. Based on this pipeline, a novel large-scale dataset, InsScene-15K, is constructed, covering 15K scenes and approximately 200 million images, including high-quality camera parameters, depth maps, and multi-view data. Figure 1 First, this application employs a unified instance segmentation mask, providing a solid data foundation for unified learning of geometry and instance understanding. Second, it designs a large-scale unified Transformer that integrates spatial reconstruction and instance-level understanding into a single model, achieving true end-to-end geometry-instance understanding joint modeling. Furthermore, it proposes an instance-aware scene understanding paradigm, using predicted instance masks as a bridge to seamlessly integrate with any visual language model (VLM) and large-scale multimodal models (LMM) in a "plug-and-play" manner, fundamentally overcoming the scalability bottleneck of tightly coupled methods. Finally, this unified representation significantly enhances the model's applicability to various spatial intelligence tasks, efficiently supporting downstream tasks such as multi-view instance matching, 2D / 3D semantic segmentation, and scene question answering.

[0039] I. This application constructs a novel large-scale dataset, InsScene-15K, aiming to overcome the problems of low instance-level annotation quality, lack of 3D consistency, and insufficient scale in existing datasets, laying a solid foundation for unified geometric reconstruction and instance understanding learning. This dataset covers 15K scenes and approximately 200 million images, including high-quality camera parameters, depth maps, and multi-view data. Figure 1A high-quality instance segmentation mask is constructed using a novel SAM2-driven instance segmentation pipeline. This pipeline integrates and optimizes data from three different sources: synthetic data, real-world video capture data, and real-world RGBD capture data (with coarse 3D instance annotations), ensuring the final instance mask has high quality and multi-view accuracy. Figure 1 To the point of being responsive.

[0040] 1. Synthetic Data. Synthetic data primarily originates from simulated environments, such as the Aria Synthetic Environments dataset and scenes generated using Infinigen in this application. In these simulated environments, RGB images, depth maps, camera parameters, and object-level segmentation masks can be generated synchronously. Because these masks are directly generated by the simulator, they possess "perfect accuracy" and can be used directly without any manual correction or post-processing. The advantage of synthetic data lies in its ability to provide extremely high-quality geometric-to-instance correspondences and accurate multi-view representations. Figure 1 Consistent instance masks. Furthermore, the high-quality annotations of the synthetic data provide robust supervision signals for the model, enabling it to maintain accurate multi-view performance while jointly learning geometric reconstruction and instance understanding. Figure 1 However, synthetic scenes are often monotonous and differ somewhat from the real world, with limited realism and diversity, making them difficult to directly generalize to complex real-world scenarios. To improve the model's generalization ability in the real world, specialized processing of real-world data is still needed to obtain high-quality, multi-viewable data. Figure 1 Consistent instance mask.

[0041] 2. Real-world video capture data. Real-world video capture data, such as the RE10K dataset, contains continuous RGB video from complex scenes. The data processing workflow is as follows: Figure 8 As shown in (a), a carefully designed SAM2 video instance mask prediction pipeline is employed. The process begins by generating an initial instance mask using SAM2 in the first frame of the video, providing foundational mask information for subsequent video frames. Subsequently, the SAM2 video object segmenter propagates these masks forward along the time axis to generate the initial instance mask for the next frame. Due to the potential for new objects or camera pose drift in the video, a keyframe iteration mechanism is introduced to ensure complete and consistent mask coverage: when the uncovered area exceeds a set threshold, the pipeline re-runs SAM2 on the new frame to detect the uncovered area and generate a new instance mask. Furthermore, to ensure high temporal consistency throughout the video sequence, the pipeline performs a bidirectional propagation, correcting potential omissions or drifts through forward and backward mask propagation. This process guarantees the continuity and temporal consistency of instance masks across video frames.

[0042] 3. Real-world RGBD acquisition data (with coarse 3D instance annotations). For real-world scenes from RGBD datasets such as ScanNet++, the original 3D annotations are usually quite coarse, and using them directly would affect mask accuracy. Therefore, this application also uses masks generated by SAM2 to optimize them. The processing pipeline is as follows: Figure 8 As shown in (b) above, the coarse 3D annotations provided by ScanNet++ are first projected onto each RGB image to generate initial 2D instance masks with consistent IDs. These masks ensure consistency of instances across multiple views, but the edges are relatively coarse and cannot meet the requirements for fine instance segmentation. To improve mask accuracy, the pipeline uses SAM2 to generate fine-edged instance masks without assigned IDs for the same RGB image. The core steps of matching and merging are then performed: the fine-edged masks generated by SAM2 are aligned with the projected coarse masks, and each fine-edged mask is assigned the correct ID from multiple views. Figure 1 The process iterates until the uncovered area still exceeds a threshold, resulting in a unique ID. The final mask inherits from the multi-view annotations of the 3D model. Figure 1 It is consistent with the quality of SAM2 and also possesses the high-quality edge details provided by SAM2.

[0043] Based on this instance segmentation pipeline, this application integrates and optimizes synthetic data, real-world video acquisition data, and RGBD acquisition data with coarse 3D annotations, ultimately constructing a large-scale, high-quality pipeline with scene diversity and multi-view capabilities. Figure 1 The InsScene-15K dataset is a consistent instance segmentation dataset. This dataset consists of 3000 synthetic scenes (1000 from Aria Synthetic Environments and 2000 generated by Infinigen), 10000 real-world video capture scenes (from RE10K), and 2000 real-world RGBD capture scenes (from ScanNet++). This large-scale and high-quality instance segmentation data provides a solid data foundation for subsequent end-to-end geometric reconstruction and instance understanding in models, enabling them to achieve stronger generalization in the real world.

[0044] II. Constructing a unified model based on instance geometry Transformer To achieve end-to-end geometry and instance understanding capabilities, this application designs and constructs a unified model based on instance geometry Transformer. .enter N Zhang Image I i It can infer camera parameters in a single forward propagation. t i Depth map Di dot map P i and 3D consistent instance-level feature maps S i :

[0045] The model has approximately 1.3 billion parameters, including core components such as a large unified Transformer backbone network, dual downstream decoders, and cross-modal fusion modules.

[0046] 1. Large-scale unified Transformer backbone network: Input N Zhang Image I i First, a pre-trained DINOv2 network is used to extract patch-level visual tokens from the images. To adapt to any number of viewpoint inputs and maintain equivariance with viewpoint arrangement, a learnable camera token is additionally concatenated into the token sequence for each viewpoint. Subsequently, the tokens are encoded using a 24-layer alternating "intra-viewpoint self-attention mechanism" and "global cross-viewpoint attention mechanism." Intra-viewpoint self-attention handles local feature associations within a single image. Global cross-viewpoint attention establishes connections between tokens from different viewpoints, capturing the global geometric structure of the 3D scene. The final output is a unified token representation T. i This enables the model to form a holistic and globally consistent understanding of the 3D scene.

[0047] 2. Dual downstream decoding heads: After obtaining the unified token representation T i Subsequently, the network splits into two parallel decoding branches, one for geometric prediction and the other for instance feature generation. The geometric decoding head contains three sub-predictors. The camera parameter predictor regresses intrinsic and extrinsic parameters using camera tokens; the depth predictor and point cloud predictor are based on a DPT structure, fusing multi-scale features through layer-by-layer upsampling. From the unified token representation T i Reconstructing multi-level hierarchical geometric features This yields a depth map and a point map. Example decoding head. Similarly, using a DPT-based dense prediction structure, the unified token representation T is used. i Decoding to initial instance features :

[0048] 3. Cross-modal fusion module: To address the issue of separation between geometric features and instance features, this application introduces a cross-modal fusion module in the decoding stage. This module integrates low-level geometric reconstruction and high-level instance understanding. Through cross-attention and sliding window cross-attention, it efficiently embeds geometric features into instance features, utilizing high-frequency geometric details to guide edge refinement of instance features. The specific processing of the cross-modal fusion is as follows: the geometric features output by the geometric decoding head are used as key and value vectors, and the instance features output by the instance decoding head are used as query vectors. A schematic diagram is shown below. Figure 4 As shown. This enhances the sensitivity of instance features to object boundaries and spatial layout, while avoiding the secondary complexity of global attention. The module can be formally expressed as:

[0049] Subsequently, this application maps the refined instance features into a final 8-dimensional instance feature map. .

[0050] 4. 3D Consistency Supervision: This is key to achieving 3D consistent instance understanding in this application. To enable the model to learn 3D consistent instance features across different viewpoints from only 2D input... This application introduces multi-view contrast loss. When two pixels belong to the same 3D instance (even if they are in different views), 3D consistency supervision promotes that their feature distances be as consistent as possible; when two pixels belong to different instances, the loss function promotes that their feature distances be as large as possible, exceeding a threshold. M Through this 3D consistency supervision, the model can construct a 3D consistent feature field, ensuring that the same instance has similar feature representations in space. Given a set of sampled pixels... Multi-view contrast loss The definition is as follows:

[0051] in, Indicates the L2 distance between normalized features. For pixels Corresponding instance ID. Coefficient and The weights of the zoom-in and zoom-out terms are controlled separately, and the hyperparameters are... M This loss function is used to adjust the discriminative power between different instances. It constrains instance features based on the geometric structure of the 3D scene, which helps improve the model's generalization ability.

[0052] 5. Overall Loss Function: During the training phase, this application adopts a multi-task joint optimization strategy. Among these, the geometric supervision term is the camera parameter loss function. Depth loss function Plot loss function This follows the training paradigm of VGGT. These terms are primarily used to supervise the output of the geometry decoder, ensuring that the model can accurately predict camera parameters, scene depth information, and point maps, thus guaranteeing the accuracy of geometry reconstruction. The multi-view contrast loss... This is used to supervise the output of the instance decoding head, ensuring that the generated instance features maintain 3D consistency across different viewpoints. The overall loss function is as follows:

[0053] III. Spatial Intelligent Applications After model building and training are completed, the trained model is used for forward inference to achieve 3D reconstruction and instance understanding of new scenes. This model is then further applied to various downstream spatial intelligence tasks to achieve scene understanding. The process for application in spatial intelligence tasks is as follows: Figure 9 As shown, unlike existing methods (such as some models for open vocabulary segmentation that are often tightly coupled to specific visual language models and inherently limited to a single task type), this application fundamentally decouples the overall framework from specific visual language models, making it independent of any particular model. Based on this, this application proposes a novel instance-aware scene understanding paradigm, enabling the model to naturally support and empower a wider range of more flexible downstream tasks. Compared to existing methods, the model in this application is the only unified framework capable of simultaneously performing tasks such as 2D-3D scene reconstruction, multi-view instance matching, and scene understanding, achieving state-of-the-art performance across all tasks.

[0054] Multi-view instance matching. This application utilizes 3D consistent instance features obtained through model forward inference to achieve dense instance segmentation and matching across viewpoints. Specifically, this application applies the density-based clustering algorithm HDBSCAN. This algorithm is used to segment 2D instance features from multi-view images. Gathered K The model is divided into several distinct clusters. Each cluster represents a unique 3D object instance present in the scene. Once feature clustering is complete, the model reprojects the ID assigned to each cluster back to its corresponding pixel location. This reprojection process generates a set of 2D instance masks with 3D consistency. This paradigm enables dense instance segmentation and matching across multi-view images by explicitly utilizing 3D prior knowledge (i.e., consistent 3D instance features output by the model). This contrasts sharply with existing methods, which are either limited to distinguishing category-level features (such as “chair” rather than “this chair”) or fail to maintain matching targets due to loss of instance target information when the camera undergoes significant motion.

[0055] Open-vocabulary semantic segmentation. This is another key application of the unified framework of this application for instance scene understanding. The generated 3D consistent instance mask acts as an effective cue and can be directly used in any off-the-shelf visual language model, such as CLIP, OpenSeg, etc. This application can perform robust open-vocabulary semantic segmentation tasks by assigning a semantic category to each region defined by the instance mask. Here, this application uses OpenSeg as an example for illustration. First, OpenSeg generates image-level semantic features. These features, by taking into account contextual information, achieve an initial alignment between visual and semantic features. Next, this application utilizes the 2D instance mask obtained through clustering. The semantic features generated by OpenSeg are averaged and masked, and features are aggregated within each 2D instance mask. This process yields a compact representation for each 2D instance mask. This crucial step not only integrates instance priors into the visual-linguistic space but also helps sharpen object boundaries and capture fine-grained local category information. This makes subsequent semantic segmentation more accurate and robust. Finally, this application ensures that all instance masks with the same instance ID are ultimately segmented to a consistent semantic result during the segmentation process. Building upon 2D segmentation, consistent 3D semantic segmentation can also be achieved based on inferred camera parameters and depth information. This application effectively solves the problem of inconsistent semantic segmentation in multi-view scenes, ensuring the consistency of semantic understanding results throughout the entire 3D scene.

[0056] Scene question answering. This is where the framework of this application demonstrates its support for complex scene understanding tasks. Unlike traditional methods that directly align 3D features with language embeddings, the method in this application offers greater flexibility by interacting with large multimodal models (LMMs, such as Qwen-VL 2.5) through decoupled instance clustering results to support object-centric scene question answering in 3D scenes. Specifically, given... N From one perspective, this application first utilizes the instance mask obtained through clustering. Highlighting corresponds to the same instance kThe image region is then used to perform scene question answering by querying the LMM with highlighted images and related questions. For example, tasks such as recognition and segmentation, such as "Please segment the animal sitting on the chair? Do the highlighted red areas in these multiple views meet the requirements?", can be posed with object counting tasks, such as "How many chairs are there in the scene represented by these images?", or spatial relationship reasoning, such as "Where is the black speaker marked by the blue dashed line located?". Compared with existing unified generative understanding models, this instance-based question answering approach perfectly combines the model's accurate spatial reconstruction and instance understanding capabilities with the powerful reasoning capabilities of the LMM, greatly improving robustness and accuracy in handling complex 3D scene understanding and reasoning tasks.

[0057] This application proposes a novel end-to-end framework that achieves a unified representation for spatial reconstruction and instance understanding in 3D scenes. The core of this application lies in coupling geometric features with instance-level features through joint training and fully leveraging the potential of a unified large-scale Transformer to achieve mutual promotion between low-level geometric reconstruction and high-level instance understanding. To support this task, this application designs a real-world data instance segmentation pipeline based on SAM2 and constructs a large-scale dataset, InsScene-15K, containing high-quality RGB images, camera parameters, depth maps, and multi-view data. Figure 1 This application provides a precise instance segmentation mask. Furthermore, the instance-based scene understanding paradigm proposed in this application enables seamless "plug-and-play" integration with various visual language models (VLMs) and large multimodal models (LMMs), significantly expanding its applicability and effectiveness in various spatial intelligence tasks such as open-vocabulary semantic segmentation and scene question answering. This application offers a promising research direction for the joint reconstruction and understanding of complex 3D scenes and is expected to drive further research in this field.

[0058] It should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," and "counterclockwise" in the above description indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this disclosure and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this disclosure.

[0059] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, "a plurality of" means two or more, unless otherwise explicitly specified.

[0060] In the embodiments of this disclosure, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances.

[0061] In embodiments of this disclosure, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0062] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0063] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. An instance-understanding 3D reconstruction model based on instance geometry Transformer, characterized in that, include: The Transformer backbone network, dual downstream decoders, and cross-modal fusion module; among which... The Transformer backbone network includes 24 layers of alternating intra-view self-attention mechanisms and global cross-view attention mechanisms, which are used to fuse local information from a single view with global structure from multiple views and output a unified token representation. The dual downstream decoding heads include a geometry decoding head and an instance decoding head. The geometry decoding head is used to decode the geometric features of the scene from the token representation, and the instance decoding head is used to decode the instance features from the token representation. The geometry decoding head includes a camera parameter predictor, a depth predictor, and a point cloud predictor. The cross-modal fusion module embeds geometric features into instance features through cross-attention and sliding window cross-attention, and uses geometric details to guide the edge refinement of instance features to obtain instance feature maps.

2. The instance-understanding 3D reconstruction model based on instance geometry Transformer according to claim 1, characterized in that, The total loss function of a 3D reconstruction model can be understood through examples. in, Let the camera parameter loss function be... For depth loss function, For point plot loss function, This is the loss function for multi-view contrast.

3. The instance-understanding 3D reconstruction model based on instance geometry Transformer according to claim 2, characterized in that, The processing procedure of the Transformer backbone network is as follows: Image The input is fed into the Transformer backbone network, where 24 layers of alternating intra-view self-attention and global cross-view attention mechanisms fuse single-view local information with multi-view global structure to process the image. Encoded as a unified token representation Where H is the height of the image, W is the width of the image, M is the number of tokens in each image, and D is the dimension of the token.

4. The instance-understanding 3D reconstruction model based on instance geometry Transformer according to claim 3, characterized in that, The processing procedure of the geometry decoder is as follows: The token is input into the camera parameter predictor, which uses the camera token to regress intrinsic and extrinsic parameters. The depth predictor and point cloud predictor reconstruct multi-level hierarchical geometric features from a unified token representation through layer-by-layer upsampling and multi-scale feature fusion networks to obtain the first geometric feature, the second geometric feature, and the third geometric feature.

5. The instance-understanding 3D reconstruction model based on instance geometry Transformer according to claim 4, characterized in that, The processing procedure for the instance decoding header is as follows: The token representation is formatted by LayerNorm and Reshape sequentially, and then input into Conv2d for preliminary feature extraction to obtain basic features. The basic features are simultaneously fed into four parallel branches: the first branch, the second branch, the third branch, and the fourth branch, with each branch processed independently. First branch: First, the feature is upsampled by 4 times through the first ConvTransposed, then it enters the Projects Block for feature mapping; then it is input to the first ConvTransposed and the Projects Block in sequence to further refine the feature. The second branch: First, it performs a 2x upsampling through the second ConvTransposed, and then enters the Projects Block for feature mapping; The third branch: First, perform normal 1x sampling through Identity, then enter the Projects Block for feature mapping; Fourth branch: First, perform 2x downsampling using Conv2d2, then proceed to the Projects Block for feature mapping; The outputs of the four branches are fed into PositionEmbeddingSine to add position encoding, resulting in multi-level instance features; the instance features include the first instance feature, the second instance feature, the third instance feature, and the fourth instance feature.

6. The instance-understanding 3D reconstruction model based on instance geometry Transformer according to claim 5, characterized in that, The processing procedure of the cross-modal fusion module is as follows: The first instance features, second instance features, third instance features, and fourth instance features are all input into Conv2d to adjust the feature dimensions to enhance local features; The fourth instance features are input into the Feature Fusion Block for multi-scale feature fusion and resolution restoration through progressive upsampling to obtain the first fused features; Based on cross attention, the third geometric feature is used as the key vector and value vector, the first fusion feature is used as the query vector, and the third geometric feature is embedded into the first fusion feature to obtain the first embedded feature; The first embedded feature and the third instance feature are input into the Feature Fusion Block for processing to obtain the second fused feature; Based on cross-attention, the second geometric feature is used as the key vector and value vector, the second fusion feature is used as the query vector, and the second geometric feature is embedded into the second fusion feature to obtain the second embedded feature; The second embedded feature and the second instance feature are input into the Feature Fusion Block for processing to obtain the third fused feature; Based on sliding window cross attention, the first geometric feature is used as the key vector and value vector, and the third fusion feature is used as the query vector. The first geometric feature is embedded into the third fusion feature to obtain the third embedded feature. The third embedded feature and the first instance feature are input into the Feature Fusion Block for processing to obtain the fourth fused feature; The global association of the fourth fusion feature is enhanced by using a sliding window self-attention method, then upsampled to the image resolution, and finally mapped through a convolutional layer to obtain the final 8-dimensional instance feature map.

7. A method for using instance-based geometry Transformer-based instance-understanding 3D reconstruction models, characterized in that, include: Construct a multi-view consistent instance segmentation dataset; the instance segmentation dataset includes synthetic data, real-world video acquisition data, and real-world RGBD acquisition data with coarse 3D instance annotations; Differential optimization is performed on real-world video capture data and real-world RGBD capture data to obtain instance segmentation masks, which are then combined with synthetic data to construct a training dataset; The instance understanding 3D reconstruction model is trained using the training dataset to obtain a well-trained instance understanding 3D reconstruction model. The instance understanding 3D reconstruction model is applied to downstream spatial intelligence tasks. The instance mask output by the instance understanding 3D reconstruction model is used as an interaction bridge to combine with any visual language model or large multimodal model in a plug-and-play manner to complete multi-view instance matching, open vocabulary semantic segmentation and scene question answering tasks.

8. The method for using the instance-based geometry Transformer-based instance-understanding 3D reconstruction model according to claim 7, characterized in that, The steps involved in constructing a training dataset by performing differential optimization on real-world video capture data and real-world RGBD capture data to obtain instance segmentation masks, and combining these with synthetic data, include: For real-world video capture data, the first frame is processed using the instance segmentation pipeline driven by SAM2 to generate an initial instance mask as the base mask information; The mask is propagated along the timeline using an instance segmentation pipeline, and the temporal consistency and integrity of the mask are ensured through keyframe iteration and bidirectional propagation to generate instance masks with multi-view consistency. For real-world RGBD acquisition data, the real-world RGBD acquisition data with coarse 3D instance annotations is projected onto the RGB image to generate an initial 2D instance mask with coarse edges but consistent IDs; The RGB image is processed using the instance segmentation pipeline to obtain a fine mask with fine edges but no assigned ID; Align the fine-grained mask with the initial 2D instance mask and assign the correct, multi-view consistent ID to each fine-grained mask to generate a fine-grained instance mask; The training dataset is constructed using multi-view consistent instance masks, fine-grained instance masks, and synthetic data.

Images