Cross-domain three-dimensional perception method combining geometric and semantic dual paths

Abstract

The invention relates to the technical field of computer vision and artificial intelligence, in particular to a cross-domain three-dimensional perception method combining geometric and semantic dual paths, which comprises the following steps of: 1, acquiring a multi-view image, and extracting a multi-scale two-dimensional feature map by using a two-dimensional feature encoder; 2, inputting the extracted two-dimensional feature map into a geometric generalization path, and carrying out the operation of self-supervised depth enhancement and explicit fusion to obtain geometric enhancement features; 3, inputting the extracted two-dimensional feature map into a semantic generalization path, and carrying out semantic query and domain confrontation fusion operation to obtain semantic enhancement features; and 4, finally fusing and decoding the geometric enhanced features and the semantic enhanced features to obtain a three-dimensional occupancy graph. The generalization performance of the method is improved in a breakthrough mode, and the method has the advantages that the practicability of the model is greatly enhanced, geometric perception is more accurate, three-dimensional reconstruction is more fidelity, semantic understanding is more consistent, and cross-domain recognition is more reliable.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a cross-domain 3D perception method that combines geometric and semantic approaches. Background Technology

[0002] With the rapid development of autonomous driving technology, accurate and robust perception of the vehicle's surrounding environment has become a core prerequisite for achieving high-level autonomous driving. Among numerous perception tasks, vision-based 3D occupancy perception (3DO ccupancy perception) technology has attracted much attention due to its ability to provide refined, voxel-based geometric and semantic information about the scene. This technology divides the space around the vehicle into a series of 3D voxels and predicts for each voxel whether it is occupied and what semantic category of object occupies it, such as vehicles, pedestrians, or vegetation. Compared to traditional 3D object detection that only outputs bounding boxes, occupancy grating can describe objects and backgrounds of arbitrary shapes, providing a richer and more complete environmental representation for downstream path planning and decision control modules.

[0003] Existing 3D occupancy perception models typically rely on a fully supervised learning paradigm, which requires training with data containing accurate 3D occupancy ground truth labels. Obtaining these 3D ground truth labels usually requires multi-frame scanning, point cloud registration, and manual annotation using high-precision LiDAR, a complex and costly process.

[0004] However, those skilled in the art have found that a more serious challenge lies in the generally poor scene transferability, or domain generalization ability, of these models. The deployment environments of autonomous vehicles vary greatly. For example, street scenes in different cities and countries, different weather and lighting conditions, and even minor differences in the number of cameras, field of view, and installation location on the same model of vehicle all constitute different data "domains." When a model trained in a specific scenario (source domain, such as a specific city or dataset) is directly deployed to another new, unseen scenario (target domain, such as different cities, countries, or vehicles with different sensor configurations), its perception performance experiences a significant "cliff-like" drop. This performance degradation limits the scalable deployment and practical application value of the models.

[0005] In summary, the performance bottlenecks and technical shortcomings of existing vision-based 3D occupancy perception technologies in scene migration applications mainly include: How can we effectively utilize the abundant, easily accessible, but unlabeled 3D ground truth data in the target domain to improve the model's adaptability to new scenarios and avoid significant performance degradation? Existing methods heavily rely on expensive source domain labels while underutilizing readily available unlabeled target domain data, causing the model to fail to adapt to the data distribution of new scenarios.

[0006] Geometric information mismatch is a significant issue. During scene transfer, differences in camera intrinsic and extrinsic parameters, field of view, etc., between the target domain and the source domain lead to biases in the model's understanding of the scene's geometry, particularly in depth estimation. This, in turn, affects the accuracy of feature projection from 2D images to 3D space, causing distortion or misalignment of the occupied grid. A key challenge is how to correct and enhance the model's geometric perception capability in the target domain using only unlabeled data.

[0007] The semantic feature shift problem. Different scenes exhibit significant differences in object appearance, background texture, and lighting conditions, causing a "domain shift" in the distribution of semantic features extracted from images by the model. This difference prevents the semantic recognition ability learned in the source domain from being directly generalized to the target domain, leading to object classification errors or missed detections. Finding effective methods to align the semantic feature spaces of the source and target domains is key to improving generalization.

[0008] To address this issue, while some domain-adaptive methods for 2D image or 3D object detection have emerged in the existing technology, how to effectively perform scene migration for 3D occupancy perception tasks with more complex structures and higher information dimensions, especially when the target domain only has unlabeled data, remains an unresolved technical challenge. Summary of the Invention

[0009] To address the aforementioned technical problems in existing technologies, this invention proposes a cross-domain 3D perception method that combines geometric and semantic approaches. This method seamlessly integrates the two generalization paths of geometry and semantics into an end-to-end framework, achieving end-to-end cross-domain adaptive optimization from input image to final 3D occupancy prediction. The specific technical solution is as follows: A cross-domain 3D perception method combining geometric and semantic approaches includes: Step 1: Acquire multi-view images and extract multi-scale two-dimensional feature maps using a two-dimensional feature encoder; Step 2: Input the extracted 2D feature map into the geometric generalization path, perform self-supervised depth enhancement and explicit fusion operations to obtain geometrically enhanced features; Step 3: Input the extracted 2D feature map into the semantic generalization path to perform semantic query and domain adversarial fusion operations to obtain semantically enhanced features; Step 4: Finally, fuse and decode the geometric enhancement features and semantic enhancement features to obtain the 3D occupancy map.

[0010] Furthermore, in step one, the two-dimensional feature encoder uses a ResNet-50 pre-trained on the ImageNet dataset as the backbone network, followed by a feature pyramid network (FPN).

[0011] Furthermore, in step two, a self-supervised deep network, DepthNet, is provided on the geometric generalization path. DepthNet is a lightweight convolutional network that receives two-dimensional feature maps. F i As input, the output is a pixel-by-pixel depth map. D pred .

[0012] Furthermore, during training, the geometric generalization path utilizes unlabeled temporal image data from the target domain. Through known inter-frame pose transformations of the camera, it projects images from adjacent time points onto the current time's viewpoint, calculating photometric reprojection loss and geometric consistency loss for training enhancement. The photometric reprojection loss... L rep The projected image is required to be pixel-wise consistent with the current image; the geometric consistency loss is... L adj The key points in the image should be in similar positions before and after projection.

[0013] Furthermore, the display fusion specifically involves: combining two-dimensional feature maps... F i and the depth map predicted by DepthNet D pred Input the LSS module to generate an initial 3D voxel feature map. F dvox This refers to geometric enhancement features.

[0014] Furthermore, in step three, a semantic query network, SemanticNet, is provided on the semantic generalization path, and the Semantic query network, SemanticNet, receives two-dimensional feature maps. F i As input, and with two outputs: a two-dimensional semantic segmentation graph S pred And semantically aware queries generated through the semantic attention layer at the end of the network. q sem The semantic attention layer will use the two-dimensional feature map... F iThe projection consists of three entities: query Q, key K, and value V; where the feature dimensions of Q and K are consistent with the number of semantic categories, thus projecting the features into the semantic space.

[0015] Furthermore, during domain adversarial fusion, the initial 3D voxel feature map... F dvox Compression in the height dimension yields a bird's-eye view of the BEV-level geometric query. q dbev And use spatial attention mechanism to handle geometric queries q dbev With semantic-aware query q sem The fusion process yields a BEV bird's-eye view feature that simultaneously incorporates geometric location and semantic enhancements. F bev This refers to semantic enhancement features.

[0016] Furthermore, during domain adversarial fusion training, the bird's-eye view features obtained from the source domain are... F bev _S Bird's-eye view features obtained from the target domain F bev _T Simultaneously, a domain classifier is fed in to distinguish the source of features. At the same time, through network training with gradient inversion layer and domain adversarial loss Lad, features that the domain classifier cannot distinguish are generated, so that the semantic feature distributions of the source domain and the target domain are successfully aligned.

[0017] Furthermore, step four specifically involves: applying semantic enhancement features. F bev Expanding in the height dimension and combining with geometrically enhanced initial 3D voxel features. F dvox By adding or concatenating elements, a unified three-dimensional voxel feature representation can be obtained. F vox Finally F vox The data is fed into a 3D convolutional decoding head to regress the final 3D occupancy prediction result. O pred .

[0018] The beneficial effects of this invention include: 1. A breakthrough improvement in generalization performance has been achieved, greatly enhancing the practicality of the model.

[0019] 2. More accurate geometric perception and more faithful 3D reconstruction.

[0020] Through innovative self-supervised deep learning, this method can self-calibrate using any video data, learning a universal "spatial scale sense" across different scenes. This ensures that the position and shape of objects in three-dimensional space are accurately reconstructed, effectively avoiding serious errors such as "object deformation and drift," and significantly improving the geometric accuracy of perception.

[0021] 3. Semantic understanding is more consistent, and cross-domain recognition is more reliable.

[0022] Through a unique semantic query adversarial fusion mechanism, this method forces the model to form a unified internal "cognitive encoding" for the same type of object with different appearances and locations, such as trucks from different countries. This solves the problem of the model failing to recognize new objects due to "limited knowledge," ensuring accurate understanding of the semantic content of the scene in any environment and achieving reliable cross-domain recognition. Attached Figure Description

[0023] Figure 1 This is a system architecture diagram of a cross-domain 3D perception method that combines geometric and semantic approaches in this embodiment; Figure 2 This is a diagram illustrating the adversarial learning principle of the semantic generalization path in this embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and technical effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0025] This invention discloses a cross-domain 3D perception method combining geometric and semantic approaches, implemented in an end-to-end neural network. The system model architecture used is as follows: Figure 1 As shown. The specific steps of this method are as follows: Step 1: Acquire multi-view images and input them into a 2D feature encoder for shared 2D feature extraction.

[0026] Input: A sequence of multi-view images captured simultaneously by multiple onboard cameras.

[0027] Processing: The image is input into a two-dimensional backbone network ResNet50 with shared weights and a feature pyramid network FPN.

[0028] Output: Generate multi-scale 2D feature maps for each viewpoint image. F i .

[0029] Step 2: Input the extracted 2D feature map into the geometric generalization path, perform self-supervised depth enhancement and explicit fusion to obtain geometrically enhanced features, specifically including: Step 2.1, Self-supervised depth enhancement: Objective: To train DepthNet, a deep network capable of accurately predicting metric depth across different domains. During training, unlabeled temporal image data from both the source and target domains are used. Through known inter-frame pose transformations of the camera, images from adjacent time steps are projected onto the viewpoint at the current time step, constructing a self-supervised loss function. L depth The training enhancement consists of two parts: Photometric reprojection loss L rep The goal is to make the projected image as pixel-wise identical as possible to the current image.

[0030] Geometric consistency loss L adj The requirement is to minimize the positional error of key points in the image before and after projection.

[0031] The training process can also use supervised depth loss from the source domain. L sp .

[0032] Results: Through this self-supervised training, DepthNet extracts cross-domain general geometric depth information from unlabeled videos and learns cross-domain invariant geometric rules to ensure the accuracy of the 3D structure, thus significantly enhancing its generalization ability.

[0033] Step 2.2, Explicit Deep Blending: Processing: Convert the two-dimensional feature map F i and the depth map predicted by the enhanced DepthNet D pred Input the LSS module (Lift-Splat-Shot).

[0034] Output: Generates a geometrically reliable initial 3D voxel feature map F based on accurate depth projection. dvox This refers to geometric enhancement features.

[0035] Step 3: Input the extracted 2D feature map into the semantic generalization path, perform semantic query and adversarial fusion to obtain semantically enhanced features, specifically including: Step 3.1, Semantic query generation: like Figure 2 As shown, a small semantic query network, SemanticNet, is used to extract data from two-dimensional feature maps. F i Decoding the two-dimensional semantic graph S pred .

[0036] A semantic attention layer is designed at the end of the SemanticNet network to process the two-dimensional feature map. F i The projection consists of three entities: query Q, key K, and value V. The feature dimensions of Q and K are consistent with the number of semantic categories, thus projecting the features into a "semantic space".

[0037] Calculate semantically perceptual queries that carry rich semantic information q sem The calculation formula is: q sem = SoftMax(Q*K^T)*V .

[0038] This step makes semantic information explicit and serves as a guide for the fusion process.

[0039] Step 3.2, Domain Adversarial Fusion: The initial three-dimensional voxel feature map obtained in step two F dvox Compression in the height dimension yields a geometric query at the bird's-eye view (BEV) level. q dbev .

[0040] Using spatial attention mechanisms, geometric queries q dbev With semantic-aware query q sem The data is fused to obtain an enhanced BEV bird's-eye view feature that simultaneously incorporates geometric location and semantic priors. F bev .

[0041] During training, the bird's-eye view features obtained from the source domain are used. F bev _S Bird's-eye view features obtained from the target domain F bev _T Simultaneously, a domain classifier is input. The task of this classifier is to determine which domain the input features come from.

[0042] Through a gradient reversal layer and domain adversarial loss Lad The feature extraction and fusion network in the training path generates features that can "fool" the domain classifier. When the classifier cannot distinguish the source of the features, it means that the feature distributions of the source and target domains have been successfully aligned.

[0043] Results: By using adversarial learning, the model is forced to generate semantic features that are independent of the source domain, so as to ensure the consistency of object recognition and achieve the alignment of the semantic distribution of the source and target domains at the level of fused features.

[0044] Step 4: Finally, fuse and decode the geometric enhancement features and semantic enhancement features to obtain the 3D occupancy map.

[0045] Processing: The semantically enhanced BEV features obtained in step three are processed... F bev Expand in the height dimension and combine with the geometrically enhanced initial 3D voxel features obtained in step two. F dvox Element-wise addition or concatenation is performed to obtain the final, highly robust, unified three-dimensional voxel feature representation. F vox .

[0046] Output: F vox The data is fed into a simple 3D convolutional decoder to regress the final 3D occupancy prediction result. O pred .

[0047] Based on the above method, this embodiment also discloses a cross-domain 3D perception system that combines geometric and semantic dual paths, deployed on a server equipped with six NVIDIA 4090 GPUs. Details are as follows: The system's software framework is developed based on the PyTorch deep learning framework. Two industry-recognized large-scale autonomous driving datasets are used as the source and target domains for validation: the nuScenes dataset and the Waymo Open Dataset. nuScenes contains six cameras, providing a 360° surround view; Waymo contains five cameras, offering wide forward and side views. The cross-domain 3D perception system performs data preprocessing on the datasets, including: Label Space Unification: Since the original semantic annotation categories of the two datasets are different, in order to achieve cross-domain learning, the semantic categories of the two are first mapped to a shared label space containing 10 core categories, such as: {obstacles, vehicles, pedestrians, bicycles, traffic cones, curbs, drivable surfaces, sidewalks, terrain, others}. Image size normalization: The size of all input multi-view camera images is uniformly adjusted to 256×704 pixels to meet network input requirements; 3D Space Definition: Define a unified 3D grid space covering the area surrounding the vehicle. For example, in the vehicle coordinate system, the X-axis (representing forward direction) and the Y-axis (representing lateral direction) range from [-51.2 meters to +51.2 meters], while the Z-axis ranges from [-5 meters to +3 meters]. Discretize this space into a 200×200×8 voxel grid, with each voxel having a physical size of approximately 0.51 meters × 0.51 meters × 1.0 meters.

[0048] The network model architecture of the cross-domain 3D perception system specifically includes: Two-dimensional feature encoder: This module uses a ResNet-50 pre-trained on the ImageNet dataset as the backbone, followed by a Feature Pyramid Network (FPN). This module is responsible for extracting multi-scale two-dimensional feature maps from the normalized input image. F i .

[0049] The geometry generalization module employs the self-supervised deep network DepthNet: a lightweight convolutional network that receives... F i As input, the output is a pixel-by-pixel depth distribution. D pred This network is the core of the geometric generalization path.

[0050] The semantic generalization module employs the SemanticNet semantic query network: a lightweight convolutional network that also receives... F i As input, and with two outputs: a two-dimensional semantic segmentation graph S pred Semantic-aware queries are generated through the semantic attention layer at the network endpoint. q sem .

[0051] Feature fusion and decoding module: In the geometric generalization path, explicit fusion is achieved by employing the LSS (Lift-Splat-Shoot) mechanism, utilizing... D pred and F i Generate initial 3D voxel features F dvox .

[0052] In the semantic generalization path, adversarial fusion: the initial 3D voxel features F dvox Compressed to BEV-level geometric queries q dbev and semantically aware queries q semFusion to generate enhanced BEV features F bev .

[0053] Final Fusion: F bev Expanded dimensions compared to the initial 3D voxel features F dvox Add them together to obtain the final voxel features. F vox .

[0054] 3D Decoding Head: A 3D convolutional network for... F vox The data is processed to output the final 200×200×8 3D occupancy prediction.

[0055] Domain classifier: A multilayer perceptron (MLP) receives... F bev Features are used as input for domain adversarial learning.

[0056] The training process of the system is divided into two main stages: Phase 1: Source Domain Pre-training.

[0057] Objective: To train a high-performance base model in the source domain.

[0058] Data: Only labeled data from the source domain are used: images, camera parameters, 3D occupancy ground truth, and depth ground truth.

[0059] Loss function: Total loss Lsource = λocc * Locc + λdepth * Ldepth_spv + λsem * Lsem .

[0060] Locc Occupation prediction loss, employing a combination of Focal Loss and Lovasz-softmax Loss, is used to supervise the final 3D occupancy prediction.

[0061] Ldepth_spv : Supervised depth loss, using L1 loss to calculate the difference between the predicted depth and the true depth.

[0062] Lsem Two-dimensional semantic loss, using cross-entropy loss to supervise the two-dimensional semantic segmentation map generated by SemanticNet. S pred .

[0063] Training parameters: The AdamW optimizer was used, the learning rate was set to 7e-5, and the training lasted for 24 epochs.

[0064] Phase Two: Cross-Domain Adaptive Training.

[0065] Objective: To adapt a pre-trained model from the source domain to the target domain, such as Waymo.

[0066] Data: In each training batch, labeled data from the source domain and unlabeled temporal data from the target domain are loaded simultaneously, and the data consists of continuous video frames.

[0067] Loss function: Total loss Ltotal It is a composite loss, with different components calculated based on the data source: For source domain data: Calculate the same supervised loss as in Phase 1. Lsource .

[0068] For target domain data: Calculate the following adaptive loss: Self-monitored depth loss Ldepth_ss This approach utilizes three consecutive frames of images from the target domain. Through camera pose transformation, adjacent frames are projected onto the current frame, and photometric reprojection loss and geometric consistency loss are calculated. This loss is used to optimize DepthNet on the unlabeled target domain.

[0069] Domain confrontation loss Lad : Transfer source domain data F bev _S and target domain data F bev _T Simultaneously, the data is fed into the domain classifier. The goal of the domain classifier is to correctly classify the source domain, while the goal of the feature extraction and fusion network is to generate features that can "fool" the domain classifier through a gradient inversion layer (GRL). The loss is a binary cross-entropy loss.

[0070] Overall optimization objective: Minimize the optimizer. Ltotal = Lsource + λss * Ldepth_ss + λadv * Ladv .

[0071] Training parameters: The AdamW optimizer was used, and the learning rate was adjusted to 3.5e-5. Weight coefficients were set empirically, for example... λss =0.1, λadv =0.1. Continue training until the model's performance on the target domain validation set converges.

[0072] This method addresses the core challenge of poor cross-domain generalization ability of models. When switching from an environment like nuScenes to a new environment like Waymo, traditional methods experience a performance drop of over 95%, almost becoming ineffective. This invention restores and improves performance by over 400%, ensuring the model remains reliable in real-world, diverse scenarios and possessing genuine practical application value. Experimental results are shown in the table below: .

[0073] In summary, this invention utilizes a unified framework to run two enhancement paths in parallel: one is a geometric generalization path that leverages self-supervised learning to improve depth estimation, and the other is a semantic generalization path that leverages semantic queries and adversarial learning to improve feature alignment. The outputs of these two paths are ultimately and effectively combined to generate a highly robust unified 3D voxel feature that possesses both accurate geometric structure and consistent semantic representation, thereby fundamentally solving the problem of cross-domain performance degradation.

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the implementation process of the present invention has been described in detail above, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-domain 3D perception method combining geometric and semantic approaches, characterized in that, include: Step 1: Acquire multi-view images and extract multi-scale two-dimensional feature maps using a two-dimensional feature encoder; Step 2: Input the extracted 2D feature map into the geometric generalization path, perform self-supervised depth enhancement and explicit fusion operations to obtain geometrically enhanced features; Step 3: Input the extracted 2D feature map into the semantic generalization path to perform semantic query and domain adversarial fusion operations to obtain semantically enhanced features; Step 4: Finally, fuse and decode the geometric enhancement features and semantic enhancement features to obtain the 3D occupancy map.

2. The method as described in claim 1, characterized in that, In step one, the two-dimensional feature encoder uses a ResNet-50 pre-trained on the ImageNet dataset as the backbone network, followed by a feature pyramid network (FPN).

3. The method as described in claim 1, characterized in that, In step two, a self-supervised deep network, DepthNet, is provided on the geometric generalization path. DepthNet is a lightweight convolutional network that receives two-dimensional feature maps. F i As input, the output is a pixel-by-pixel depth map. D pred .

4. The method as described in claim 3, characterized in that, The geometric generalization path utilizes unlabeled temporal image data in the target domain during training. Through known inter-frame pose transformations of the camera, it projects images from adjacent time points onto the current time's viewpoint, calculating photometric reprojection loss and geometric consistency loss for training enhancement. The photometric reprojection loss... L rep The projected image is required to be pixel-wise consistent with the current image; the geometric consistency loss is... L adj The key points in the image should be in similar positions before and after projection.

5. The method as described in claim 3, characterized in that, The display fusion specifically involves: combining two-dimensional feature maps... F i and the depth map predicted by DepthNet D pred Input the LSS module to generate an initial 3D voxel feature map. F dvox This refers to geometric enhancement features.

6. The method as described in claim 5, characterized in that, In step three, a semantic query network, SemanticNet, is provided on the semantic generalization path, and the Semantic query network, SemanticNet, receives two-dimensional feature maps. F i As input, and with two outputs: a two-dimensional semantic segmentation graph S pred And semantically aware queries generated through the semantic attention layer at the end of the network. q sem The semantic attention layer will use the two-dimensional feature map... F i The projection consists of three entities: query Q, key K, and value V; where the feature dimensions of Q and K are consistent with the number of semantic categories, thus projecting the features into the semantic space.

7. The method as described in claim 6, characterized in that, During domain adversarial fusion, the initial 3D voxel feature map F dvox Compression in the height dimension yields a bird's-eye view of the BEV-level geometric query. q dbev And use spatial attention mechanism to handle geometric queries q dbev With semantic-aware query q sem The fusion process yields a BEV bird's-eye view feature that simultaneously incorporates geometric location and semantic enhancements. F bev This refers to semantic enhancement features.

8. The method as described in claim 7, characterized in that, During domain adversarial fusion training, the bird's-eye view features obtained from the source domain are... F bev _S Bird's-eye view features obtained from the target domain F bev _T Simultaneously, a domain classifier is fed in to distinguish the source of features, and gradient reversal layer and domain adversarial loss are applied. Lad The network training process generates features that the source domain classifier cannot distinguish, thus successfully aligning the semantic feature distributions of the source and target domains.

9. The method as described in claim 7, characterized in that, Step four specifically involves: adding semantic enhancement features. F bev Expanding in the height dimension and combining with geometrically enhanced initial 3D voxel features. F dvox By adding or concatenating elements, a unified three-dimensional voxel feature representation can be obtained. F vox Finally F vox The data is fed into a 3D convolutional decoding head to regress the final 3D occupancy prediction result. O pred .

Images