Automatic driving perception method based on domain-specific model and open set model integration

The integration of domain-specific and open-set models in autonomous driving perception systems enhances performance on both whitelist and non-whitelist categories, addressing training costs and generalization challenges for 3D object detection and segmentation.

CN119068456APending Publication Date: 2024-12-03PEKING UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411089565.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-09
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing autonomous driving perception systems face challenges in integrating domain-specific models and open-set models for 3D object detection and segmentation due to high training costs, slow convergence, and limited generalization capabilities, especially in scenarios with non-whitelisted objects.

Method used

An autonomous driving perception method that integrates domain-specific models and open-set models, utilizing multi-camera image and radar point cloud inputs to achieve 2D and 3D multi-task predictions, and fuses these results through an integration module to enhance performance on both whitelist and non-whitelist categories without additional training costs.

Benefits of technology

The method achieves high-performance predictions for both domain-specific and open-set scenarios, combining the strengths of both models to improve generalization and reduce computational overhead.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068456A_ABST
    Figure CN119068456A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic driving perception method based on integration of a domain-specific model and an open set model, and belongs to the technical field of computer vision. According to the method, for multi-camera image input or multi-mode input of multi-camera images and radar point clouds, domain-specific prediction results of multiple automatic driving perception tasks are given through a domain-specific model branch; open set prediction results of the multiple automatic driving perception tasks are given through open set model branches; and finally, fusing the domain proprietary prediction result and the open set prediction result through an integration module to obtain a prediction result with domain proprietary and open set advantages. According to the method, the existing basic models are combined, and the 2D-to-3D object conversion module is designed, so that the 3D target detection open set model for the non-white list object is obtained, and the problem that a 3D task large model cannot be obtained through direct training in a data driving mode is solved; and the neural network modules are directly combined, so that extra training cost is not introduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and relates to the use of computer vision and deep learning technology to establish an autonomous driving perception neural network model and algorithm, and in particular to an autonomous driving perception method based on the integration of domain-specific models and open open set models. Background Art

[0002] In recent years, the research and application of machine learning have continued to advance, with deep learning methods achieving significant breakthroughs in many application areas, such as generative natural language dialogue, computer vision, and speech processing. As more and more deep learning algorithms are deployed in real-world applications, model generalization capabilities are gaining increasing attention. In the field of autonomous driving, model generalization capabilities specifically include open-scenario capabilities and open-set semantic capabilities: open-scenario capabilities refer to the model's ability to reason in new scenarios that differ from the training data; open-set semantic capabilities refer to the ability to identify and predict semantic categories that were not labeled during training (hereinafter, semantic categories with labels in the training set are referred to as "whitelist categories," and semantic categories without labels in the training set are referred to as "non-whitelist categories") during reasoning.

[0003] In areas such as multimodal natural language dialogue, 2D image segmentation, and 2D depth estimation, a series of open-set models have emerged, driven by large-scale data. These models possess certain open-set semantic capabilities. However, in autonomous driving perception tasks, especially those in 3D space, manual data annotation is extremely expensive, and it is difficult to obtain sufficient labeled autonomous driving data from the internet. This makes it difficult to use data-driven methods to train large models for 3D tasks. How to leverage existing open-set models and apply their capabilities to 3D object detection and 3D semantic segmentation tasks is a question worthy of attention.

[0004] At the same time, in tasks such as 2D object detection and 2D instance segmentation, the existing open set model can provide a certain degree of model generalization ability. For objects not in the whitelist category, the open set model can give a certain prediction; while the traditional domain-specific model uses a fixed whitelist category classification method, so it is difficult to predict objects not in the whitelist category. However, for objects in the whitelist, the current domain-specific model has stronger prediction performance than the open set model. As shown in the attached figure Figure 3As shown in the figure, for the half-occluded vehicle in the first row and the two overlapping people in the second row, the domain-specific model can provide relatively complete and accurate predictions, while the open set model's predictions are less accurate. Furthermore, for the dog on the road in the first row, the baby stroller and lamppost in the second row, and the sign in the third row, the open set model can provide both positional and semantic predictions, while the domain-specific model cannot. Both types of models have their strengths, and how to integrate their capabilities so that an autonomous driving perception system maintains high performance for whitelisted objects within its domain while also generalizing to non-whitelisted objects is a worthy challenge.

[0005] Finally, deep learning methods require iterative training on datasets using methods like gradient descent. Training for various autonomous driving perception tasks is costly and converges slowly. Data-driven open-set models, on the other hand, require significant computational resources to train. Consequently, much research in recent years has focused on zero-shot learning or fine-tuning these open-set models. Therefore, controlling training costs is also a crucial consideration when addressing these issues. Summary of the Invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art and solve the above-mentioned problems, the present invention proposes an autonomous driving perception method based on the integration of domain-specific models and open open set models. For multi-camera image input (or multi-modal input of multi-camera images and radar point clouds), domain-specific prediction results of multiple autonomous driving perception tasks (including 2D classification, 2D instance segmentation, 2D target detection, 3D classification, 3D top view semantic segmentation, 3D target detection and other tasks) are given through domain-specific model branches; open open set prediction results of multiple autonomous driving perception tasks are given through open open set model branches; finally, the domain-specific prediction results and the open open set prediction results are fused through the integration module to obtain a prediction result that has both domain-specific and open open set advantages.

[0007] The technical solutions of the present invention are as follows:

[0008] An autonomous driving perception method based on the integration of domain-specific models and open set models, wherein the sensor input of the method is multi-camera image input or multi-modal input of multi-camera images and radar point clouds, comprises the following steps:

[0009] 1) Based on sensor input, a domain-specific model is trained to simultaneously reason about six tasks, and 2D and 3D multi-task prediction results are obtained using this domain-specific model.

[0010] 2) According to the sensor input, an open set model is obtained by combining a series of basic models, and a 2D to 3D object conversion module is designed to obtain 2D and 3D multi-task prediction results; the 2D target detection bounding box, semantic features and confidence are obtained through the 2D open set detection model, and the semantic features are decoded through the natural language text decoder to obtain their corresponding semantic predictions, thereby obtaining the open set 2D target detection prediction results; the 2D target detection box is used as a prompt and input with multiple camera images through the 2D instance segmentation basic model to obtain the 2D instance segmentation prediction result; the pixels segmented from the 2D instance are combined with their point cloud depth or the depth estimation basic model to obtain their three-dimensional coordinates, thereby obtaining a pseudo point cloud, and the pseudo point cloud is aggregated to obtain a 3D bounding box. The 3D bounding box corresponds one-to-one to the semantic prediction and confidence of the 2D detection. The bounding box, confidence and semantic prediction together constitute the 3D target detection prediction result; for 2D or 3D classification tasks, prompt words are designed and the visual-language large model is used directly to obtain the results;

[0011] 3) Through the integration module, the multi-task prediction results of the domain-specific model and the multi-task prediction results of the open set model are integrated to obtain the final result; for 2D object detection and 3D object detection, after the confidence distribution is aligned, the integration is carried out using dual threshold non-maximum suppression based on semantic feature distance and intersection-over-union ratio. For 2D instance segmentation, the integration is guided by the 2D object detection prediction results. For classification tasks, the category selection is determined according to the confidence level.

[0012] Furthermore, the step 1) specifically includes the following steps:

[0013] 1-1) Domain-specific model training

[0014] During training, different multimodal data augmentation strategies are used for different inputs;

[0015] During training, select different optimizers and learning rate schedulers based on the actual training situation of the model;

[0016] During training, the multi-task loss function is designed as follows: classification and segmentation tasks use a classification loss function; detection tasks use a regression loss function for bounding box prediction, and a classification loss function for category prediction; based on the increase or decrease in accuracy of multiple tasks compared to a single task, the multi-task loss function weights are adjusted manually or automatically.

[0017] 1-2) Perform task prediction inference on the trained domain-specific model, including the following steps:

[0018] 1-2-1) Sensor input feature extraction: Select the required backbone network and neck network as the feature extraction network based on the type of input sensor and the required task. For multi-camera image input, the backbone network is used to extract the multi-scale features of the image, and the neck network is used to fuse the multi-scale features of the image; thus, a 2D image feature sequence is obtained. When there is also multimodal input of a radar point cloud, the radar point cloud is first voxelized to obtain feature voxels. Then, the required backbone network and neck network are selected based on the needs of the task to extract the 3D features of the radar point cloud, which are then fused with the 3D features of the image in step 1-2-3);

[0019] 1-2-2) 2D-3D feature conversion: For the image backbone network, select a 2D to 3D view conversion module that is compatible with it. Use a method based on depth estimation and depth distribution projection to project 2D features into 3D features, or perform position encoding on the BEV grid and directly extract features from multi-frame multi-view images through the attention mechanism to achieve 2D to 3D feature conversion.

[0020] 1-2-3) Temporal fusion of 2D and 3D features, using any existing temporal fusion method. For 2D tasks, the input of the 2D feature temporal fusion network is a 2D image feature sequence, and the output is the fused 2D features. For 3D tasks, the input of the 3D feature temporal fusion network is a 3D feature map sequence after 2D-3D feature conversion. If there is a radar point cloud input, it also includes the 3D features of the radar point cloud, and the output is the fused 3D features.

[0021] 1-2-4) For each task, select the feature map generated for that task, connect it to the decoding network of the corresponding task, and predict the results of each task; the decoding network of the corresponding task includes: 2D classification task decoder, 3D classification task decoder, 2D object detection task decoder, 3D object detection task decoder, 2D instance segmentation task decoder, 3D top view semantic segmentation decoder; obtain the results of each domain-specific task, including: 2D classification prediction result R spec2Dcls =(c spec2D ,s spec2D ) and 3D classification prediction results R spec3Dcls =(C spec3D ,s spec3D ), where c spec2D and c spec3D is the category prediction, s spec2d and s spec3D is the confidence level; 2D target detection prediction results in is the 2D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, Nspec2Ddet is the predicted number of objects; 3D target detection prediction results in is the 3D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N spec3Ddet is the predicted number of objects; 2D instance segmentation prediction results in Refers to the mask of the i-th instance, c i is the category corresponding to the i-th instance, N spec2Dseg is the number of predicted instances, and the 3D top view semantic segmentation prediction result R spec3Dseg .

[0022] Furthermore, the step 2) specifically includes the following steps:

[0023] 2-1) Open set 2D object detection task prediction, through an open set detection model composed of image encoder, 2D open set detector and text decoder, obtain the open set 2D object detection prediction result in is the 2D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N open2Ddet is the predicted number of objects;

[0024] 2-2) Open set 2D instance segmentation task prediction, using the instance segmentation basic model, first take the multi-camera image input as its input, and use the bounding box of the open set 2D object detection prediction result obtained in step 2-1) As a hint, get the mask corresponding to each instance Combined with the open set 2D object detection categories, we get the open set 2D instance segmentation prediction results The number of instances N open2Dseg =N open2Ddet ;

[0025] 2-3) Open-set 3D object detection task prediction: Using the 2D-3D object conversion module, 3D object detection prediction is performed based on the 2D object detection prediction results and the 2D instance segmentation prediction results. This includes the following steps:

[0026] 2-3-1) Use the depth estimation basic model to perform depth estimation on the image and obtain the depth estimation result matrix D; for each instance mask of the 2D instance segmentation prediction result For each pixel (u, v), the corresponding depth d = D can be found in the depth estimation result. u,v ; When a radar point cloud input is provided, the point cloud is projected onto the image plane and the depth d is obtained by interpolating the k nearest neighbor point clouds near (u,v);

[0027] 2-3-2) For instance mask Each pixel and its depth (u, v, d) in the image is transformed into a coordinate system using the camera's internal and external parameter matrices to obtain a pseudo point cloud (x, y, z) in the corresponding space.

[0028] 2-3-3) For all pseudo point clouds corresponding to a mask, find the minimum volume cuboid that contains all point clouds, that is, convert it into a three-dimensional minimum directed bounding box problem; use a heuristic search algorithm to approximate the solution to solve the corresponding 3D bounding box parameters Or adopt a learning-based method, use a neural network model, take all pseudo point clouds corresponding to a mask as input, use PointNet to extract features, and predict the parameters of the corresponding 3D bounding box through MLP Finally, the open set 3D object detection prediction results are obtained:

[0029] in in is the 3D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N open3Ddet is the predicted number of objects;

[0030] 2-4) Open set 2D and 3D classification task prediction, using the visual-language large model, directly input the required classification as the prompt text, and obtain the open set 2D classification prediction result R spec2Dcls =(c open2D ,S open2D ) and open set 3D classification prediction results R open3Dcls =(C spec3D ,s spec3D ), where c open2D and c open3D is the category prediction, s open2D and s open3D is the confidence level.

[0031] Furthermore, the step 3) specifically includes the following steps:

[0032] 3-1) Classification task integration, category selection is determined based on confidence, and for domain-specific 2D classification prediction results R spec2Dcls =(c spec2D,s spec2D ) and open set 2D classification prediction results R open2Dcls =(c open2D ,s open2D ), take the prediction result with high confidence as the final 2D classification prediction result:

[0033]

[0034] And update the confidence level. When the two prediction results are consistent, the confidence level is increased, otherwise the confidence level is decreased:

[0035]

[0036] Finally, the integrated 2D classification prediction result R is obtained 2Dcls =(c 2D ,s 2D ); The integration process of 3D classification is exactly the same as that of 2D classification;

[0037] 3-2) Detection task integration, for domain-specific 2D target detection prediction results R spec2Ddet And the open set 2D target detection prediction results R open2Ddet After the confidence distribution is aligned, the integration is performed using double threshold non-maximum suppression based on semantic feature distance and intersection-over-union ratio. Specifically, the confidence of the prediction results of the domain-specific 2D object detection on the specified dataset is first calculated. Distribution, get the mean μ and standard deviation σ; then open the confidence of the open set 2D target detection prediction result Redistribution is in accordance with the Gaussian distribution N9μ,σ) confidence distribution Finally, the non-maximum suppression NMS processing is performed on the set of domain-specific detection prediction results and the open set detection prediction results after confidence redistribution. The combined set is:

[0038]

[0039] Specifically, for any two detection prediction results in the detection prediction result union set, set as the i-th and j-th, when the intersection of the two result bounding boxes is IoU (b i ,b j ) exceeds the specified threshold T, and the results with lower confidence s are removed from the set; finally, the integrated 2D target detection prediction result R is obtained 2Ddet ; The integration process of 3D object detection is exactly the same as that of 2D object detection;

[0040] 3-3) Segmentation task integration, for 2D instance segmentation tasks, the integration is guided by the 2D object detection prediction results. Each 2D segmentation instance result Both with a 2D target detection prediction result One-to-one correspondence, therefore, record the target detection prediction results removed by NMS in step 3-2) and remove the corresponding segmentation instances from R spec2Dseg ∪R open2Dseg By removing it, we can get the integrated 2D instance segmentation prediction result R 2Dseg ; For the 3D top view semantic segmentation task, the integrated prediction result R 3Dseg =R spec3Dseg .

[0041] The technical effects of the present invention are as follows:

[0042] (1) The method of the present invention combines existing basic models such as 2D open set detection, 2D instance segmentation, depth estimation, and vision-language large models, and designs a 2D to 3D object conversion module to obtain a 3D target detection open set model for non-whitelisted objects, thereby solving the problem that it is impossible to directly train a 3D task large model through data-driven methods.

[0043] (2) The method of the present invention integrates the domain-specific model and the open set model, which enables the entire system to have the advantages of both the domain-specific model and the open set model, and has good prediction capabilities in both whitelist categories and non-whitelist categories.

[0044] (3) The method of the present invention directly combines the neural network modules without consuming additional computing resources for integrated training and without introducing additional training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a basic flow chart of the method proposed by the present invention;

[0046] Figure 2 It is a specific diagram of the method proposed by the present invention;

[0047] Figure 3 This is an example display of the integration results of the domain-specific model and the open set model proposed in this invention. DETAILED DESCRIPTION

[0048] The present invention will be further clearly and completely described below through specific embodiments in conjunction with the accompanying drawings.

[0049] The present invention provides an autonomous driving perception method based on the integration of domain-specific models and open open set models. It solves multiple autonomous driving perception tasks (including 2D classification, 2D instance segmentation, 2D object detection, 3D classification, 3D top-view semantic segmentation, 3D object detection, etc.) for multi-camera image input (or multi-modal input of multi-camera images and radar point clouds).

[0050] like Figure 1 The basic flow chart of the method of the present invention is shown, which provides domain-specific prediction results of multiple autonomous driving perception tasks through domain-specific model branches; provides open open set prediction results of multiple autonomous driving perception tasks through open open set model branches; finally, the domain-specific prediction results and the open open set prediction results are integrated through the integration module to obtain prediction results that have the advantages of both domain-specific and open open set.

[0051] The method proposed by the present invention is specifically shown in FIG. Figure 2 , for multi-frame inputs of one or more types of vehicle-mounted sensors such as multi-view cameras (Multi-view Camera), LiDAR, millimeter-wave radar (Radar), etc., obtain multi-task prediction results in whitelist semantic categories through existing domain-specific model branches; obtain open prediction results containing whitelists and non-whitelists through existing open set model branches, and convert them into 2D and 3D multi-task formats; finally, through the integration module, obtain corresponding integrated prediction results for each task. For input, camera images are required, and radar point clouds are optional. For output, you can choose one or more tasks such as 2D classification, 2D instance segmentation, 2D target detection, 3D classification, 3D top view semantic segmentation, 3D target detection, etc. Specifically, it includes the following steps:

[0052] S1) For sensor input, a domain-specific model that can reason about six tasks at the same time is obtained through training, and 2D and 3D multi-task prediction results are obtained through the domain-specific model. Domain-specific models refer to common autonomous driving perception models. These models are trained on specific data sets and have good performance in the domain, but cannot handle non-whitelisted semantic categories. The present invention is applicable to improving the generalization performance of any domain-specific model. The following describes a specific implementation of a type of domain-specific model and describes in detail how to obtain a domain-specific model that can reason about six tasks at the same time through training. The domain-specific model selects the backbone network and neck network for Radar point cloud input, LiDAR point cloud input and image input according to the type of input sensor and the required task. For the image backbone network, a 2D to 3D perspective conversion module adapted thereto is selected. This module can adopt a depth prediction-projection-based approach or an attention-based approach. For 3D tasks, various sensor features are extracted using the backbone network for multi-scale feature extraction and fused using the neck network. These features are then transformed into a unified BEV spatial coordinate system via the viewpoint conversion module. This is followed by independent encoding and decoding with each task's decoder to produce the proprietary model's 3D task prediction results. For 2D tasks, the 2D feature maps after the image backbone network and neck network are directly extracted, followed by independent encoding and decoding with each task's decoder to produce the proprietary model's 2D task prediction results.

[0053] S1-1) Domain-specific model training

[0054] During training, different multimodal data augmentation strategies are used for different inputs;

[0055] During training, select different optimizers and learning rate schedulers based on the actual training situation of the model;

[0056] During training, the multi-task loss function adopts the following design: classification loss function is used for classification and segmentation tasks; regression loss function is used for bounding box prediction in detection tasks, and classification loss function is used for category prediction; based on the increase or decrease in accuracy of multiple tasks compared with a single task, the weights of the multi-task loss function are manually adjusted or automatically adjusted; commonly used classification loss functions include L1 loss function, L2 loss function, Huber loss function, etc.; common regression loss functions include cross entropy loss function, Focal loss function, etc.

[0057] S1-2) Perform task prediction inference on the trained domain-specific model, including the following steps:

[0058] S1-2-1) Sensor input feature extraction: Based on the input sensor type and the required task, the required backbone network and neck network are selected as the feature extraction network. The image backbone network is used to extract multi-scale features of the image. Lightweight models include ResNet, DarkNet, MobileNet, and Res2Net, while high-precision models include SwinTransformer, InternImage, VovNetv2, and CBNet. The neck network is used to fuse the multi-scale features of the image, often using a feature pyramid network and its various variants to aggregate multi-scale features. After obtaining the 2D image feature sequence, proceed to step S1-2-2). If there is multimodal input of radar point cloud from lidar and millimeter-wave radar sensors, the radar point cloud is first voxelized to obtain feature voxels, and then the required backbone network and neck network are selected according to the needs of the task. Common methods include SECOND, etc. Then, the voxel coordinate system is transformed using the pose parameters of the radar sensor to align different sensors to a unified BEV coordinate system. Then, cylindrical pooling is performed to convert the feature voxels into BEV feature maps. The radar BEV features directly enter step S1-2-3) for 3D feature fusion.

[0059] S1-2-2) 2D-3D feature conversion. For the image backbone network, select a 2D to 3D perspective conversion module that is compatible with it, and use a method based on depth estimation and projection according to depth distribution to project two-dimensional features into three-dimensional features, such as the monocular convolution method of BEVDepth, the multi-eye and multi-frame depth estimation method of MVSNet, etc.; or perform position encoding on the BEV grid and extract features directly from multi-frame and multi-perspective images through the attention mechanism to achieve 2D to 3D feature conversion.

[0060] S1-2-3) Temporal fusion of 2D and 3D features can be performed using any existing temporal fusion method, such as designs based on Transformer encoders, designs based on channel-dimensional direct connections and convolutions, etc. The input to the 2D feature temporal fusion network is a 2D image feature sequence, and the fused 2D features are decoded by the 2D task decoder in step S1-2-4). The input to the 3D feature temporal fusion network includes 3D image features and, if a radar point cloud input is present, 3D radar point cloud features. The fused 3D feature map is decoded by the 3D task decoder in step S1-2-4).

[0061] S1-2-4) For each task, select the feature map generated for the task, connect it to the decoding network of the corresponding task, and predict the results of each task; the decoding network of the corresponding task includes: 2D classification task decoder, 3D classification task decoder, 2D object detection task decoder, 3D object detection task decoder, 2D instance segmentation task decoder, and 3D top view semantic segmentation decoder.

[0062] The decoder for 2D classification tasks and 3D classification tasks usually obtains a global feature vector through convolution, downsampling, and linear layers, and then reduces the dimension of the global feature vector through a multi-layer linear perceptron (MLP) to obtain the domain-specific branch 2D classification prediction result R spec2Dcls =(c spec2D ,c spec2D ) and domain-specific branch 3D classification prediction results R spec3Dcls =(c spec3D ,s spec3D ), where c spec2D and c spec3D is the category prediction, s spec2D and s spec3D is the confidence level.

[0063] The 2D object detection task decoder can choose one-stage or multi-stage, anchored, anchorless or Transformer-based decoder, and finally obtain the domain-specific branch 2D object detection prediction result. in is the 2D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N spec2Ddet is the predicted number of objects.

[0064] The 2D instance segmentation task decoder usually relies on 2D object detection and MLP to predict the mask and category corresponding to each instance. in Refers to the mask of the i-th instance, c i is the category corresponding to the i-th instance, N spec2Dseg is the number of predicted instances.

[0065] The 3D object detection task decoder can select different 3D detection decoders according to the needs of specific scenarios, such as the first-stage PointPillars detection head with anchors, the first-stage Centerpoint detection head without anchors, the first-stage Transformer-based PETR detection head, or the second-stage Transfusion-L detection head, etc., and finally obtain the domain-specific branch 3D object detection prediction results. in is the 3D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N spec3Ddet is the predicted number of objects.

[0066] 3D top view semantic segmentation decoder, you can choose different semantic segmentation decoders according to the needs of the specific scene, such as convolution-based SegNet or Transformer-based PETR SegHead, and finally get the 3D top view semantic segmentation prediction result R spec3Dseg , it is a size and overlooking Figure 1 A discrete matrix with the same number of categories as the number of semantic segmentation categories.

[0067] S2) Based on the sensor input, an open set model is obtained by combining a series of basic models, and combined with a 2D to 3D object conversion module to obtain 2D and 3D multi-task prediction results. The open set model branch may not have strong prediction capabilities within its own whitelist domain, but it has prediction capabilities for open scenarios and non-whitelist semantics. This branch uses the 2D open set detection model to obtain 2D object detection bounding boxes, semantic features, and confidences. The semantic features are decoded through a natural language text decoder to obtain their corresponding semantic predictions, thereby obtaining open set 2D object detection prediction results. The 2D object detection box is used as a prompt and is passed through the 2D instance segmentation basic model with multi-camera image input to obtain 2D instance segmentation prediction results. The pixels of the 2D instance segmentation are combined with their point cloud depth (if there is a radar point cloud) or using a depth estimation basic model to obtain their 3D coordinates, thereby obtaining a pseudo point cloud. The pseudo point cloud is aggregated to obtain a 3D bounding box. The 3D bounding box corresponds one-to-one with the semantic prediction and confidence of the aforementioned 2D object detection. The 3D bounding box, confidence, and semantic prediction together constitute the 3D object detection prediction result. For 2D or 3D classification tasks, design prompt words and directly use the visual-language model to obtain prediction results.

[0068] S2-1) Open-set 2D object detection task prediction. Typically, an open-set detection model consists of three modules: an image encoder, a 2D open-set detector, and a text decoder. These modules produce open-set 2D object detection prediction results. For example, open-set detection methods such as GenerateU, which are fine-tuned from a large vision-language model, can directly use the image encoder, 2D open-set detector, and text decoder to produce open-set 2D object detection prediction results:

[0069]

[0070] in is the 2D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N open2Ddet is the predicted number of objects.

[0071] In addition, we can also use an instance segmentation base model (such as SAM) followed by a 2D bounding box fitting module as an image encoder and a 2D open set detector to obtain and Then use the visual-language model as a text decoder and input the drawn The image of the bounding box requires the model to give a prediction of the semantics of the bounding box

[0072] S2-2) Open set 2D instance segmentation task prediction. This embodiment adopts the instance segmentation basic model SAM, takes multi-camera image input as its input, and converts the bounding box of the open set 2D object detection prediction result obtained in step S2-1) into As a hint, get the mask corresponding to each instance Combined with the open set 2D object detection categories, we get the open set 2D instance segmentation prediction results The number of instances N open2Dseg =N open2Ddet .

[0073] S2-3) Open set 3D object detection task prediction. Due to the insufficient data mentioned in the background technology section, it is impossible to directly train an open set 3D detector. This invention uses a 2D-3D object conversion module to further convert the 2D object detection and 2D instance segmentation prediction results with strong versatility into 3D object detection prediction results. The following are the steps for performing open set 3D object detection. Specifically,

[0074] S2-3-1) Use the depth estimation basic model (such as UniDepth, ZoeDepth, etc.) to perform depth estimation on the image and obtain the depth estimation result matrix D. For each instance mask of the 2D semantic segmentation prediction result For each pixel (i, v), the corresponding depth d = D can be found in the depth estimation result. u,v ; When a radar point cloud input is provided, the point cloud can be projected onto the image plane, and the depth d is obtained by interpolating the k nearest neighbor point clouds near (u,v).

[0075] S2-3-2) For instance mask Each pixel and its depth (u, v, d) in the image is transformed into a coordinate system using the camera's internal and external parameter matrices to obtain a pseudo point cloud (x, y, z) in the corresponding space.

[0076] S2-3-3) For all pseudo point clouds corresponding to a mask, we can find the minimum volume cuboid that contains all the point clouds, that is, convert it into a three-dimensional minimum directed bounding box problem (OBB problem). We use principal component analysis (PCA), genetic algorithm, simulated annealing or other heuristic search algorithms to approximate the solution to solve the parameters of the corresponding 3D bounding box. You can also use a learning-based method, using a neural network model, taking all pseudo point clouds corresponding to a mask as input, using PointNet to extract features, and using MLP to predict the parameters of the corresponding 3D bounding box Finally, we get the open set 3D target detection prediction results in is the 3D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N open3Ddet is the predicted number of objects.

[0077] S2-4) Open set 2D and 3D classification task prediction: Use the large visual-language model VLM (GPT-4V, InternVL 1.5, etc.) to directly input the required classification as a prompt text to obtain the open set branch 2D classification prediction result R open2Dcls =(c open2D ,c open2D ) and the open set branch 3D classification prediction results R open3Dcls =(c spec3D ,s spec3D ), where c open2D and c open3D is the category prediction, s open2D and s open3D Confidence. For example, when the classification target is weather, you can use "Please use one word to describe the weather in the image" or "Which of the following types of weather is the weather in the image: sunny; cloudy; rainy; snowy?" as a prompt word (for GPT-4V and InternVL 1.5, corresponding English prompt words are more accurate). The prompt word should be designed based on the content of the specific classification. To estimate the confidence level, you can ask the VLM to provide a confidence estimate in the range [0,1] for its prediction in the prompt word, or use the VLM's text encoder to calculate the similarity of the text feature vector between the model output and several preset categories one by one.

[0078] S3) The prediction results of the domain-specific model branch and the open set model branch are fused through the integration module to obtain prediction results that reflect both strong capabilities within the domain and a certain degree of open set prediction capability. For 2D and 3D object detection, after aligning the confidence distributions, dual threshold non-maximum suppression based on semantic feature distance and intersection-over-union ratio is used for integration. For 2D instance segmentation, the integration is guided by the 2D object detection prediction results. For classification tasks, the category selection is determined by the confidence level.

[0079] S3-1) Classification task integration: For the domain-specific 2D classification prediction results R obtained from the above two branches spec2Dcls =(s spec2D ,s spec2D ) and open set 2D classification prediction results R open2Dcls =(c open2D ,s open2D ), take the prediction result with high confidence as the final 2D classification prediction result:

[0080]

[0081] And update the confidence level. If the two prediction results are consistent, increase the confidence level, otherwise decrease the confidence level:

[0082]

[0083] Finally, the integrated 2D classification prediction result R is obtained 2Dcls =(c 2D ,s 2D The integration process of 3D classification is exactly the same as that of 2D classification, so it will not be repeated here.

[0084] S3-2) Detection task integration: For domain-specific 2D object detection prediction results and open set 2D object detection prediction results First, calculate the confidence of the prediction results of the domain-specific 2D target detection on the specified dataset Distribution, get the mean μ and standard deviation σ; then open the confidence of the open set 2D target detection prediction result Redistribution is a confidence distribution that conforms to the Gaussian distribution N(μ,σ) Finally, the non-maximum suppression (NMS) process is performed on the set of domain-specific detection prediction results and the open set prediction results after confidence redistribution, and the combined set is:

[0085]

[0086] Specifically, for any two detection prediction results in the detection prediction result union set, set to the i-th and j-th, if the intersection of the two result bounding boxes is IoU (b i ,b j ) exceeds the specified threshold T, the results with lower confidence s are removed from the set. Finally, the integrated 2D target detection prediction result R is obtained 2Ddet The integration process of 3D object detection is exactly the same as the above 2D object detection integration process, so it will not be repeated here.

[0087] S3-3) Segmentation task integration: For 2D instance segmentation tasks, each 2D segmentation instance result Both with a 2D target detection prediction result One-to-one correspondence, therefore, record the target detection prediction results removed by NMS in step S3-2) and remove the corresponding segmentation instances from R spec2Dseg ∪R open2Dseg By removing it, we can get the final integrated 2D instance segmentation prediction result R 2Dseg For the 3D top view semantic segmentation task, the prediction results are integrated, R 3Dseg=R spec3Dseg .

[0088] For the above embodiments, Figure 3 Several examples of 2D object detection are presented to demonstrate how the present invention effectively combines the strengths of domain-specific and open set models. In the first row of examples, the traditional domain-specific model cannot identify a dog on the road because it lacks animal data in its training data, but it can provide a relatively complete estimate of a distant, partially occluded vehicle. The open set model can identify the dog on the road, but its prediction for the distant, partially occluded vehicle is biased. The proposed algorithm combines the advantages of both. In the second row of examples, the domain-specific model can identify two overlapping people in the middle of the road and an overlapping, occluded vehicle on the left, but the open set model performs poorly for these objects. Furthermore, for non-whitelisted semantic objects such as baby strollers and lampposts, the open set model can make predictions, while the domain-specific model cannot. The proposed algorithm combines the advantages of both. In the third row of examples, the open set model can predict road signs, while the domain-specific model can predict pedestrians at a distance. The proposed algorithm combines the advantages of both.

[0089] Finally, it should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. An autonomous driving perception method based on the integration of domain-specific models and open source models, It is characterized in that The sensor input of the method is a multi-camera image input or a multi-modal input of a multi-camera image and a radar point cloud, and includes the following steps: 1) Based on the sensor input, a domain-specific model is trained to reason about six tasks simultaneously, and 2D and 3D multi-task prediction results are obtained through this domain-specific model; 2) According to the sensor input, an open set model is obtained by combining a series of basic models, and a 2D to 3D object conversion module is designed to obtain 2D and 3D multi-task prediction results; through the 2D open set detection model, a 2D target detection bounding box, semantic features and confidence are obtained, and the semantic features are decoded through a natural language text decoder to obtain the corresponding semantic prediction, thereby obtaining an open set 2D target detection prediction result; the 2D target detection box is used as a prompt and multi-camera image input is passed through a 2D instance segmentation basic model to obtain a 2D instance segmentation prediction result; the pixels segmented from the 2D instance are combined with their point cloud depth or a depth estimation basic model to obtain their three-dimensional coordinates, thereby obtaining a pseudo point cloud, and the pseudo point cloud is aggregated to obtain a 3D bounding box, which corresponds one-to-one to the semantic prediction and confidence of the 2D detection. The bounding box, confidence and semantic prediction together constitute the 3D target detection prediction result; for 2D or 3D classification tasks, prompt words are designed and the results are obtained directly using the visual-language large model; 3) Through the integration module, the multi-task prediction results of the domain-specific model and the multi-task prediction results of the open set model are integrated to obtain the final result; for 2D object detection and 3D object detection, after the confidence distribution is aligned, the double threshold non-maximum suppression based on semantic feature distance and intersection-over-union ratio is adopted for integration; for 2D instance segmentation, the integration is guided by the 2D object detection prediction results; for classification tasks, the category selection is determined according to the confidence.

2. The method according to claim 1, It is characterized in that The step 1) specifically comprises the following steps: 1-1) Domain-specific model training During training, different multimodal data augmentation strategies are used for different inputs; During training, select different optimizers and learning rate schedulers based on the actual training situation of the model; During training, the multi-task loss function is designed as follows: classification and segmentation tasks use classification loss functions; detection tasks use regression loss functions for bounding box predictions, and classification loss functions for category predictions; based on the increase or decrease in accuracy of multi-tasks compared to single tasks, the weights of the multi-task loss function are adjusted manually or automatically; 1-2) Perform task prediction reasoning on the trained domain-specific model, including the following steps: 1-2-1) Sensor input feature extraction, according to the type of input sensor and the required task, select the required backbone network and neck network as the feature extraction network; for multi-camera image input, the backbone network is used to extract the multi-scale features of the image, and the neck network is used to fuse the multi-scale features of the image; obtain a 2D image feature sequence; when there is also a multi-modal input of radar point cloud, first voxelize the radar point cloud to obtain feature voxels, and then select the required backbone network and neck network according to the needs of the task, extract the radar point cloud 3D features, and fuse them with the image 3D features in step 1-2-3); 1-2-2) 2D-3D feature conversion: For the image backbone network, select a 2D to 3D view conversion module that is compatible with it, use a method based on depth estimation and depth distribution projection to project 2D features to 3D features, or perform position encoding on the BEV grid, and extract features directly from multi-frame multi-view images through the attention mechanism to achieve 2D to 3D feature conversion; 1-2-3) Temporal fusion of 2D and 3D features, any existing temporal fusion method can be used; for 2D tasks, the input of the 2D feature temporal fusion network is the 2D image feature sequence, and the output is the fused 2D features; for 3D tasks, the input of the 3D feature temporal fusion network is the 3D feature map sequence after 2D-3D feature conversion, if there is a radar point cloud input, it also includes the 3D features of the radar point cloud, and the output is the fused 3D features; 1-2-4) For each task, select the feature map generated for the task, connect to the decoding network of the corresponding task, and predict the results of each task; the decoding network of the corresponding task includes: 2D classification task decoder, 3D classification task decoder, 2D object detection task decoder, 3D object detection task decoder, 2D instance segmentation task decoder, 3D top view semantic segmentation decoder; get the results of each domain-specific task, including: 2D classification prediction result R spec2Dcls =(c spec2D ,s spec2D ) and 3D classification prediction results R spec3Dcls =(C spec3D ,s spec3D ), where C spec2D and C spec3D is the category prediction, s spec2d and spec3D is the confidence level; 2D target detection prediction result in is the 2D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N spec2Ddet is the number of predicted objects; 3D target detection prediction results in is the 3D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N spec3Ddet is the number of predicted objects; 2D instance segmentation prediction result in refers to the mask of the i-th instance, c i is the category corresponding to the i-th instance, N spec2Dseg is the number of predicted instances, and the 3D top view semantic segmentation prediction result R spec3Dseg .

3. The method according to claim 1, It is characterized in that The step 2) specifically comprises the following steps: 2-1) Open set 2D object detection task prediction, through an open set detection model composed of image encoder, 2D open set detector and text decoder, the open set 2D object detection prediction result is obtained in is the 2D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N open2Ddet is the predicted number of objects; 2-2) Open set 2D instance segmentation task prediction, using the instance segmentation basic model, first take the multi-camera image input as its input, and convert the bounding box of the open set 2D object detection prediction result obtained in step 2-1) As a hint, get the mask corresponding to each instance Combined with the open set 2D object detection categories, we get the open set 2D instance segmentation prediction results The number of instances N open2Dseg ×N open2Ddet ; 2-3) Open set 3D object detection task prediction, through the 2D-3D object conversion module, based on the 2D object detection prediction results and the 2D instance segmentation prediction results, 3D object detection prediction is performed; including the following steps: 2-3-1) Use the depth estimation basic model to perform depth estimation on the image and obtain the depth estimation result matrix D; for each instance mask of the 2D instance segmentation prediction result For each pixel (u, v) in the depth estimation result, the corresponding depth d = D u , v; when the radar point cloud input is provided, the point cloud is projected to the image plane, and the depth d is obtained by interpolating the k nearest neighbor point cloud near (u, V); 2-3-2) For instance mask For each pixel and its depth (u, v, d), use the camera's internal and external parameter matrices to transform the coordinate system and obtain a pseudo point cloud (x, y, z) in the corresponding space; 2-3-3) For all pseudo point clouds corresponding to a mask, find the cuboid with the smallest volume that contains all point clouds, that is, convert it into a three-dimensional minimum directed bounding box problem; use a heuristic search algorithm to approximate the solution to solve the corresponding 3D bounding box parameters Or adopt a learning-based method, use a neural network model, take all pseudo point clouds corresponding to a mask as input, use PointNet to extract features, and predict the parameters of the corresponding 3D bounding box through MLP Finally, the open set 3D object detection prediction results are obtained: in in is the 3D bounding box corresponding to the i-th object, is the confidence corresponding to the i-th object, is the category corresponding to the i-th object, N open3Ddet is the predicted number of objects; 2-4) Open set 2D and 3D classification task prediction, using the visual-language large model, directly input the required classification as the prompt text, and obtain the open set 2D classification prediction result R spec2Dcls =(C open2D ,s open2D ) and the open set 3D classification prediction results R open3Dcls =(C spec3D ,s spec3D ), where c open2D and C open3D is the category prediction, s open2D and open3D For confidence.

4. The method according to claim 1, It is characterized in that The step 3) specifically comprises the following steps: 3-1) Classification task integration, determine the category selection based on the confidence level, and for the domain-specific 2D classification prediction result R spec2Dcls =(c spec2D ,s spec2D ) and the open set 2D classification prediction results R open2Dcls =(C open2D ,s open2D ), take the prediction result with high confidence as the final 2D classification prediction result: And update the confidence level. When the two prediction results are consistent, the confidence level is increased, otherwise the confidence level is decreased: Finally, the integrated 2D classification prediction result R is obtained 2Dcl s=(C 2D ,s 2D ) The integration process of 3D classification is exactly the same as that of 2D classification; 3-2) Detection task integration, for domain-specific 2D target detection prediction results R spec2Ddet And the open set 2D object detection prediction results R open2Ddet After the confidence distribution is aligned, the integration is performed by adopting the double threshold non-maximum suppression based on the semantic feature distance and intersection-over-union ratio; specifically, the confidence of the prediction results of the domain-specific 2D object detection on the specified data set is first calculated. Distribution, get the mean μ and standard deviation σ; then open the confidence of the open set 2D target detection prediction result Redistribute to a confidence distribution that conforms to the Gaussian distribution N(μ,σ) Finally, the set of domain-specific detection prediction results and the open set detection prediction results after confidence redistribution is processed by non-maximum suppression NMS, and the combined set is: Specifically, for any two detection prediction results in the detection prediction result union set, set to the i-th and j-th, when the intersection over union (IoU) of the bounding boxes of the two results is i , b j ) exceeds the specified threshold T, and the results with lower confidence s are removed from the set; finally, the integrated 2D target detection prediction result R is obtained 2Ddet ; The integration of 3D object detection is exactly the same as the integration process of 2D object detection; 3-3) Segmentation task integration. For 2D instance segmentation tasks, the 2D object detection prediction results are used to guide the integration. Each 2D segmentation instance result Both with a 2D target detection prediction result One-to-one correspondence, therefore, record the target detection prediction results removed by NMS in step 3-2) and remove the corresponding segmentation instances from R spec2Dseg ∪R open2Dseg By removing it, we can get the integrated 2D instance segmentation prediction result R 2Dseg ; For the 3D top view semantic segmentation task, the integrated prediction result R 3Dseg =R spec3Dseg .

Citation Information

Cited By

  • Printing quality monitoring method and system based on artificial intelligence

    CN121861004A