3d target detection method for autonomous driving using synergy of heterogeneous sensors

Abstract

A method of performing target detection during autonomous driving, comprising: performing 3D target detection in a 3D target detection segment; uploading outputs of a plurality of sensors in communication with the 3D target detection segment into a plurality of point clouds; transmitting point cloud data of the plurality of point clouds to a region proposal network (RPN); independently performing 2D target detection in a 2D target detector, the 3D target detection being performed in parallel in the 3D target detection segment; and obtaining a given input image and simultaneously learning bounding box coordinates and class label probabilities in a 2D target detection network that operates to treat target detection as a regression problem.

Description

Technical Field

[0001] This disclosure relates to target detection during the operation of autonomous vehicles. Background Technology

[0002] One of the most critical components of autonomous driving is 3D object detection. Autonomous vehicles need to accurately detect and locate other vehicles and pedestrians in 3D space for safe driving. Recently, 2D object detection technology has made significant progress. While 2D detection algorithms are mature, 3D object detection still faces considerable challenges. In current autonomous driving systems, 3D object detection is primarily based on cameras or 3D sensors. The most commonly used 3D sensor is the Laser Imaging Detecting and Ranging (LIDAR) sensor, which generates 3D point clouds to capture the 3D structure of the scene.

[0003] Image-based methods can utilize monocular or stereo images. Methods that rely solely on 2D object detection impose additional geometric constraints to create 3D candidates. Due to a lack of depth information, these methods only produce coarse 3D detection results and are heavily influenced by appearance variations. Other methods apply monocular or stereo-based depth estimation to obtain the 3D coordinates of each pixel. These 3D coordinates are then fed as additional input channels into the 2D detection pipeline or used to extract handcrafted features.

[0004] Therefore, existing autonomous driving methods primarily rely on LiDAR sensors for accurate 3D target detection. While pseudo-LiDAR has recently been introduced as a promising alternative, a significant performance gap remains, and this gap widens when tested on different datasets (other than KITTI). This indicates that pseudo-LiDAR remains inaccurate in generalization.

[0005] Therefore, while current autonomous driving methods have achieved their intended purpose, a new and improved approach is still needed for target detection during autonomous driving. Summary of the Invention

[0006] According to several aspects, a method for performing object detection during autonomous driving includes: performing 3D object detection in a 3D object detection segment; uploading the outputs of multiple sensors communicating with the 3D object detection segment to multiple point clouds; transmitting the point cloud data of the multiple point clouds to a region candidate network (RPN); performing 2D object detection independently in a 2D object detector and performing 3D object detection in parallel in the 3D object detection segment; and acquiring a given input image in the 2D object detection network and simultaneously learning bounding box coordinates and class label probabilities, wherein the 2D object detection network operates to treat object detection as a regression problem.

[0007] In another aspect of this disclosure, the method also includes operating multiple laser imaging detection and ranging (LIDAR) sensors in a 3D target detection segment to generate the outputs of the multiple sensors, thereby further generating a 3D point cloud to capture a set of 3D structures in a vehicle-visible scene.

[0008] In another aspect of this disclosure, the method also includes operating the RPN to assign data of multiple point clouds to individual points in the point cloud within a 3D point cloud segmentation component, and assigning labels representing real-world entities.

[0009] In another aspect of this disclosure, the method also includes transmitting the output of the RPN to a region-based convolutional neural network (RCNN).

[0010] In another aspect of this disclosure, the method further includes: applying a 3D box estimator to generate one or more bounding boxes (BBs); and fusing the output of the 3D box estimator with the 2D object output of the 2D object detector and passing it to the box consistency and filtering unit.

[0011] In another aspect of this disclosure, the method also includes enhancing 2D detection by combining one-stage 2D object detection with two-stage instance segmentation.

[0012] In another aspect of this disclosure, the method also includes automatically segmenting and constructing pixel-level masks for each target in the image in an instance segmentation network.

[0013] In another aspect of this disclosure, the method further includes: generating regions in the image that may contain the target; sorting the regions according to a score indicating the probability that any region in the determined region may contain the target; and retaining the top "N" regions with the highest confidence scores.

[0014] In another aspect of this disclosure, the method further includes: passing the image output of the camera to an instance segmentation deep neural network (DNN) having an instance segmentation device, wherein different instances of the target receive different labels; and moving the output of the instance segmentation device to an instance mask detector, wherein the output of the segmentation device is a binary mask of the region.

[0015] In another aspect of this disclosure, the method further includes: transmitting 2D data of a 2D object detection segment of data defining the image of the camera to a 2D object detector; transmitting the output of the 2D object detector together with the output of an instance mask detector to a constraint device; and sending the output of the constraint device and the DNN to an enhanced 2D detector.

[0016] According to several aspects, a method for performing target detection during autonomous driving includes: receiving sensor data from multiple sensors and applying the sensor data to generate a 3D point cloud to capture 3D structures; performing 3D target detection in a 3D target detector, including identifying multiple 3D targets directly from the point cloud; performing enhanced 2D target detection in parallel with the 3D target detection segment to identify 2D targets using the enhanced 2D target detector; co-processing the 2D targets and 3D targets in a 2D and 3D co-processing segment; and generating a final 3D target detection for convergent perception.

[0017] In another aspect of this disclosure, the method also includes inputting data of the 3D target into a 3D bounding box estimator.

[0018] In another aspect of this disclosure, the method further includes passing the output of the 3D box estimator and the output of the enhanced 2D detector to a box consistency and filtering unit to generate multiple bounding boxes (BBs).

[0019] In another aspect of this disclosure, the method further includes filtering multiple bounding boxes (BBs) based on their high overlap with high-confidence 2D candidates after being projected onto multiple images used as filters, thereby reducing false positives that are incorrectly detected in the point cloud.

[0020] In another aspect of this disclosure, the method further includes: generating regions of multiple images; sorting the regions according to a score indicating the probability that any region may contain one of multiple 3D targets; retaining the top "N" highest-confidence-rated regions; sending a single region from the N highest-confidence-rated regions through three parallel branches of an instance segmentation network, wherein the three parallel branches define label prediction, BB prediction, and mask prediction; computing a binary mask for each of the N highest-confidence-rated regions and automatically segmenting and constructing a pixel-level mask for each target in the image; and removing redundant candidates using the confidence score and non-maximum suppression (NMS) based on IoU_t.

[0021] In another aspect of this disclosure, the method also includes enhancing 2D object detection by applying instance segmentation together with a 2D object detector.

[0022] In another aspect of this disclosure, the method further includes: fusing image data and sensor data; and retaining a single target consistent in both the 2D target detection segment and the 2D target detection segment among the 2D target and the 3D target.

[0023] A system for performing object detection during autonomous driving includes a 3D object detection segment that performs 3D object detection. Multiple sensors communicate with the 3D object detection segment, each sensor having its output uploaded to one of multiple point clouds. Point cloud data is transmitted from the multiple point clouds to a Region Candidate Network (RPN). A 2D object detector independently performs 2D object detection and performs it in parallel with the 3D object detection in the 3D object detection segment. The 2D object detection network operates by treating object detection as a regression problem, taking a given input image, and simultaneously learning bounding box coordinates and class label probabilities.

[0024] In another aspect of this disclosure, multiple sensors are defined as laser imaging detection and ranging (LIDAR) sensors for capturing a set of 3D structures in a vehicle-visible scene.

[0025] In another aspect of this disclosure, the camera outputs an image. An instance segmentation deep neural network (DNN) has an instance segmentation device, wherein different instances of a target receive different labels. An instance mask detector receives the output of the instance segmentation device, wherein the output of the instance segmentation device defines a binary mask of the region of the scene visible to the vehicle.

[0026] Further areas of applicability will become apparent from the description provided herein. It should be understood that the descriptions and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0027] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0028] Figure 1 This is a flowchart of an autonomous driving 3D target detection method and system based on an exemplary aspect;

[0029] Figure 2 yes Figure 1 The flowchart of the 2D and 3D collaborative part of the method. Detailed Implementation

[0030] The following description is merely exemplary in nature and is not intended to limit this disclosure, application, or use.

[0031] See Figure 1An autonomous driving 3D target detection method 10 provides 3D target detection in a 3D target detection segment 12 and independently provides 2D target detection in a 2D target detection segment 14. As described in Algorithm 1, a pseudocode algorithm is applied. The 3D target detection segment 12 receives data from one or more LiDAR (LiDAR) sensors 16 to generate 3D (3D) point clouds to capture a set of 3D structures in the vehicle's visible scene. The output of the LiDAR sensors 16 is uploaded to multiple point clouds 18. Starting from point clouds 18, the point cloud data is transmitted to a Region Proposal Network (RPN) 20, where the data is processed in a 3D point cloud segmentation component 22, which performs a task in which each point in the point cloud is assigned a label representing a real-world entity, and the data is processed by a 3D candidate generator 24.

[0032] The output of RPN 20 is fed to a Region-based Convolutional Neural Network (RCNN) 26, where a 3D box estimator 28 is applied to generate one or more bounding boxes (BBs). RCNN 26 first uses Selective Search to identify a controlled number of bounding boxes or BB object region candidates in the Region of Interest (ROI). Then, RCNN 26 extracts Convolutional Neural Network (CNN) features from the searched independent regions of the scene for classification. The output of RCNN 26 is passed through the 3D detector 29 of the 3D object detection segment 12, and the output of the 3D object detection segment 12 is fused with the output of the enhanced 2D detector 46 of the 2D object detection segment 14 (discussed below) and passed to the box consistency and filtering unit 30.

[0033] 2D object detection is performed concurrently with 3D object detection, summarized as follows: 2D detection is enhanced by combining one-stage 2D object detection with two-stage instance segmentation. The 2D object detection network treats object detection as a regression problem, taking a given input image and simultaneously learning BB coordinates and corresponding class label probabilities. Then, the instance segmentation network automatically segments and constructs pixel-level masks for each object in the image. The same two-stage procedure is also employed, where the first stage is the RPN (Regions Per Numerator) to generate regions (RPNs) in the image that may contain objects. Each region is ranked according to a score indicating the probability that a given region may potentially contain an object, and then the top "N" regions with the highest confidence scores are retained for the second stage.

[0034] While the 3D object detection segment 12 operates in parallel, the 2D object detection segment 14 is equipped with a camera 32. The image output of the camera 32 is passed to an instance segmentation deep neural network (DNN) 34 with an instance segmentation device 36, where each instance of a given object receives a different label. The output of the instance segmentation device 36 is passed to an instance mask detector 38, which produces a binary mask for each region instead of a bounding box. While the instance segmentation device 36 operates in parallel within the DNN 34, the 2D object detector DNN 40 passes the 2D data of the camera 32 images to a 2D object enhancer 42. The output of the 2D object enhancer 42, along with the output of the instance mask detector 38, is transmitted to a constraint device 44. Due to consistency constraints, the 2D bounding box (BB) of the deep learning-based object detection output of the 2D object enhancer 42 also highly overlaps with the corresponding instance mask of the DNN 34 output, thus providing a high-confidence combined 2D object candidate. The outputs of the constraint device 44 and the DNN 34 are then passed to the enhanced 2D detector 46.

[0035] In addition to the output of the 3D bounding box estimator 28, the output of the enhanced 2D detector 46 is also passed to the box consistency and filtering unit 30 of the 2D and 3D co-processing segment 48. After being projected onto the image, the box consistency and filtering unit 30 filters the predicted 3D bounding boxes based on their high overlap with the corresponding high-confidence 2D candidates. This function is used for filtering, i.e., reducing false positives that are incorrectly detected in the point cloud 18, and including detected targets in the image that were not detected from the point cloud 18, in order to reduce the false negative rate. The final result produced by the 3D co-processing segment 48 is the final 3D object detection 50 used for aggregation sensing 52.

[0036] Therefore, in summary, the autonomous driving 3D object detection method 10 can be divided into three steps: step 1) 3D object detection in the 3D object detection segment 12; step 2) enhanced 2D object detection in the 2D object detection segment 14; and step 3) coordination of 2D and 3D detection in the 2D and 3D coordination segment 48. 3D object detection is performed directly from the point cloud 18, and 2D object detection is enhanced using a 2D object detector and instance segmentation. Finally, image data is fused with LiDAR sensor data, preserving consistent targets in both the 3D and 2D detectors to improve performance and make the results more reliable across different datasets.

[0037] The pseudocode for this architecture is shown in Algorithm 1 below.

[0038] Algorithm 1:

[0039] Algorithm 1 object_detection3D (image, lidar, calibration_file)

[0040] Input: image, lidar, calibration_file

[0041] Output: (3D_detection, class_type, scores, 2D_detection) for each object.

[0042] 3D_detection includes: 3D_location (x, y, z), 3D_bbs_sizes (h, w, l),

[0043] and object orientation (θ). (x, y, z, h, w, l, θ)

[0044] - Load the trained model

[0045] model_3D ← Loads a trained 3D object detection model

[0046] model_2D ← Loads a trained 2D object detection model

[0047] model_mask ← Loads a trained instance segmentation model

[0048] lidar← pre_process_lidar(lidar, calibration_file)

[0049] image← ​​pre_process_image(image, calibration_file)

[0050] Step 1: 3D Object Detection

[0051] rpn_output_3D ← rpn_3D (model_3D, lidar) / / RPN, Stage 1

[0052] rpn_filtered← filter_rpn (rpn_output _3D)

[0053] rcnn_output_3D ← rcnn_network_3D(rpn_filtered, model_3D, lidar) / / RCNN, Stage 2

[0054] pred_boxes_3D, class_type_3D, scores_3D ← rcnn_output_3D

[0055] pred_boxes3d_final ← filter_by_scores_3D (pred_boxes_3D,

[0056] scores_3D, scores_threshold_3D)

[0057] Step 2: Enhance 2D Object Detection

[0058] / / Parallel execution phase 1 2D object detection

[0059] pred_boxes_2D,class_type_2D, scores_2D← network_2D (model_3D, image)

[0060] pred_boxes_2D_final ← filter_by_scores(pred_boxes_2D, scores_2D, scores_threshold_2D, IoU_t) / / Algorithm 2

[0061] / / Parallel execution of instance splitting

[0062] mask, pred_boxes_mask, class_type_mask, scores_mask← mask_network(model_mask, image)

[0063] pred_mask_final ← filter_by_scores(pred_boxes_mask, scores_mask, scores_threshold_mask, IoU_t) / / Algorithm 2

[0064] / / Enhanced 2D object detection

[0065] combined_2D_detection ← combine_2D_detection(pred_boxes_2D_final,pred_mask_final, IoU_t)

[0066] / / Algorithm 3

[0067] - Step 3: Collaboration of 2D and 3D Detection

[0068] B2D = {b2D1, … b2Dm}, where B2D is a combined 2D detection list.

[0069] combined_2D_detection

[0070] B3D = {b3D1, … b3Dm}, where B3D is a list of 3D bounding boxes.

[0071] pred_boxes3d_final

[0072] Filtered _3D_detection← {}

[0073] For b3Di in B3D:

[0074] b3D_projectedi = project_3dto2D(b3Di, calibration_file)

[0075] IoU_result={}

[0076] For bj in B2D:

[0077] IoU_result← IoU_result U IoU(bj, b3D_projectedi)

[0078] IoU_max ← argmax IoU_result

[0079] If IoU_result [IoU_max]>IoU_t:

[0080] Filtered _3D_detection ← Filtered _3D_detection U b3Di

[0081] end

[0082] end

[0083] end

[0084] return Filtered _3D_detection, class_type_3D, scores_3D,

[0085] combined_2D_detection

[0086] The final output of the architecture disclosed herein is 3D_detection, class_type, scores, and 2D_detection for each target, including: 3D_detection: 3D_bbs_sizes (3D target size: height, width, length (in meters)), 3D_location (3D target position x, y, z (in meters) in camera coordinates) and θ is the target orientation; class_type is the target category type; scores are the detection confidence; 2D_detection is the 2D BB of the target in the image (0-based indexing), containing the left pixel coordinates, top pixel coordinates, right pixel coordinates, and bottom pixel coordinates.

[0087] In step 2, each of the N selected regions undergoes three parallel branches of the instance segmentation network: label prediction, bounding box prediction, and mask prediction. In this step, class offsets and bounding box offsets are predicted in parallel, and a binary mask is computed for each region, thereby automatically segmenting and constructing a pixel-level mask for each object in the image. Redundant candidates are removed using confidence scoring and IoU_t-based non-maximum suppression (NMS) (see Algorithm 2 below).

[0088] See Figure 2 And refer again Figure 1 See Figure 1 The signals from the 3D detector 29 and the enhanced 2D detector 46 described above are passed through the box consistency and filtering unit 30. Within the box consistency and filtering unit 30, the 2D projection of the 3D BB 58 and the enhanced 2D BB with pixel mask 60 are passed through the first filter 62 and then to the final 3D BB filter 64 to generate the final 3D object detection 50. During the enhanced 2D detection process, a BB consistency constraint is considered, where a 2D BB is selected as the output of the deep learning-based 2D object detector that highly overlaps with the corresponding instance mask output by the defined instance segmentation network (IoU > IoU_t). Therefore, Algorithm 3 is also used to obtain combined 2D object candidates with high confidence.

[0089] Therefore, the confidence of the 2D detector is increased, retaining only the targets that are consistent in both detectors, to achieve synergy between the 2D target detectors. In the final step, the enhanced 2D BB is used to filter the predicted 3D BB based on the high overlap (IoU > IoU_t) between the projected BB and its corresponding enhanced 2D candidate on the image.

[0090] The pseudocode for filtering 2D BBs from 2D detection based on scoring and nonmaximal suppression is shown in Algorithm 2 below.

[0091] Algorithm 2:

[0092] Algorithm 2 filter_by_scores(BBs, Scores, S_t, IoU_t):

[0093] Input: BBs={bb1, … bbm}, Scores={s1, … sm}

[0094] BBs is a list of detection boxes.

[0095] Scores includes detection scores

[0096] IoU_t is the IoU threshold.

[0097] S_t is the scoring threshold.

[0098] Final_BBs← {}

[0099] When BBs is not empty:

[0100] s_max ← argmax Scores

[0101] If Scores[s_max] <S_t:

[0102] BBs ← BBs – BBs[s_max]

[0103] Scores← Scores - Scores[s_max]

[0104] continue

[0105] end

[0106] B_max ← BBs[s_max]

[0107] Final_BBs← Final_BBs U B_max; BBs ← BBs – B_max

[0108] Forbi in BBs:

[0109] IfIoU(B_max, bi)>IoU_t and same_class_type:

[0110] BBs← BBs - bi ; Scores← Scores - bi

[0111] end

[0112] end

[0113] end

[0114] return Final_BBs

[0115] The pseudocode for combining instance segmentation and 2D object detection in 2D BB is shown in Algorithm 3 below.

[0116] Algorithm 3:

[0117] Algorithm 3 combine_2D_detection pred_boxes2d_final,pred_mask_final,IoU_t

[0118] Input: B2D={b2D1, … b2Dm} , Bmask={bmask1, … bmaskm}

[0119] B2D is a list of 2D detection boxes.

[0120] Bmask is a list of mask detection bounding boxes.

[0121] IoU_t is the IoU threshold.

[0122] Combined_BBs← {}

[0123] For bi in B2D:

[0124] IoU_result={}

[0125] For bj in Bmask:

[0126] IoU_result← IoU_result U IoU(bj, bi)

[0127] IoU_max ← argmax IoU_result

[0128] If IoU_result [IoU_max]>IoU_t:

[0129] To perform 3D object detection from point cloud 18, operations are performed directly on the unordered point cloud in 3D, unlike other methods that use projected point clouds to electric vehicles (BEVs) or operate on quantized 3D tensor data (voxels). For example, a visual benchmark can be used to train the 3D detection network, which can display up to 15 cars and 30 pedestrians per image. The RCNN network disclosed herein includes two separately trainable subnetworks: a Region Proposal Network (RPN) 20 and a Region CNN or RCNN 26. RPN 20 is trained first, and after online training of RCNN 26, ground truth boxes are used for augmentation, which copies the object boxes and interior points from one 3D point cloud scene to the same location in another 3D point cloud scene. For each 3D point cloud scene in the training set, points in each 3D point cloud scene are subsampled as input, so the input size is always the same n_points. For 3D point cloud scenes with fewer than n_points, the points are randomly repeated to obtain an exact count of n_points. Redundant candidates are also removed using NMS based on directional IoU from the BEV (Body EV) to generate a small number of high-quality candidates. For example, directional NMS with an IoU threshold IoU_tx is used, and only the top-ranked candidates are retained for refining the two-stage subnetwork. The 3D BB is represented in LiDAR coordinates as (x, y, z, h, w, l, θ), where (x, y, z) is the target center position, (h, w, l) is the target size, and θ is the target orientation of the BEV.

[0130] In the final 3D detection, and after projection onto the image, corresponding enhanced 2D candidates are used, and the predicted 3D BB is filtered based on the high overlap defined as IoU > IoU_t. This filters out targets that are incorrectly detected in point cloud 18, reducing the false detection rate, and includes detected targets in the image that were not detected from point cloud 18, reducing the false negative rate. The final result includes the synergy of 2D BB and 3D BB from heterogeneous sensors.

[0131] This disclosed autonomous driving 3D object detection method 10 offers several advantages. These advantages include a hybrid, aggregated perception approach that simultaneously utilizes 2D object detectors and enhanced 3D object detection, rather than relying solely on 3D candidates. Utilizing learning directly received from the raw point cloud, the 3D bounding box (BB) can be accurately estimated even in scenes with strong occlusion or very sparse points, and 2D object detection is further applied. Since the point cloud does not consider any visual information also relevant to detection, noise and erroneous detections generated by the point cloud are filtered out. To perform 3D object detection from the point cloud, operations are performed directly on the unordered point cloud in 3D, contrasting with known methods that use projected point clouds to the BEV or operate on quantized 3D tensor data (voxels). 2D detection is enhanced by combining a one-stage 2D object detection (treating object detection as a regression problem) with a two-stage instance segmentation. The first stage is a Region Proposal Network (RPN), and in the second stage, class offsets and box offsets are predicted in parallel, and a binary mask is computed for each region, thereby automatically segmenting and constructing pixel-level masks for each object in the image. Furthermore, consistent targets are preserved in both detectors. The end result is an improvement to the current perception pipeline through the synergy of 2D BB and 3D BB from heterogeneous sensors.

[0132] This disclosure provides a combined approach to improve 3D object detection results.

[0133] The description in this disclosure is merely exemplary in nature, and any changes that do not depart from the spirit and scope of this disclosure are intended to fall within its scope. Such changes should not be considered as departing from the spirit and scope of this disclosure.

Claims

1. A method for performing object detection during autonomous driving, comprising: Perform 3D object detection in the 3D object detection section; The outputs of multiple sensors communicating with the 3D target detection segment are uploaded to multiple point clouds; The point cloud data of the multiple point clouds is transmitted to the Regional Candidate Network (RPN). 2D target detection is performed independently in the 2D target detection segment, and 3D target detection is performed in parallel in the 3D target detection segment; as well as A given input image is acquired in a 2D object detection network and bounding box coordinates and class label probabilities are learned simultaneously. The 2D object detection network operates by treating object detection as a regression problem. Among them, the region in the 2D target detection segment where multiple images are generated; The regions are sorted according to the probability of including 3D targets in the region, and the top N regions with the highest confidence scores are retained. The three parallel branches of the instance segmentation network are used to send N scoring regions with the highest confidence, wherein the three parallel branches include label prediction, BB prediction and mask prediction. Calculate a binary mask for each of the N highest-confidence scoring regions, and automatically segment and construct a pixel-level mask for each object in the image; and Redundant candidates were removed using confidence scores and non-maximum suppression based on IoU_t.

2. The method according to claim 1, further comprising operating a plurality of laser imaging detection and ranging LIDAR sensors in the 3D target detection segment to generate the outputs of the plurality of sensors, thereby further generating a 3D point cloud to capture a set of 3D structures in a vehicle-visible scene.

3. The method of claim 2, further comprising operating the RPN to allocate data of the plurality of point clouds to individual points in the point cloud in a 3D point cloud segmentation component, and assigning labels representing real-world entities.

4. The method of claim 3 further comprises transmitting the product of the RPN to a region-based convolutional neural network RCNN.

5. The method according to claim 4, further comprising: Apply a 3D box estimator to generate one or more bounding boxes (BBs); as well as The output of the 3D box estimator is fused with the 2D target output of the 2D target detector and then passed to the box consistency and filtering unit.

6. The method of claim 1 further includes enhancing 2D detection by combining one-stage 2D object detection and two-stage instance segmentation.

7. The method according to claim 1, further comprising: The image output from the camera is passed to an instance segmentation deep neural network (DNN), which has an instance segmentation device, wherein different instances of the target receive different labels; and The instance output of the instance segmentation device is moved to the instance mask detector, wherein the output of the segmentation device is the binary mask of the region.

8. The method according to claim 7, further comprising: The 2D data of the 2D target detection segment, which defines the image data of the camera, is transmitted to the 2D target detector; The output of the 2D target detector and the output of the instance mask detector are transmitted together to the constraint device; as well as The constraint device and the output of the DNN are sent to the enhanced 2D detector.

Images