A Weakly Supervised Multimodal Image and Point Cloud Instance Segmentation Method for Autonomous Driving

Abstract

This invention discloses a weakly supervised multimodal image and point cloud instance segmentation method for autonomous driving. The method involves processing point cloud data using 2D bounding box labels from image data to obtain coarse pseudo-labeled point cloud data; processing this coarse pseudo-labeled data using a pseudo-label generator to obtain trained pseudo-labeled point cloud data; constructing a multimodal network using BoxInst and FSD Segmentor; feeding the training dataset into the multimodal network for forward propagation to obtain pseudo-labeled instance segmentation results from the multimodal data during forward propagation; determining the self-supervised loss function of the image network branch and the cross-supervised loss function of the multimodal network based on the pseudo-labeled instance segmentation results to determine the final loss function; backpropagating the parameters of the neural network using the final loss function to obtain a trained multimodal neural network; and using the trained multimodal neural network to perform instance segmentation prediction on the prediction dataset to determine the instance targets in the image and point cloud.

Description

Technical Field

[0001] This invention belongs to the fields of autonomous driving and deep learning, specifically relating to a weakly supervised method for segmenting multimodal images and point cloud instances for autonomous driving. Background Technology

[0002] The application of autonomous driving technology in the market has been expanding in recent years, thanks to the continuous reduction in the cost of sensors such as cameras, millimeter-wave radar, and lidar, as well as the rapid development of deep learning and related hardware computing power. Autonomous driving tasks can be subdivided into several sub-tasks, including object detection, instance segmentation, object tracking, and decision-making. Instance segmentation plays a crucial role in these tasks, as it not only perceives information about the space around the vehicle but also provides accurate shape perception; this provides fundamental information for subsequent tracking and decision-making, making it essential.

[0003] Among many sensors, lidar can acquire point cloud data, which has many advantages such as high resolution, accurate recognition, fast measurement speed and strong anti-interference ability. However, it lacks precise semantic information, and image data can make up for these shortcomings. Therefore, multimodal research on lidar and images has received widespread attention.

[0004] Instance segmentation aims to assign each pixel in an image or point cloud to a corresponding object instance and provide precise boundaries for each object instance. Unlike object detection, instance segmentation not only requires identifying different objects in an image but also performing pixel-level segmentation of each object, i.e., assigning a label to each pixel or point to identify which object instance it belongs to. A common approach in instance segmentation is to generate a binary mask where each pixel or point either belongs to an object instance or to the background. This mask can be obtained using deep learning techniques and convolutional neural networks for pixel-level prediction. Typically, this task requires training a model using labeled training data to learn the features and boundary information of object instances.

[0005] However, 2D and 3D mask annotation requires a lot of manpower and financial resources to annotate images and point clouds. Compared with instance mask and 3D bounding box annotation, 2D bounding box annotation is simpler and more cost-effective than instance mask annotation. Although this direction has important practical value, research on multimodal weak supervision methods is still relatively limited. On the one hand, there are inherent differences between point clouds and images. Point clouds mainly focus on geometric information, while images contain semantic and texture information.

[0006] Therefore, existing 2D weakly supervised methods are difficult to transfer to 3D instance segmentation tasks. On the other hand, point clouds lack suitable weakly supervised labels to distinguish instances; although 3D bounding box annotations can be used to train neural networks, the annotation burden is also expensive and exploratory. Therefore, it is necessary to design a weakly supervised multimodal instance segmentation algorithm that can improve the accuracy of both 2D and 3D instance segmentation. Summary of the Invention

[0007] In view of this, the main objective of the present invention is to provide a method for image and point cloud instance segmentation for autonomous driving based on weak supervision.

[0008] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0009] This invention provides a method for image and point cloud instance segmentation for weakly supervised autonomous driving. The method is as follows:

[0010] Point cloud data is processed by 2D bounding box labels from image data to obtain coarse point cloud pseudo-label data.

[0011] The coarse point cloud pseudo-label data is processed by a pseudo-label generator to obtain trained point cloud pseudo-label data.

[0012] A multimodal network is constructed using BoxInst and FSD Segmentor;

[0013] The training dataset is fed into a multimodal network for forward propagation to obtain the pseudo-label instance segmentation results of the multimodal data during forward propagation.

[0014] The self-supervised loss function of the image network branch and the cross-supervised loss function of the multimodal network are determined based on the pseudo-label instance segmentation results.

[0015] The final loss function is obtained by weighted summation of the loss functions of the BoxInst network supervised by real labels, the FSD Segmentor network supervised by pseudo-labels, the self-supervised loss function, and the cross-supervised loss function.

[0016] The parameters of the neural network are backpropagated using the final loss function to obtain a trained multimodal neural network.

[0017] The trained multimodal neural network is used to perform instance segmentation prediction on the prediction dataset to determine the instance targets of images and point clouds.

[0018] In the above scheme, the step of processing point cloud data through 2D bounding box labels of image data to obtain coarse point cloud pseudo-label data specifically involves: given a LiDAR point cloud... N in C represents the number of points in the input point cloud.in The input point cloud represents the feature dimensions; the 3D point cloud is projected onto the image P using the sensor calibration matrices of the LiDAR and camera. 2d =Proj(P 3d )=M×T (l→c) P 3d The mapping relationship between image pixels and point clouds is obtained; based on the mapping relationship, point clouds not on the image are removed, and point clouds falling within the 2D bounding box are used as foreground points, and point clouds outside the 2D bounding box are used as background points, resulting in coarse pseudo-point cloud label data.

[0019] In the above scheme, the step of processing the coarse point cloud pseudo-label data through a pseudo-label generator to obtain trained point cloud pseudo-label data specifically involves: processing the LiDAR point cloud data... Clustering is performed on the depth distance dist of each row m, and each point cloud obtains a unique cluster label. The pseudo-label generator uses R to process coarse pseudo-point cloud label data. Then, the spatial distances of the points are clustered again to obtain the training point cloud pseudo-label data.

[0020] In the above scheme, the construction of a multimodal network through BoxInst and FSD Segmentor specifically involves constructing a multimodal neural network using a BoxInst neural network, an FSD Segmentor neural network, a pseudo-label generator, a dynamic pseudo-mask label generation module, a noise reduction module, and a multimodal cross-supervision module.

[0021] In the above scheme, the step of feeding the training dataset into a multimodal network for forward propagation and obtaining the pseudo-label instance segmentation result of the multimodal data in the forward propagation is specifically as follows: obtaining the prediction result of the BoxInst network through the dynamic pseudo-mask label generation module, and obtaining the network prediction result through the exponential moving average (EMA) FSD Segmentor network.

[0022] In the above scheme, determining the self-supervised loss function of the image network branch and the cross-supervised loss function of the multimodal network based on the pseudo-label instance segmentation results specifically involves: determining the self-supervised loss function L of the BoxInst network based on the pseudo-label instance segmentation results. pseudo (BCELoss and DiceLoss) and multimodal cross-supervision loss function L CSCS ( and ).

[0023] In the above scheme, the BoxInst network loss function supervised by the real labels and the pseudo labels... and The weighted sum of the supervised FSD Segmentor network loss function, self-supervised loss function, and cross-supervised loss function is used as the final loss function. Specifically, the BoxInst network loss function L is determined by the ground truth 2D bounding boxes. boxinst (FocalLoss, GIoULoss, CrossEntropyLoss, DiceLoss, and PairwiseLoss) are used to determine the loss function L of the FSD Segmentor network through pseudo-labels. FSD (FocalLoss and L1Loss), self-supervised loss function L pseudo (BCELoss and DiceLoss), cross-supervised loss function L CSCS ( and The final loss function is obtained by weighted summation of the aforementioned loss functions.

[0024] In the above scheme, the step of backpropagating the parameters of the neural network through the final loss function to obtain the trained multimodal neural network specifically involves defining the maximum training period E. max If the batch size is B and the total number of training samples is D, then the number of batches trained in each cycle is _____. Optimize the FSD Segmentor network using the AdamW optimizer and set the initial learning rate. The learning rate strategy involves single-cycle cosine annealing; the BoxInst network is optimized using the SGD optimizer, and the learning rate is set accordingly. Read in a batch of training data {P, I}, input it into the network for forward propagation, calculate the loss function, and update the network parameters using gradient backpropagation; determine if the total training steps step is divisible by the number of batches k trained in one epoch; if yes, start reading training data from the beginning next time; if no, read training data {P, I} sequentially next time; determine if the total training steps step is equal to B. n If ×k is true, then training ends and a well-trained multimodal neural network is obtained; otherwise, training of the multimodal neural network continues.

[0025] In the above scheme, the step of determining the instance targets of images and point clouds by performing instance segmentation prediction on the prediction dataset using a trained multimodal neural network specifically involves: obtaining the instance mask probability map and confidence score of the BoxInst network; sequentially filtering high-quality instance mask probability maps based on the confidence score; generating the final instance mask using a mask probability threshold to determine the image instance targets; obtaining the semantic label prediction value and offset position of each point through the FSD Segmentor network; and performing spatial distance clustering on all points with the same semantic prediction value to obtain the final point cloud instance targets.

[0026] Compared with existing technologies, this invention combines the capabilities of both image and point cloud modalities for cross-supervision, thereby improving the instance segmentation performance of BoxInst and FSD Segmentor. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0028] Figure 1 This invention provides a flowchart for a method of segmenting images and point cloud instances for autonomous driving based on weak supervision.

[0029] Figure 2 This invention provides a structural diagram of a pseudo-label generator for a weakly supervised autonomous driving image and point cloud instance segmentation method.

[0030] Figure 3 This invention provides a structural diagram of a multimodal network in an image and point cloud instance segmentation method for weakly supervised autonomous driving, as provided in this embodiment of the invention.

[0031] Figure 4 This invention provides a schematic diagram of the structure of an anti-noise module in an image and point cloud instance segmentation method for weakly supervised autonomous driving, as provided in an embodiment of the present invention.

[0032] Figure 5 This invention provides a schematic diagram of the cross-supervision module in an image and point cloud instance segmentation method for weakly supervised autonomous driving, as provided in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0034] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, article, or apparatus that includes that element.

[0035] This invention provides a method for segmenting multimodal images and point cloud instances for autonomous driving based on weak supervision, such as... Figure 1 As shown, the method is as follows:

[0036] Step 101: Process the point cloud data using 2D bounding box labels from the image data to obtain coarse point cloud pseudo-label data.

[0037] Specifically, given a lidar point cloud N in C represents the number of points in the input point cloud. in This represents the feature dimensions of the input point cloud. The 3D point cloud is projected onto the image P using the sensor calibration matrices of the LiDAR and camera. 2d =Proj(P 3d )=M×T (l→c) P 3d This allows us to obtain the mapping relationship between image pixels and point clouds.

[0038] Point clouds not in the image are removed. Point clouds falling within the 2D bounding box are used as foreground points, and point clouds outside the 2D bounding box are used as background points, resulting in coarse pseudo-point cloud label data.

[0039] For example, suppose the input is the coordinates of a point cloud frame. The camera extrinsic matrix is The camera intrinsic parameter matrix is Obtain the coordinates of the point cloud projected onto the image. The formula is as follows (1). Remove the point clouds that do not fall on the image, and then determine the coordinates P. 2d With 2D bounding box annotation (x min y min x max y max The positional relationship of x, if x min ≤u≤x max And y min ≤v≤y max If the current point is a foreground point (instance point), then it belongs to the foreground point; otherwise, it belongs to the background point, resulting in coarse pseudo-labeled point cloud data.

[0040]

[0041] Step 102: Process the coarse point cloud pseudo-label data using a pseudo-label generator to obtain the training point cloud pseudo-label data.

[0042] Specifically, such as Figure 2 As shown, for a lidar with m beams and n measurements in one scanning cycle, the return values ​​of one scan form an m×n matrix.

[0043] By analyzing lidar point cloud data Clustering is performed on the depth distance dist of each row m, and each point cloud obtains a unique cluster label.

[0044] The pseudo-label generator uses clustering label R to process coarse pseudo-point cloud label data. Then, the spatial distances of the points are clustered again to obtain the training point cloud pseudo-label data.

[0045] For example, in a point cloud P, the depth distance dist is used for clustering the points in the m-th row. The depth of the n-th point in the m-th row is d. (m，n) The depth of the (n-1)th point in the m-th row is d. (m，n-1) The cluster label is r; the distance threshold is d. threshold =0.24m. If |d (m，n) -d (m，n-1) |≤d threshold If the two points have the same cluster label r, then they have the same cluster label r; otherwise, the current point p... (m，n) The cluster label is r+1; iterate through all rows to obtain the final cluster label R.

[0046] Use cluster labels R to pair The filtering process involves taking a cluster label r and then identifying the foreground points (instances) under that label r. and background points The quantity ratio meets Then these points P r If it is a foreground point, then it is a background point; otherwise, it is a background point.

[0047] For the filtered foreground point P fg For CCL spatial clustering, if the Euclidean distance between two point clouds is ||P (3d，i) -P (3d，j) || Less than the distance threshold d ccl If they are connected, then they are connected.

[0048] In this graph, the largest connected component is considered a foreground point, and the remaining points are considered background points. This yields the final point cloud pseudo-label data.

[0049] In some embodiments, most current multimodal algorithms are implemented based on the Waymo dataset. Therefore, this invention uses the Waymo dataset as the dataset for weakly supervised multimodal instance segmentation. The Waymo dataset is a publicly available dataset collected and annotated by Waymo, the self-driving car company under Google's parent company, Alphabet. It covers a wide variety of environments, from dense urban centers to suburban landscapes, and data collected during day and night, dawn and dusk, and sunny and rainy days. The sensors used include five LiDARs and five RGB cameras, containing LiDAR and camera data from 1000 segments (each data segment is 20 seconds in size), which are widely used in tasks such as image segmentation, object detection, and object tracking.

[0050] The initial labels of the 2D bounding boxes include four categories: vehicles, pedestrians, cyclists, and signs, and each category has a unique tracking ID and a difficult label to annotate.

[0051] Due to the differences in annotation between the Waymo point cloud segmentation and panoptic segmentation datasets, instance segmentation results are trained and evaluated only on the panoptic segmentation data, while they are trained and evaluated on the point cloud segmentation data. Specifically, for the 2D instance segmentation task, the training set contains 61,480 images, and the validation set contains 9,405 images; for the 3D segmentation task, the dataset contains 23,691 frames and 5,976 frames, used for training and validation, respectively. This invention evaluates three categories: vehicles, pedestrians, and cyclists.

[0052] Set the batch size to 8, read in the training data {P, I}, and set the detection space in the Waymo LiDAR coordinate system to [-80, 80) meters in the X direction, [-80, 80) meters in the Y direction, and [-2, 4] meters in the Z direction. The feature dimension of the point cloud data used is C. in =7, including the point cloud's X-dimensional coordinates, Y-dimensional coordinates, Z-dimensional coordinates, signal reflection intensity, depth distance (dist), the m-th beam, and the n-th measurement; each frame of the point cloud corresponds to five camera images. Therefore, the actual image batch size is 40.

[0053] Step 103: Construct a multimodal network using BoxInst and FSD Segmentor;

[0054] Specifically, such as Figure 3 As shown, a multimodal neural network is constructed using a BoxInst neural network, an FSD Segmentor neural network, a pseudo-label generator, a dynamic pseudo-mask label generation module, a noise reduction module, and a multimodal cross-supervision module.

[0055] The image and point cloud neural networks use BoxInst and FSD Segmentor respectively, without changing any network structure or parameters.

[0056] Build a pseudo-tag generator, such as Figure 2 As shown, deep clustering d is set threshold =0.24m and CCL clustering threshold d ccl d for different types of instances ccl different;

[0057] For example, vehicle category d ccl =0.6m, pedestrian category d ccl =0.1m, Cyclist category d ccl =0.15m.

[0058] A dynamic pseudo-mask label generation module is constructed to calculate the Interchange of U (IoU) between the predicted 2D bounding boxes and the ground truth 2D bounding boxes in BoxInst. Then, the IoU and predicted box scores are used to weight the matching instance mask prediction results. The formula is defined as follows:

[0059]

[0060] in M represents the predicted probability map corresponding to the i-th truth box. i，j Let represent the j-th instance corresponding to the i-th baseline truth box, k represent the centrality weight, and s i，j This represents the confidence score of the predicted bounding box corresponding to the i-th instance. After obtaining the weighted probability map, we set two thresholds τ. low and τ high The dynamic pseudomask corresponding to the i-th instance is obtained.

[0061] For example, given a vehicle object O with a corresponding 2D bounding box truth value B, BoxInst predicts N instance masks M. i，j Assign the N instance masks to the vehicle object O, set k=1, and calculate the weights w of these N instance masks to the vehicle object O. i，j Then, a weighted summation is performed to obtain the predicted probability map of vehicle object O. Then take τ low =0.3, τ high =0.7, Greater than τ high Binarization to 1, less than τ low Binarize the result to 0, ignore the remaining part, and obtain the dynamic pseudomask.

[0062] Construct noise-resistant modules, such as Figure 4 As shown, create an EMA historical prediction result matrix H with a dimension of N.frame ×N his ×N points When the number of training iterations reaches the set EpochE g At that time, the current input point cloud is used to query the corresponding historical prediction results in H, and a vote is taken to obtain the voting results. The voting results are then used to assign labels to the pseudo-label generator. The final noise-resistant pseudo-tags are obtained after modification.

[0063] For example, set E g =12, N his =4, the current training epoch is 13, for the current input point cloud pseudo-label data Voting is performed on the historical prediction results from the first four predictions (i.e., 9, 10, 11, 12). If the most frequent category appears at least three times, then that category is chosen; otherwise, it is ignored and classified as category -1. The voting results are then used to further refine the predictions. After modification, the final noise-resistant pseudo-label is obtained.

[0064] Construct a multimodal cross-supervision module, such as Figure 5 As shown, the predicted probability map M obtained using the dynamic pseudo-mask label generation module ema Supervise the instance mask predicted by the FSD Segmentor; use the instance pseudo-mask predicted by the 3D EMA to supervise the instance mask predicted by BoxInst.

[0065] Step 104: Feed the training dataset into the multimodal network for forward propagation, and obtain the pseudo-label instance segmentation results and 2D instance masks of the multimodal data during forward propagation. 2D prediction probability map M ema and 3D instance mask

[0066] Step 105: Determine the self-supervised loss function of the image network branch and the cross-supervised loss function of the multimodal network based on the pseudo-label instance segmentation results. The final loss function is obtained by weighted summation of the BoxInst network loss function supervised by real labels, the FSD Segmentor network loss function supervised by pseudo labels, the self-supervised loss function, and the cross-supervised loss function.

[0067] Specifically: The loss function L of the BoxInst network is determined using the real label 2D bounding boxes. boxinst (FocalLoss, GIoULoss, CrossEntropyLoss, DiceLoss, and PairwiseLoss) are used to determine the loss function L of the FSD Segmentor network through pseudo-labels. FSD(FocalLoss and L1Loss), self-supervised loss function L pseudo (BCELoss and DiceLoss), cross-supervised loss function L CSCS ( and The final loss function is obtained by weighted summation of the aforementioned loss functions.

[0068] For example, the instance mask probability map predicted by the BoxInst network. Dynamic pseudomask 2D instance mask Dynamic 2D prediction probability map M ema The instance mask predicted by the 3D network is 3D EMA prediction instance mask is

[0069] Define the self-supervised loss function as:

[0070]

[0071] The cross-supervision loss function is defined as:

[0072]

[0073]

[0074]

[0075] BoxInst network loss function L boxinst And the loss function L of the FSD Segmentor network FSD Remain unchanged;

[0076] The final weighted sum of the loss functions yields:

[0077] L total =L 2d +L 3d (7)

[0078]

[0079]

[0080] The values ​​of α1 to α6 are set to 1.0, 1.0, 0.5, 100.0, 1.0, and 2.0, respectively, to balance the loss term.

[0081] Step 106: Backpropagate the parameters of the neural network using the final loss function to obtain the trained multimodal neural network;

[0082] Specifically, the maximum training period E is defined. maxIf the batch size is B and the total number of training samples is D, then the number of batches trained in each cycle is _____. Optimize the FSD Segmentor network using the AdamW optimizer and set the initial learning rate. The learning rate strategy involves single-cycle cosine annealing; the BoxInst network is optimized using the SGD optimizer, and the learning rate is set accordingly. Read in a batch of training data P{P, I}, input it into the network for forward propagation, calculate the loss function, and update the network parameters using gradient backpropagation; determine if the total training steps step is divisible by the number of batches k trained in one epoch; if yes, start reading training data from the beginning next time; if no, read training data {P, I} sequentially next time; determine if the total training steps step is equal to B. n If ×k is true, then training ends and a well-trained multimodal neural network is obtained; otherwise, training of the multimodal neural network continues.

[0083] Step 107: Use the trained multimodal neural network to perform instance segmentation prediction on the prediction dataset to determine the instance targets of the image and point cloud.

[0084] Specifically, for the BoxInst neural network, the predicted 2D bounding box is first obtained, and then the corresponding mask probability map is obtained. The probability threshold is set to 0.5. If it is greater than the threshold, it is 1; otherwise, it is 0, and the corresponding 2D instance mask prediction result is obtained. For the FSD Segmentor, the maximum predicted probability of each point is used as the category of the current point. Then, each point is clustered by adding the offset distance to obtain the final 3D instance target.

[0085] This invention introduces 2D bounding boxes into point cloud segmentation tasks, mines the depth and geometric prior information of point clouds, and uses the prior information to continuously refine the labels of point clouds within the 2D bounding boxes, thereby reducing the labeling burden.

[0086] The dynamic pseudo-mask label generation module and noise reduction module proposed in this invention introduce current prediction information and historical prediction information to correct the labels, thereby improving the noise resistance of the neural network.

[0087] The multimodal cross-supervision module proposed in this invention realizes independent training and joint optimization of multimodal networks, further improving the segmentation performance of each modality network.

[0088] The solution was finally tested on the Waymo validation dataset. The AP (Average Per Second) metrics for 2D instance segmentation were 49.72%, 30.18%, and 33.32% for vehicles, pedestrians, and cyclists, respectively; the AP metrics for 3D instance segmentation were 62.09%, 48.86%, and 36.27% for vehicles, pedestrians, and cyclists, respectively; and the IoU (Interchange of Usage) metrics for 3D semantic segmentation were 89.94%, 82.31%, and 64.12% for vehicles, pedestrians, and cyclists, respectively.

[0089] Experimental data:

[0090] Determine the validation dataset and evaluation metrics;

[0091] This invention uses the validation set from the Waymo dataset as validation data, and employs COCO's official image instance segmentation evaluation metric, Average Precision (AP), and the standard point cloud semantic segmentation metric, Interchange of Units (IoU). AP and IoU represent the detection accuracy for a specific class and are used to evaluate the model's detection performance for a single class. Higher AP and IoU values ​​indicate higher detection accuracy and greater practical value.

[0092] The AP metric is defined as follows:

[0093]

[0094] Where P is precision and R is recall, TP represents positive samples predicted as positive, FP represents positive samples predicted as negative, and FN represents negative samples predicted as negative, the mathematical definitions of precision and recall are as follows:

[0095]

[0096]

[0097] For each sample, its Precision and Recall can be obtained. By arranging the results of multiple samples in descending order of confidence and plotting them as a Precision-Recall curve, the AP result can be calculated. To reduce the computational load, the interval [0, 1] can be divided into L-1 equal parts using L equal division points, and the AP value can be calculated in a manner similar to infinitesimal elements. We chose L=11 to calculate the final AP index.

[0098] Specifically, in COCO, AP50 and AP75 refer to the condition that when the IoU is greater than 0.5 or 0.75, it is judged as TP; AP refers to [0.50:0.05:0.95], that is, the IoU threshold is set to 0.5, 0.55, 0.60, 0.65...0.95, ten APs are calculated, and then the average is obtained.

[0099] For each category, the IoU metric is defined as follows:

[0100]

[0101] To illustrate the practicality of this invention, three comparative experiments were conducted for 2D instance segmentation. Experiment A used the CondInst algorithm for training and detection; Experiment B used the BoxInst algorithm for training and detection; and Experiment C used the algorithm described above for training and detection. All experiments were performed on the same machine, with a training duration of 24 epochs. The detection targets were selected from three categories: vehicles, pedestrians, and cyclists. The results are shown in Table 1.

[0102] Table 1: Results of the comparative experiment

[0103]

[0104] Compared with experimental groups B and C, which used 2D Box supervision, experimental group C showed a 3.13% increase in mAP, a 1.24% increase in AP for vehicles, a 2.45% increase for pedestrians, and a 5.63% increase for cyclists. Compared with A, the weakly supervised group C achieved 96.57% of the fully supervised group's mAp under AP50 conditions, 95.10% for vehicles, 115.36% for pedestrians (an increase of 9.07%), and 106.18% for cyclists (an increase of 3.1%).

[0105] Three comparative experiments were conducted for 3D instance segmentation and semantic segmentation. Experiment A used the fully supervised FSDSegmentor algorithm for training and detection; Experiment B used the 3D bounding box-supervised FSD Segmentor algorithm for training and detection; Experiment C used the aforementioned algorithm for training and detection; and Experiment D used the aforementioned algorithm for training and detection on the entire Waymo training set. All experiments were conducted on the same machine, with a training duration of 24 epochs. The detection objects were selected from three categories: vehicles, pedestrians, and cyclists. The results are shown in Table 2.

[0106] Table 2: Results of the comparative experiment

[0107]

[0108] Compared to experimental group B, experimental group D showed a 0.78% improvement in instance segmentation mAP and a 6.09% improvement in semantic segmentation mIoU. Compared to experimental group D and fully supervised experimental group B, cyclist instance segmentation and semantic segmentation improved by 0.98% and 2.61%, respectively. Compared to experimental group C, experimental group D showed improvements of 3.31% in mAP and 2.49% in mIoU.

[0109] The above results demonstrate that experimental groups C in Table 1 and C and D in Table 2, compared with the more label-heavy instance masking and 3D bounding box supervision algorithms, achieve comparable or higher performance on small targets (pedestrians and cyclists). This proves the advantages of the present invention under weak label (2D bounding box) supervision.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A weakly supervised based automatic driving multi-modal image and point cloud instance segmentation method, characterized in that, The method is as follows: Point cloud data is processed by 2D bounding box labels from image data to obtain coarse point cloud pseudo-label data. The coarse point cloud pseudo-label data is processed by a pseudo-label generator to obtain trained point cloud pseudo-label data. A multimodal network is constructed using BoxInst and FSD Segmentor; The training dataset is fed into a multimodal network for forward propagation to obtain the pseudo-label instance segmentation results of the multimodal data during forward propagation. The self-supervised loss function of the image network branch and the cross-supervised loss function of the multimodal network are determined based on the pseudo-label instance segmentation results. The final loss function is obtained by weighted summation of the loss functions of the BoxInst network supervised by real labels, the FSD Segmentor network supervised by pseudo-labels, the self-supervised loss function, and the cross-supervised loss function. The parameters of the neural network are backpropagated using the final loss function to obtain a trained multimodal neural network. The trained multimodal neural network is used to perform instance segmentation prediction on the prediction dataset to determine the instance targets of images and point clouds. The 2D box label processing point cloud data through the picture data is used to obtain rough point cloud pseudo-label data, specifically: given the laser radar point cloud , represents the number of input point clouds, represents the feature dimension of the input point cloud; the 3D point cloud is projected to the picture by the sensor calibration matrix of the laser radar and the camera, and the mapping relationship between the picture pixels and the point cloud is obtained; According to the mapping relationship, point clouds not on the picture are removed, point clouds falling into the 2D frame are regarded as foreground points, and point clouds outside the 2D frame are regarded as background points, to obtain rough pseudo point cloud label data ; The process of processing coarse point cloud pseudo-label data using a pseudo-label generator to obtain trained point cloud pseudo-label data specifically involves: processing the LiDAR point cloud data... Each line depth distance Clustering is performed, and each point cloud element obtains a unique cluster label. Pseudo-tag generator uses Processing coarse pseudo-point cloud label data Then, the spatial distances of the points are clustered again to obtain the training point cloud pseudo-label data. .

2. The method for segmenting multimodal images and point cloud instances for autonomous driving based on weak supervision according to claim 1, characterized in that, The construction of a multimodal network using BoxInst and FSD Segmentor specifically involves constructing a multimodal neural network using a BoxInst neural network, an FSD Segmentor neural network, a pseudo-label generator, a dynamic pseudo-mask label generation module, a noise reduction module, and a multimodal cross-supervision module.

3. The method for segmenting multimodal images and point cloud instances for autonomous driving based on weak supervision according to claim 2, characterized in that, The step of feeding the training dataset into a multimodal network for forward propagation and obtaining the pseudo-label instance segmentation results of the multimodal data during forward propagation is specifically as follows: obtaining the prediction results of the BoxInst network through the dynamic pseudo-mask label generation module, and obtaining the network prediction results through the exponential moving average (EMA) FSD Segmentor network.

4. The method for segmenting multimodal images and point cloud instances for autonomous driving based on weak supervision according to claim 3, characterized in that, The process of determining the self-supervised loss function of the image network branch and the cross-supervised loss function of the multimodal network based on the pseudo-labeled instance segmentation results specifically involves: determining the self-supervised loss function of the BoxInst network based on the pseudo-labeled instance segmentation results. (BCELoss and DiceLoss) and multimodal cross-supervision loss function ( and ).

5. The method for segmenting multimodal images and point cloud instances for autonomous driving based on weak supervision according to claim 4, characterized in that, The BoxInst network loss function, supervised by real labels, and pseudo labels are described. and The weighted sum of the supervised FSD Segmentor network loss function, self-supervised loss function, and cross-supervised loss function is used as the final loss function. Specifically, the BoxInst network loss function is determined using the ground truth 2D bounding boxes. (FocalLoss, GIoULoss, CrossEntropyLoss, DiceLoss, and PairwiseLoss) The loss function of the FSDSegmentor network is determined using pseudo-labels. (FocalLoss and L1Loss), self-supervised loss functions (BCELoss and DiceLoss), cross-supervised loss functions ( and The final loss function is obtained by weighted summation of the aforementioned loss functions.

6. The method for segmenting multimodal images and point cloud instances for autonomous driving based on weak supervision according to claim 5, characterized in that, The process of backpropagating the parameters of the neural network using the final loss function to obtain a trained multimodal neural network specifically involves defining the maximum training period. Training batch size If the total number of training samples is D, then the number of batches trained in each cycle is... ; Optimize the FSD Segmentor network using the AdamW optimizer and set the initial learning rate. The learning rate strategy involves single-cycle cosine annealing; the BoxInst network is optimized using the SGD optimizer, and the learning rate is set accordingly. Read in a batch of training data Input the network and perform forward propagation to calculate the loss function; then perform gradient backpropagation to update the network parameters; finally, determine the total number of training steps. Can it be trained on a batch within one cycle? Divisible by integer; if true, the training data is read from the beginning next time; if false, the training data is read sequentially next time. Determine the total number of training steps Is it equal to If the condition is met, training ends and a well-trained multimodal neural network is obtained; otherwise, training of the multimodal neural network continues.

7. The method for segmenting multimodal images and point cloud instances for autonomous driving based on weak supervision according to claim 6, characterized in that, The process of determining the instance targets of images and point clouds by performing instance segmentation prediction on the prediction dataset using a trained multimodal neural network involves: obtaining the instance mask probability map and confidence score of the BoxInst network; sequentially filtering high-quality instance mask probability maps based on the confidence score; generating the final instance mask using a mask probability threshold to determine the image instance targets; obtaining the semantic label prediction value and offset position of each point through the FSD Segmentor network; and performing spatial distance clustering on all points with the same semantic prediction value to obtain the final point cloud instance targets.

Images