Multimodal fusion-based unsupervised 3D bounding box annotation error detection method and system

Abstract

The application provides a multi-modal fusion unsupervised 3D box label error detection method and system, comprising the following steps: S1, data features are extracted by using a two-dimensional image segmentation method and a three-dimensional laser radar segmentation method to obtain semantic information; S2, the features extracted from the data of different modes are fused; S3, the segmentation result is calculated according to the fused features; and S4, the label error score of the 3D box is calculated according to the segmentation result of the points in the 3D box. The application focuses on 3D box label error detection. 3DboxAD is an unsupervised label error detection method, which does not need to label whether the 3D box is wrong, and also does not need any prior knowledge. 3DboxAD firstly realizes the detection of 3D box label error, and through the fusion of the semantic information of two-dimensional image data and three-dimensional laser radar point cloud data, the detection of 3D box label error is completed together.

Description

Technical Field

[0001] This invention relates to the field of 3D bounding box labeling error detection, specifically to a multimodal fusion-based unsupervised 3D bounding box labeling error detection method and system. Background Technology

[0002] With the rapid development of autonomous driving technology, 3D object detection models play a crucial role in many applications. The training phase of these models relies on accurate and high-quality 3D bounding box annotation data. This annotation data can be provided manually or obtained through semi-automatic or automatic methods (such as automatic 3D bounding box annotation techniques). However, these 3D bounding box annotations are not entirely reliable and may contain various errors. In the field of autonomous driving, accurate 3D object detection of the scene surrounding the vehicle is critical, as it directly affects driving safety. If erroneous 3D bounding boxes are used as ground truth labels during training, it will severely impact model performance, potentially leading to missed detections and false positives in object detection tasks. Therefore, we need a method to detect and eliminate these erroneous 3D bounding box annotations.

[0003] However, multimodal 3D bounding box annotation error detection is a challenging task for two reasons. First, 3D bounding boxes can contain various types of errors. For example, it might incorrectly label certain objects, such as misidentifying a car as a bus, or incorrectly outline objects that shouldn't be considered, such as non-target objects in the background environment. Second, as... Figure 2 As shown, data acquisition vehicles are typically equipped with multiple sensors, such as cameras and LiDAR, which generate multimodal data at different time stamps. Effectively fusing this multimodal data for 3D bounding box annotation error detection is a challenging task. This requires a thorough understanding and utilization of the characteristics of different modalities to design effective feature fusion strategies to improve the accuracy of error detection.

[0004] Patent document CN116630937A discloses a 3D target detection method based on laser point clouds, specifically disclosing the construction of an overall network. This overall network includes a laser radar feature extraction branch, a 4D millimeter-wave feature extraction branch, a feature fusion module, a 2D backbone network, and an RPN detection head. Real-world data from autonomous driving is then acquired to train the overall network. Finally, the currently acquired laser radar point cloud data and 4D millimeter-wave point cloud data are input into the trained overall network, and the inspection results are output. However, this invention does not improve the accuracy of error detection. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a multimodal fusion-based unsupervised 3D bounding box annotation error detection method and system.

[0006] The present invention provides a multimodal fusion-based unsupervised 3D bounding box annotation error detection method, comprising:

[0007] Step S1: Extract data features using two-dimensional image segmentation and three-dimensional LiDAR segmentation methods to obtain semantic information;

[0008] Step S2: Fuse the features extracted from data from different modalities;

[0009] Step S3: Calculate the segmentation result based on the fusion features;

[0010] Step S4: Calculate the annotation error score of the 3D bounding box based on the segmentation results of the points in the 3D bounding box.

[0011] Preferably, in step S1:

[0012] The multimodal feature extraction module includes an image feature extractor and a 3D point cloud feature extractor, which extract image features and 3D point cloud features.

[0013] Image feature extractor:

[0014] A semantic segmentation network is used to generate pixel semantic features. The segmentation network takes multi-view images as input and represents them as follows: Where w and h are the width and length of the input image, respectively, and the output of the image feature extractor is denoted as... Where w′ and h′ are proportional scalings of the width and length of the input image, respectively. f 2D The input image size of the feature extractor is an integer multiple of the output size, representing the feature dimension of the output.

[0015] 3D point cloud feature extractor:

[0016] A 3D segmentation network is used to obtain the features of each point. The LiDAR point cloud is used as the input to the 3D point cloud feature extractor, and the output is... Where N is the number of points in the lidar point cloud, f 3D The output feature dimension.

[0017] Preferably, in step S2:

[0018] Features from RGB images and LiDAR point clouds are fused together. Semantic information from both RGB images and LiDAR point clouds is used to detect 3D bounding box marking errors. Semantic information from RGB images and LiDAR point clouds is fused through camera calibration. The 3D point cloud is projected from the LiDAR coordinate system to the camera coordinate system using camera calibration. The correspondence between points in the 3D point cloud and image pixels is established. Image features and point cloud features are then fused.

[0019] Establish the relationship between points in the point cloud and pixels in the image. The 2D image and 3D points are linked through projection geometry, projecting each LiDAR point p onto the RGB image:

[0020] [x,y,1] T =K(Rp+t)

[0021] Where (x, y) are the pixel coordinates of the 3D point p projected onto the RGB image, K is the camera's intrinsic calibration matrix, and R and t are the rotation matrix and translation vector that transform the 3D point from the LiDAR coordinate system to the camera coordinate system, resulting in a mapping from the LiDAR point cloud to the RGB image; if the feature map F output by the image feature extractor... 2D The resolution differs from the original image, w′≠w and h′≠h. Update the mapping between points and pixels by dividing the pixel coordinates (x,y) by the scaling factor i. And it is rounded to the nearest integer value;

[0022] Using mapping relationships, image features Convert to When multiple images exist, points in a point cloud are projected onto multiple images, and the features of multiple images are fused by taking the maximum value of each point.

[0023] Image features F 2D With 3D point cloud features F 3D By performing point-by-point connections, the fused features are obtained.

[0024] Preferably, in step S3:

[0025] The features P are fused and input into a fully connected neural network MLP, where each layer is followed by a batch normalization layer and a dropout layer. The activation function used is LeakyReLU. The output of the MLP is then input into a softmax layer to obtain the class probability distribution for each point, denoted as . Where m represents the number of possible categories of a point;

[0026] In step S4:

[0027] The annotation error score of a 3D bounding box is the average of the segmentation errors of all points within the 3D bounding box, where the segmentation error is calculated based on the probability distribution P. j The actual category y of the j-th point is greater than j The probability P j [y j The sum of all probabilities of ] is formalized as follows:

[0028]

[0029] in It is a probability distribution The probability of the i-th category;

[0030] The cross-entropy loss function is:

[0031]

[0032] Where P j [y j ] represents the class probability distribution P of point j. j In the middle, category y j The probability of;

[0033] After the training phase, the points in the 3D bounding box are input into the model. The output P of the Softmax layer is used to determine if the 3D bounding box is valid. For the j-th point in the point cloud, the segmentation error is within the probability distribution P. j The actual category y of the j-th point is greater than j The probability P j [y j The sum of all probabilities of ] is formalized as follows:

[0034]

[0035] in It is a probability distribution The probability of the i-th category in the bounding box; the 3D bounding box labeling error score is the average of the segmentation errors of all points in the 3D bounding box.

[0036] A threshold τ is used to determine the error score of the 3D bounding box annotation, and the 3D bounding box is marked as correct or incorrect.

[0037] Preferably, a three-stage training method is provided, wherein the first stage uses transfer learning to train the multimodal feature extraction module, the second stage trains the point cloud segmentation module separately, and the third stage adjusts the entire model.

[0038] The image feature extractor and the 3D point cloud feature extractor are pre-trained. During the pre-training process, the semantic segmentation dataset with correct data annotation in a scenario with a higher similarity to the target task than a preset standard is used for training. A linear layer and a softmax layer are added after the feature extractor to adjust the output dimension to match the number of categories in the dataset.

[0039] The parameters of the multimodal feature extraction module are fixed, and the point cloud segmentation module is trained. Segmentation labels are generated by 3D bounding boxes. For points inside the 3D bounding boxes, the corresponding category labels are directly assigned. All points outside the 3D bounding boxes are used as background. The Adam optimizer is used for training to adjust the overall model.

[0040] According to the present invention, an unsupervised 3D bounding box annotation error detection system based on multimodal fusion includes:

[0041] Module M1: Extracts data features using two-dimensional image segmentation and three-dimensional LiDAR segmentation methods to obtain semantic information;

[0042] Module M2: fuses features extracted from data from different modalities;

[0043] Module M3: Calculates the segmentation result based on the fusion features;

[0044] Module M4: Calculates the annotation error score of the 3D bounding box based on the segmentation results of the points in the 3D bounding box.

[0045] Preferably, in module M1:

[0046] The multimodal feature extraction module includes an image feature extractor and a 3D point cloud feature extractor, which extract image features and 3D point cloud features.

[0047] Image feature extractor:

[0048] A semantic segmentation network is used to generate pixel semantic features. The segmentation network takes multi-view images as input and represents them as follows: Where w and h are the width and length of the input image, respectively, and the output of the image feature extractor is denoted as... Where w′ and h′ are proportional scalings of the width and length of the input image, respectively. f 2D The input image size of the feature extractor is an integer multiple of the output size, representing the feature dimension of the output.

[0049] 3D point cloud feature extractor:

[0050] A 3D segmentation network is used to obtain the features of each point. The LiDAR point cloud is used as the input to the 3D point cloud feature extractor, and the output is... Where N is the number of points in the lidar point cloud, f 3D The output feature dimension.

[0051] Preferably, in module M2:

[0052] Features from RGB images and LiDAR point clouds are fused together. Semantic information from both RGB images and LiDAR point clouds is used to detect 3D bounding box marking errors. Semantic information from RGB images and LiDAR point clouds is fused through camera calibration. The 3D point cloud is projected from the LiDAR coordinate system to the camera coordinate system using camera calibration. The correspondence between points in the 3D point cloud and image pixels is established. Image features and point cloud features are then fused.

[0053] Establish the relationship between points in the point cloud and pixels in the image. The 2D image and 3D points are linked through projection geometry, projecting each LiDAR point p onto the RGB image:

[0054] [x,y,1] T =K(Rp+t)

[0055] Where (x, y) are the pixel coordinates of the 3D point p projected onto the RGB image, K is the camera's intrinsic calibration matrix, and R and t are the rotation matrix and translation vector that transform the 3D point from the LiDAR coordinate system to the camera coordinate system, resulting in a mapping from the LiDAR point cloud to the RGB image; if the feature map F output by the image feature extractor... 2D The resolution differs from the original image, w′≠w and h′≠h. Update the mapping between points and pixels by dividing the pixel coordinates (x,y) by the scaling factor i. And it is rounded to the nearest integer value;

[0056] Using mapping relationships, image features Convert to When multiple images exist, points in a point cloud are projected onto multiple images, and the features of multiple images are fused by taking the maximum value of each point.

[0057] Image features F 2D With 3D point cloud features F 3D By performing point-by-point connections, the fused features are obtained.

[0058] Preferably, in module M3:

[0059] The features P are fused and input into a fully connected neural network MLP, where each layer is followed by a batch normalization layer and a dropout layer. The activation function used is LeakyReLU. The output of the MLP is then input into a softmax layer to obtain the class probability distribution for each point, denoted as . Where m represents the number of possible categories of a point;

[0060] In module M4:

[0061] The annotation error score of a 3D bounding box is the average of the segmentation errors of all points within the 3D bounding box, where the segmentation error is calculated based on the probability distribution P. j The actual category y of the j-th point is greater than j The probability P j [y j The sum of all probabilities of ] is formalized as follows:

[0062]

[0063] in It is a probability distribution The probability of the i-th category;

[0064] The cross-entropy loss function is:

[0065]

[0066] Where P j [y j ] represents the class probability distribution P of point j. j In the middle, category y j The probability of;

[0067] After the training phase, the points in the 3D bounding box are input into the model. The output P of the Softmax layer is used to determine if the 3D bounding box is valid. For the j-th point in the point cloud, the segmentation error is within the probability distribution P. j The actual category y of the j-th point is greater than j The probability P j [y j The sum of all probabilities of ] is formalized as follows:

[0068]

[0069] in It is a probability distribution The probability of the i-th category in the bounding box; the 3D bounding box labeling error score is the average of the segmentation errors of all points in the 3D bounding box.

[0070] A threshold τ is used to determine the error score of the 3D bounding box annotation, and the 3D bounding box is marked as correct or incorrect.

[0071] Preferably, a three-stage training method is provided, wherein the first stage uses transfer learning to train the multimodal feature extraction module, the second stage trains the point cloud segmentation module separately, and the third stage adjusts the entire model.

[0072] The image feature extractor and the 3D point cloud feature extractor are pre-trained. During the pre-training process, the semantic segmentation dataset with correct data annotation in a scenario with a higher similarity to the target task than a preset standard is used for training. A linear layer and a softmax layer are added after the feature extractor to adjust the output dimension to match the number of categories in the dataset.

[0073] The parameters of the multimodal feature extraction module are fixed, and the point cloud segmentation module is trained. Segmentation labels are generated by 3D bounding boxes. For points inside the 3D bounding boxes, the corresponding category labels are directly assigned. All points outside the 3D bounding boxes are used as background. The Adam optimizer is used for training to adjust the overall model.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] This invention focuses on 3D bounding box annotation error detection. 3DboxAD is an unsupervised annotation error detection method that does not require labeling 3D bounding boxes to determine if they are incorrect, nor does it require any prior knowledge. 3DboxAD is the first method to detect 3D bounding box annotation errors by fusing semantic information from 2D image data and 3D LiDAR point cloud data to jointly complete the detection of 3D bounding box annotation errors. Attached Figure Description

[0076] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0077] Figure 1 This is a schematic diagram of the 3DboxAD framework for annotation error detection according to the present invention;

[0078] Figure 2 This is a schematic diagram of the multimodal data correspondence of the present invention;

[0079] Figure 3 This is a schematic diagram illustrating the correspondence between points in a 3D point cloud and image pixels according to the present invention.

[0080] Figure 4 This is a schematic diagram of the probability distribution and point segmentation error output by the 3DboxAD point cloud segmentation module of the present invention.

[0081] Figure 5 This is a schematic diagram of the three-stage training of the present invention. Detailed Implementation

[0082] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0083] Example 1:

[0084] This invention discloses a multimodal fusion-based unsupervised 3D bounding box annotation error detection method called 3DboxAD, applicable to data preparation in autonomous driving. This method fuses 2D RGB images and 3D point cloud data to jointly detect 3D bounding box annotation errors. Specifically, 3DboxAD mainly comprises four modules: a multimodal feature extraction module, a semantic fusion module, a point cloud segmentation module, and an annotation error score generation module. First, data features are extracted using 2D image segmentation and 3D LiDAR segmentation methods to obtain semantic information. Then, features extracted from different modalities are fused. This step is one of the key steps in 3DboxAD, achieving cross-modal fusion of different data types. By fusing features from RGB images and LiDAR point clouds, we can fully utilize the advantages of different data types, thereby more accurately detecting 3D bounding box annotation errors. Next, the segmentation result is calculated based on the fused features. Finally, based on the segmentation result of the points in the 3D bounding box, we can further calculate the 3D bounding box annotation error score. This score can be used to measure the degree of error in the 3D bounding box, thus helping us better understand the error situation in 3D bounding box annotation. It's worth mentioning that 3DboxAD uses an unsupervised approach to detect annotation errors, meaning it doesn't require any expert knowledge or annotation of whether the 3D boxes are incorrect.

[0085] By fusing multimodal data, we can better detect annotation errors in 3D bounding boxes. This means that if the 3D bounding boxes appear different from the actual situation when viewed from an image perspective or a LiDAR perspective, the model will generate a large annotation error score.

[0086] According to the present invention, a multimodal fusion unsupervised 3D bounding box annotation error detection method is provided, such as... Figures 1-5 As shown, it includes:

[0087] Step S1: Extract data features using two-dimensional image segmentation and three-dimensional LiDAR segmentation methods to obtain semantic information;

[0088] Specifically, in step S1:

[0089] The multimodal feature extraction module includes an image feature extractor and a 3D point cloud feature extractor, which extract image features and 3D point cloud features.

[0090] Image feature extractor:

[0091] A semantic segmentation network is used to generate pixel semantic features. The segmentation network takes multi-view images as input and represents them as follows: Where w and h are the width and length of the input image, respectively, and the output of the image feature extractor is denoted as... Where w′ and h′ are proportional scalings of the width and length of the input image, respectively. f 2D The input image size of the feature extractor is an integer multiple of the output size, representing the feature dimension of the output.

[0092] 3D point cloud feature extractor:

[0093] A 3D segmentation network is used to obtain the features of each point. The LiDAR point cloud is used as the input to the 3D point cloud feature extractor, and the output is... Where N is the number of points in the lidar point cloud, f 3D The output feature dimension.

[0094] Step S2: Fuse the features extracted from data from different modalities;

[0095] Specifically, in step S2:

[0096] Features from RGB images and LiDAR point clouds are fused together. Semantic information from both RGB images and LiDAR point clouds is used to detect 3D bounding box marking errors. Semantic information from RGB images and LiDAR point clouds is fused through camera calibration. The 3D point cloud is projected from the LiDAR coordinate system to the camera coordinate system using camera calibration. The correspondence between points in the 3D point cloud and image pixels is established. Image features and point cloud features are then fused.

[0097] Establish the relationship between points in the point cloud and pixels in the image. The 2D image and 3D points are linked through projection geometry, projecting each LiDAR point p onto the RGB image:

[0098] [x,y,1] T =K(Rp+t)

[0099] Where (x, y) are the pixel coordinates of the 3D point p projected onto the RGB image, K is the camera's intrinsic calibration matrix, and R and t are the rotation matrix and translation vector that transform the 3D point from the LiDAR coordinate system to the camera coordinate system, resulting in a mapping from the LiDAR point cloud to the RGB image; if the feature map F output by the image feature extractor... 2D The resolution differs from the original image, w′≠w and h′≠h. Update the mapping between points and pixels by dividing the pixel coordinates (x,y) by the scaling factor i. And it is rounded to the nearest integer value;

[0100] Using mapping relationships, image features Convert to When multiple images exist, points in a point cloud are projected onto multiple images, and the features of multiple images are fused by taking the maximum value of each point.

[0101] Image features F 2D With 3D point cloud features F 3D By performing point-by-point connections, the fused features are obtained.

[0102] Step S3: Calculate the segmentation result based on the fusion features;

[0103] Specifically, in step S3:

[0104] The features P are fused and input into a fully connected neural network MLP, where each layer is followed by a batch normalization layer and a dropout layer. The activation function used is LeakyReLU. The output of the MLP is then input into a softmax layer to obtain the class probability distribution for each point, denoted as . Where m represents the number of possible categories of a point;

[0105] Step S4: Calculate the annotation error score of the 3D bounding box based on the segmentation results of the points in the 3D bounding box.

[0106] In step S4:

[0107] The annotation error score of a 3D bounding box is the average of the segmentation errors of all points within the 3D bounding box, where the segmentation error is calculated based on the probability distribution P. j The actual category y of the j-th point is greater than j The probability P j [y j The sum of all probabilities of ] is formalized as follows:

[0108]

[0109] in It is a probability distribution The probability of the i-th category;

[0110] The cross-entropy loss function is:

[0111]

[0112] Where P j [y j ] represents the class probability distribution P of point j. j In the middle, category y j The probability of;

[0113] After the training phase, the points in the 3D bounding box are input into the model. The output P of the Softmax layer is used to determine if the 3D bounding box is valid. For the j-th point in the point cloud, the segmentation error is within the probability distribution P. j The actual category y of the j-th point is greater than j The probability P j [y j The sum of all probabilities of ] is formalized as follows:

[0114]

[0115] in It is a probability distribution The probability of the i-th category in the bounding box; the 3D bounding box labeling error score is the average of the segmentation errors of all points in the 3D bounding box.

[0116] A threshold τ is used to determine the error score of the 3D bounding box annotation, and the 3D bounding box is marked as correct or incorrect.

[0117] Specifically, a three-stage training method is proposed: in the first stage, the multimodal feature extraction module is trained using transfer learning; in the second stage, the point cloud segmentation module is trained separately; and in the third stage, the entire model is adjusted.

[0118] The image feature extractor and the 3D point cloud feature extractor are pre-trained. During the pre-training process, the semantic segmentation dataset with correct data annotation in a scenario with a higher similarity to the target task than a preset standard is used for training. A linear layer and a softmax layer are added after the feature extractor to adjust the output dimension to match the number of categories in the dataset.

[0119] The parameters of the multimodal feature extraction module are fixed, and the point cloud segmentation module is trained. Segmentation labels are generated by 3D bounding boxes. For points inside the 3D bounding boxes, the corresponding category labels are directly assigned. All points outside the 3D bounding boxes are used as background. The Adam optimizer is used for training to adjust the overall model.

[0120] Example 2:

[0121] Example 2 is a preferred embodiment of Example 1, and is used to illustrate the present invention in more detail.

[0122] The present invention also provides a multimodal fusion unsupervised 3D bounding box error detection system. The multimodal fusion unsupervised 3D bounding box error detection system can be implemented by executing the process steps of the multimodal fusion unsupervised 3D bounding box error detection method. That is, those skilled in the art can understand the multimodal fusion unsupervised 3D bounding box error detection method as a preferred embodiment of the multimodal fusion unsupervised 3D bounding box error detection system.

[0123] According to the present invention, an unsupervised 3D bounding box annotation error detection system based on multimodal fusion includes:

[0124] Module M1: Extracts data features using two-dimensional image segmentation and three-dimensional LiDAR segmentation methods to obtain semantic information;

[0125] Specifically, in module M1:

[0126] The multimodal feature extraction module includes an image feature extractor and a 3D point cloud feature extractor, which extract image features and 3D point cloud features.

[0127] Image feature extractor:

[0128] A semantic segmentation network is used to generate pixel semantic features. The segmentation network takes multi-view images as input and represents them as follows: Where w and h are the width and length of the input image, respectively, and the output of the image feature extractor is denoted as... Where w′ and h′ are proportional scalings of the width and length of the input image, respectively. f 2D The input image size of the feature extractor is an integer multiple of the output size, representing the feature dimension of the output.

[0129] 3D point cloud feature extractor:

[0130] A 3D segmentation network is used to obtain the features of each point. The LiDAR point cloud is used as the input to the 3D point cloud feature extractor, and the output is... Where N is the number of points in the lidar point cloud, f 3D The output feature dimension.

[0131] Module M2: fuses features extracted from data from different modalities;

[0132] Specifically, in module M2:

[0133] Features from RGB images and LiDAR point clouds are fused together. Semantic information from both RGB images and LiDAR point clouds is used to detect 3D bounding box marking errors. Semantic information from RGB images and LiDAR point clouds is fused through camera calibration. The 3D point cloud is projected from the LiDAR coordinate system to the camera coordinate system using camera calibration. The correspondence between points in the 3D point cloud and image pixels is established. Image features and point cloud features are then fused.

[0134] Establish the relationship between points in the point cloud and pixels in the image. The 2D image and 3D points are linked through projection geometry, projecting each LiDAR point p onto the RGB image:

[0135] [x,y,1] R =K(Rp+t)

[0136] Where (x, y) are the pixel coordinates of the 3D point p projected onto the RGB image, K is the camera's intrinsic calibration matrix, and R and t are the rotation matrix and translation vector that transform the 3D point from the LiDAR coordinate system to the camera coordinate system, resulting in a mapping from the LiDAR point cloud to the RGB image; if the feature map F output by the image feature extractor... 2D The resolution differs from the original image, w′≠w and h′≠h. Update the mapping between points and pixels by dividing the pixel coordinates (x,y) by the scaling factor i. And it is rounded to the nearest integer value;

[0137] Using mapping relationships, image features Convert to When multiple images exist, points in a point cloud are projected onto multiple images, and the features of multiple images are fused by taking the maximum value of each point.

[0138] Image features F 2D With 3D point cloud features F 3D By performing point-by-point connections, the fused features are obtained.

[0139] Module M3: Calculates the segmentation result based on the fusion features;

[0140] Specifically, in module M3:

[0141] The features P are fused and input into a fully connected neural network MLP, where each layer is followed by a batch normalization layer and a dropout layer. The activation function used is LeakyReLU. The output of the MLP is then input into a softmax layer to obtain the class probability distribution for each point, denoted as . Where m represents the number of possible categories of a point;

[0142] Module M4: Calculates the annotation error score of the 3D bounding box based on the segmentation results of the points in the 3D bounding box.

[0143] In module M4:

[0144] The annotation error score of a 3D bounding box is the average of the segmentation errors of all points within the 3D bounding box, where the segmentation error is calculated based on the probability distribution P. j The actual category y of the j-th point is greater than j The probability P j [y j The sum of all probabilities of ] is formalized as follows:

[0145]

[0146] in It is a probability distribution The probability of the i-th category;

[0147] The cross-entropy loss function is:

[0148] L(Y,P)=∑ 0≤j≤N log P j [y j ]

[0149] Where P j [y j ] represents the class probability distribution P of point j. j In the middle, category y j The probability of;

[0150] After the training phase, the points in the 3D bounding box are input into the model. The output P of the Softmax layer is used to determine if the 3D bounding box is valid. For the j-th point in the point cloud, the segmentation error is within the probability distribution P. j The actual category y of the j-th point is greater than j The probability P j [y j The sum of all probabilities of ] is formalized as follows:

[0151]

[0152] in It is a probability distribution The probability of the i-th category in the bounding box; the 3D bounding box labeling error score is the average of the segmentation errors of all points in the 3D bounding box.

[0153] A threshold τ is used to determine the error score of the 3D bounding box annotation, and the 3D bounding box is marked as correct or incorrect.

[0154] Specifically, a three-stage training method is proposed: in the first stage, the multimodal feature extraction module is trained using transfer learning; in the second stage, the point cloud segmentation module is trained separately; and in the third stage, the entire model is adjusted.

[0155] The image feature extractor and the 3D point cloud feature extractor are pre-trained. During the pre-training process, the semantic segmentation dataset with correct data annotation in a scenario with a higher similarity to the target task than a preset standard is used for training. A linear layer and a softmax layer are added after the feature extractor to adjust the output dimension to match the number of categories in the dataset.

[0156] The parameters of the multimodal feature extraction module are fixed, and the point cloud segmentation module is trained. Segmentation labels are generated by 3D bounding boxes. For points inside the 3D bounding boxes, the corresponding category labels are directly assigned. All points outside the 3D bounding boxes are used as background. The Adam optimizer is used for training to adjust the overall model.

[0157] Example 3:

[0158] Example 3 is a preferred example of Example 1, and is used to illustrate the present invention in more detail.

[0159] An unsupervised 3D bounding box annotation error detection method based on multimodal fusion fully utilizes the semantic information of RGB images and LiDAR point clouds to jointly detect 3D bounding box annotation errors. This means that if the 3D bounding box viewed from either the image perspective or the LiDAR perspective does not match the actual situation, it will be considered an incorrectly labeled bounding box.

[0160] A deep neural network mainly consists of four modules: a multimodal feature extraction module, a semantic fusion module, a point cloud segmentation module, and a labeling error score generation module.

[0161] The multimodal feature extraction module can be integrated with any existing semantic segmentation model to provide it with multimodal feature extraction capabilities.

[0162] A method is proposed to fuse the semantic information of RGB images with the semantic information of LiDAR point clouds through camera calibration.

[0163] This paper proposes an innovative method for calculating the annotation error score of a 3D bounding box. The annotation error score of a 3D bounding box is defined as the average of the segmentation errors of all points within the 3D bounding box. The segmentation error is defined as the error calculated based on the probability distribution P. j The actual category y of the j-th point is greater than j The probability P j [y j The sum of all probabilities of ] is formalized as follows:

[0164]

[0165] in It is a probability distribution The probability of the i-th category.

[0166] We propose a label-independent loss function. Our goal is to correctly classify each point in the point cloud. The formula is as follows:

[0167]

[0168] Where P j [y j ] represents the class probability distribution P of point j. j In the middle, category y j The probability of.

[0169] We propose a three-stage training method. In the first stage, we train the multimodal feature extraction module using transfer learning, thus fully utilizing existing domain knowledge. In the second stage, we train the point cloud segmentation module separately. Finally, in the third stage, we fine-tune the entire model to optimize its performance. This staged training approach helps improve the model's accuracy and robustness.

[0170] 3DboxAD is an unsupervised deep learning model that consists of four basic modules, such as... Figure 1 As shown.

[0171] i) Multimodal feature extraction module: This module includes an image feature extractor and a 3D point cloud feature extractor, which are used to extract image features and 3D point cloud features, respectively.

[0172] ii) Semantic Fusion Module: Using camera calibration, the 3D point cloud is projected from the radar coordinate system to the camera coordinate system, thereby establishing the correspondence between points in the 3D point cloud and image pixels. Based on this, image features and point cloud features are fused to enable more comprehensive and accurate annotation error detection.

[0173] iii) Point cloud segmentation module: Calculates segmentation results using fused features. Specifically, we will determine which category each point belongs to based on these features.

[0174] iv) Labeling error score generation module: Based on the segmentation results of the points in the 3D bounding box, further calculate the labeling error score of the 3D bounding box.

[0175] The 3D bounding box annotation error detection method of the present invention is as follows:

[0176] 1. Multimodal feature extraction module

[0177] Image feature extractor:

[0178] Two-dimensional RGB images contain rich semantic and color information, providing supplementary cues for point clouds and thus facilitating 3D bounding box annotation error detection. To extract useful information from two-dimensional images, we utilize a semantic segmentation network to generate pixel semantic features. Notably, the proposed framework is independent of specific segmentation models; any state-of-the-art methods, such as Deeplabv3, PSPNet, HTC, FCN, and MobelNetV3, can be used. The segmentation network takes multi-view images as input, represented as... Where w and h are the width and length of the input image, respectively. Here, we denote the output of the image feature extractor as... Where w′ and h′ are proportional scalings of the width and length of the input image, respectively. f 2D The feature dimension is the output. This differs from traditional semantic segmentation models. To allow a single pixel in the segmentation result to contain richer semantic information, we allow the input image size of the feature extractor to be an integer multiple (i.e., s times) of the output size.

[0179] 3D point cloud feature extractor:

[0180] In addition to extracting features from 2D RGB images, we also consider obtaining semantic information from LiDAR point clouds. Specifically, we employ a 3D segmentation network to acquire features for each point. Any advanced 3D segmentation network can be used as a 3D point cloud feature extractor, such as PointNet, PolarNet, AF2S3Net, RPVNet, etc. We use the LiDAR point cloud as input to the 3D point cloud feature extractor, and the output is... Where N is the number of points in the lidar point cloud, f 3D The output feature dimension.

[0181] 2. Semantic Fusion Module

[0182] To fuse the semantic information of RGB images and point clouds, we need to establish the relationship between points in the point cloud and pixels in the image. For example... Figure 3 As shown, the two-dimensional image and three-dimensional points are linked through projection geometry. We first project each LiDAR point p onto the RGB image.

[0183] [x,y,1] T =K(Rp+t)

[0184] Where (x, y) are the pixel coordinates of the 3D point p projected onto the RGB image, K is the camera's intrinsic calibration matrix, and R and t are the rotation matrix and translation vector that transform the 3D point from the LiDAR coordinate system to the camera coordinate system. Thus, we obtain a mapping from the LiDAR point cloud to the RGB image. If the feature map F output by the image feature extractor... 2D Since the resolution of the image is different from the original image (that is, w′≠w and h′≠h), we will update the mapping between points and pixels by dividing the pixel coordinates (x,y) by the scaling factor i. And it is rounded to the nearest integer value.

[0185] Using this mapping relationship, we can extract image features Convert to When multiple images exist, meaning points in a point cloud are projected onto multiple images, we fuse the features of multiple images by maximizing the value of each point.

[0186] Finally, we will use the image features F 2D With 3D point cloud features F 3D By performing point-by-point connections, the fused features are obtained.

[0187] 3. Point cloud segmentation module

[0188] We fuse the features P and input them into a fully connected neural network (MLP), where each layer is followed by a batch normalization layer and a dropout layer to reduce overfitting and enhance the model's generalization ability. We use LeakyReLU as the activation function, which provides a small gradient when the input is negative, avoiding the vanishing gradient problem. Finally, we input the output of the MLP into a softmax layer to obtain the class probability distribution for each point, denoted as . Where m represents the number of possible categories of a point.

[0189] 4. Error labeling score calculation module

[0190] After the training phase, a well-trained model can be used to detect whether 3D bounding boxes are incorrectly labeled. We can input the points in the 3D bounding boxes into this model and then determine whether the 3D bounding boxes are correct based on the output P of the Softmax layer. Typically, the probability of incorrect categories is relatively low compared to the correct categories. Based on this idea, for the j-th point in the point cloud, the segmentation error of that point is defined as the probability distribution P... j The actual category y of the j-th point is greater than j The probability P j [y jThe sum of all probabilities of ] is formalized as follows:

[0191]

[0192] in It is a probability distribution The probability of the i-th category. Figure 4 It shows the probability distribution of a point and the segmentation error when the point belongs to each category.

[0193] The 3D bounding box annotation error score is defined as the average of the segmentation errors of all points in the 3D bounding box.

[0194] We use a threshold τ to determine the error score of 3D bounding boxes, marking them as either correct or incorrect. In our method, the higher the error score of a 3D bounding box, the more likely it is to be judged as incorrectly labeled.

[0195] 5. Model Training

[0196] We propose a three-stage training method, such as... Figure 5 The diagram shows the participation of each module in the three phases:

[0197] First, we pre-train the image feature extractor and the 3D point cloud feature extractor to give them some segmentation capabilities. During pre-training, we use semantic segmentation datasets with realistic and correctly labeled data in scenarios similar to the target task. For example, for autonomous driving-related tasks, we use 3D object detection data from the KITTI dataset for pre-training the 3D point cloud feature extractor. Simultaneously, to pre-train the image feature extractor, we use the INRIA Graz-02 (IG02) and Sift Flow datasets. However, the number of classes in these datasets may differ from the output dimension of the feature extractor. Therefore, during pre-training, we add a linear layer and a softmax layer after the feature extractor to adjust the output dimension to match the number of classes in the dataset.

[0198] Next, we fix the parameters of the multimodal feature extraction module and train the point cloud segmentation module. We don't need any additional point-by-point semantic segmentation labels; instead, we can generate segmentation labels using 3D bounding boxes. For points within the 3D bounding box, we directly assign them the corresponding class label, while all points outside the 3D bounding box are treated as background. In this way, we can freely use 3D bounding boxes to generate segmentation labels. This process can be described as minimizing the reconstruction error (i.e., cross-entropy loss). Cross-entropy loss measures the difference between the model's prediction and the true label, helping the model better learn how to correctly classify each point in the point cloud.

[0199]

[0200] Where P j [y j ] represents the class probability distribution P of point j. j In the middle, category y j The probability of [the outcome]. We use the Adam optimizer for training.

[0201] Finally, we fine-tune the overall model.

[0202] 3DboxAD is effective for two reasons: 1. Pre-training the image feature extractor and the 3D point cloud feature extractor gives them inherent object recognition capabilities. 2. Limiting the dimension of the fused feature F (that is, f... 2D +f 3D We can limit the model's learning ability, thereby enabling it to learn patterns from a large number of normal samples and abandoning the learning of patterns from incorrectly labeled 3D boxes. Ultimately, the labeling error score of incorrectly labeled 3D boxes will increase accordingly.

[0203] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0204] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A multimodal fusion-based unsupervised 3D bounding box annotation error detection method, characterized in that, include: Step S1: Extract data features using two-dimensional image segmentation and three-dimensional LiDAR segmentation methods to obtain semantic information; Step S2: Fuse the features extracted from data from different modalities; Step S3: Calculate the segmentation result based on the fusion features; Step S4: Calculate the annotation error score of the 3D bounding box based on the segmentation results of the points in the 3D bounding box; In step S3: The cross-entropy loss function is: in Point Category probability distribution In, category The probability of; The features P are fused and input into a fully connected neural network MLP, where each layer is followed by a batch normalization layer and a dropout layer. The activation function used is LeakyReLU. The output of the MLP is then input into a softmax layer to obtain the class probability distribution for each point, denoted as . ,in Indicates the number of possible categories of a point; In step S4: After the training phase, the points in the 3D bounding box are input into the model, and the results are calculated based on the output of the Softmax layer. To determine if a 3D bounding box is normal, for the first point cloud... There are points, and the segmentation error of each point is in the probability distribution. The middle is greater than the first The actual labeling category of each point probability The sum of all probabilities is formalized as follows: in It is a probability distribution The Middle The probability of each category, and the 3D bounding box labeling error score is the average of the segmentation errors of all points in the 3D bounding box; A threshold τ is used to determine the error score of the 3D bounding box annotation, and the 3D bounding box is marked as correct or incorrect.

2. The multimodal fusion-based unsupervised 3D bounding box annotation error detection method according to claim 1, characterized in that, In step S1: The multimodal feature extraction module includes an image feature extractor and a 3D point cloud feature extractor, which extract image features and 3D point cloud features. Image feature extractor: A semantic segmentation network is used to generate pixel semantic features. The segmentation network takes multi-view images as input and represents them as follows: ,in , Let the width and length of the input image be denoted as and the output of the image feature extractor be denoted as . ,in, , The width and length of the input image are scaled proportionally, respectively. , , The feature dimension is the input image size of the feature extractor, which is an integer multiple of the output size. times; 3D point cloud feature extractor: A 3D segmentation network is used to obtain the features of each point. The LiDAR point cloud is used as the input to the 3D point cloud feature extractor, and the output is... ,in The number of points in the lidar point cloud. This refers to the feature dimension of the output.

3. The multimodal fusion-based unsupervised 3D bounding box annotation error detection method according to claim 1, characterized in that, In step S2: Features from RGB images and LiDAR point clouds are fused together. Semantic information from both RGB images and LiDAR point clouds is used to detect 3D bounding box marking errors. Semantic information from RGB images and LiDAR point clouds is fused through camera calibration. The 3D point cloud is projected from the LiDAR coordinate system to the camera coordinate system using camera calibration. The correspondence between points in the 3D point cloud and image pixels is established. Image features and point cloud features are then fused. Establish the relationship between points in the point cloud and pixels in the image; connect the 2D image and 3D points through projection geometry, and link each LiDAR point... Projected onto an RGB image: in, It is a three-dimensional point The pixel coordinates projected onto the RGB image. It is the camera's inherent calibration matrix. and It involves using rotation matrices and translation vectors to transform 3D points from the LiDAR coordinate system to the camera coordinate system, resulting in a mapping from the LiDAR point cloud to an RGB image; if the feature map output by the image feature extractor... The resolution is different from the original image. and Using pixel coordinates Divide by scaling factor Update the mapping between points and pixels And rounded to the nearest integer value; Using mapping relationships, image features Convert to When multiple images exist, points in a point cloud are projected onto multiple images, and the features of multiple images are fused by taking the maximum value of each point. Image features With 3D point cloud features By performing point-by-point connections, the fused features are obtained. .

4. The multimodal fusion-based unsupervised 3D bounding box annotation error detection method according to claim 1, characterized in that: A three-stage training method is proposed. In the first stage, the multimodal feature extraction module is trained using transfer learning. In the second stage, the point cloud segmentation module is trained separately. In the third stage, the entire model is adjusted. The image feature extractor and the 3D point cloud feature extractor are pre-trained. During the pre-training process, the semantic segmentation dataset with correct data annotation in a scenario with a higher similarity to the target task than a preset standard is used for training. A linear layer and a softmax layer are added after the feature extractor to adjust the output dimension to match the number of categories in the dataset. The parameters of the multimodal feature extraction module are fixed, and the point cloud segmentation module is trained. Segmentation labels are generated by 3D bounding boxes. For points inside the 3D bounding boxes, the corresponding category labels are directly assigned. All points outside the 3D bounding boxes are used as background. The Adam optimizer is used for training to adjust the overall model.

5. A multimodal fusion unsupervised 3D bounding box annotation error detection system, characterized in that, include: Module M1: Extracts data features using two-dimensional image segmentation and three-dimensional LiDAR segmentation methods to obtain semantic information; Module M2: fuses features extracted from data from different modalities; Module M3: Calculates the segmentation result based on the fusion features; Module M4: Calculates the annotation error score of the 3D bounding box based on the segmentation results of the points in the 3D bounding box; In module M3: The cross-entropy loss function is: in Point Category probability distribution In, category The probability of; The features P are fused and input into a fully connected neural network MLP, where each layer is followed by a batch normalization layer and a dropout layer. The activation function used is LeakyReLU. The output of the MLP is then input into a softmax layer to obtain the class probability distribution for each point, denoted as . ,in Indicates the number of possible categories of a point; In module M4: After the training phase, the points in the 3D bounding box are input into the model, and the results are calculated based on the output of the Softmax layer. To determine if a 3D bounding box is normal, for the first point cloud... There are points, and the segmentation error of each point is in the probability distribution. The middle is greater than the first The actual labeling category of each point probability The sum of all probabilities is formalized as follows: in It is a probability distribution The Middle The probability of each category, and the 3D bounding box labeling error score is the average of the segmentation errors of all points in the 3D bounding box; A threshold τ is used to determine the error score of the 3D bounding box annotation, and the 3D bounding box is marked as correct or incorrect.

6. The multimodal fusion-based unsupervised 3D bounding box annotation error detection system according to claim 5, characterized in that, In module M1: The multimodal feature extraction module includes an image feature extractor and a 3D point cloud feature extractor, which extract image features and 3D point cloud features. Image feature extractor: A semantic segmentation network is used to generate pixel semantic features. The segmentation network takes multi-view images as input and represents them as follows: ,in , Let the width and length of the input image be denoted as and the output of the image feature extractor be denoted as . ,in, , The width and length of the input image are scaled proportionally, respectively. , , The feature dimension is the input image size of the feature extractor, which is an integer multiple of the output size. times; 3D point cloud feature extractor: A 3D segmentation network is used to obtain the features of each point. The LiDAR point cloud is used as the input to the 3D point cloud feature extractor, and the output is... ,in The number of points in the lidar point cloud. The output feature dimension.

7. The multimodal fusion-based unsupervised 3D bounding box annotation error detection system according to claim 5, characterized in that, In module M2: Features from RGB images and LiDAR point clouds are fused together. Semantic information from both RGB images and LiDAR point clouds is used to detect 3D bounding box marking errors. Semantic information from RGB images and LiDAR point clouds is fused through camera calibration. The 3D point cloud is projected from the LiDAR coordinate system to the camera coordinate system using camera calibration. The correspondence between points in the 3D point cloud and image pixels is established. Image features and point cloud features are then fused. Establish the relationship between points in the point cloud and pixels in the image; connect the 2D image and 3D points through projection geometry, and link each LiDAR point... Projected onto an RGB image: in, It is a three-dimensional point The pixel coordinates projected onto the RGB image. It is the camera's inherent calibration matrix. and It involves using rotation matrices and translation vectors to transform 3D points from the LiDAR coordinate system to the camera coordinate system, resulting in a mapping from the LiDAR point cloud to an RGB image; if the feature map output by the image feature extractor... The resolution is different from the original image. and Using pixel coordinates Divide by scaling factor Update the mapping between points and pixels And rounded to the nearest integer value; Using mapping relationships, image features Convert to When multiple images exist, points in a point cloud are projected onto multiple images, and the features of multiple images are fused by taking the maximum value of each point. Image features With 3D point cloud features By performing point-by-point connections, the fused features are obtained. .

8. The multimodal fusion-based unsupervised 3D bounding box annotation error detection system according to claim 5, characterized in that: A three-stage training method is proposed. In the first stage, the multimodal feature extraction module is trained using transfer learning. In the second stage, the point cloud segmentation module is trained separately. In the third stage, the entire model is adjusted. The image feature extractor and the 3D point cloud feature extractor are pre-trained. During the pre-training process, the semantic segmentation dataset with correct data annotation in a scenario with a higher similarity to the target task than a preset standard is used for training. A linear layer and a softmax layer are added after the feature extractor to adjust the output dimension to match the number of categories in the dataset. The parameters of the multimodal feature extraction module are fixed, and the point cloud segmentation module is trained. Segmentation labels are generated by 3D bounding boxes. For points inside the 3D bounding boxes, the corresponding category labels are directly assigned. All points outside the 3D bounding boxes are used as background. The Adam optimizer is used for training to adjust the overall model.

Images