Unsupervised multi-modal 3D box error detection system and method

Abstract

The application provides a kind of unsupervised multimodal 3D frame error detection system and method, comprising: the generation module of virtual error generates virtual error 3D frame;Model training module: train an end-to-end error detector to regress error score;Error detection module: input initial labeling into error detector, obtain the error score of each 3D frame, by using a threshold τ, 3D frame error score is mapped to 0 or 1, 0 indicates correct, 1 indicates error.The application does not need artificial 3D frame labeling whether error label, also does not rely on any prior knowledge, by fusing multimodal data, can more effectively detect the labeling error of 3D frame.

Description

Technical Field

[0001] This invention relates to the field of 3D bounding box annotation error detection technology, and more specifically, to an unsupervised multimodal 3D bounding box error detection system and method. Background Technology

[0002] In recent years, autonomous driving technology has advanced rapidly, and 3D object detection methods have played a crucial role in numerous fields. The training of 3D object detection models heavily relies on accurate and high-quality 3D bounding box annotations. These annotations can be provided by professional data annotation engineers or obtained through semi-automatic or automatic methods (such as automatic 3D bounding box annotation tools). However, annotated 3D bounding boxes are not 100% reliable and may contain various errors. Accurate 3D object detection plays a vital role in the accurate understanding of the vehicle's surrounding environment. It provides autonomous driving systems with the necessary information for efficient path planning and intelligent driving decisions, ensuring that the vehicle can safely and accurately reach its destination. If erroneous 3D bounding box annotations are used as ground truth labels during training, this will severely impact model performance and may lead to false positives in object detection tasks. Therefore, we urgently need a method to detect these erroneous 3D bounding box annotations to ensure that the model receives high-quality training data, thereby improving its accuracy and reliability.

[0003] However, unsupervised and multimodal 3D bounding box annotation error detection is a challenging task for three main reasons. First, 3D bounding box errors are diverse. They may misclassify objects, such as misidentifying a small car as a bus, or incorrectly bounding non-target objects in the background. Second, multiple sensors, such as cameras and LiDAR, are mounted on data acquisition vehicles. These generate multimodal data at different time stamps, and effectively fusing this multimodal data for 3D bounding box annotation error detection is a challenging task. Finally, unsupervised annotation error detection requires that we cannot label any 3D bounding box as incorrect, making it impossible to manually provide effective training data for the 3D bounding box error detection system. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide an unsupervised multimodal 3D bounding box error detection system and method.

[0005] The unsupervised multimodal 3D bounding box error detection system provided by the present invention includes:

[0006] Virtual error generation module: Generates virtual 3D error boxes;

[0007] Model training module: Trains an end-to-end error detector to regress error scores;

[0008] Error detection module: The initial annotations are input into the error detector to obtain the error score of each 3D box. By using a threshold τ, the error score of the 3D box is mapped to 0 or 1, where 0 represents correct and 1 represents error.

[0009] Preferably, during the multimodal fusion process, points in the LiDAR point cloud are projected onto RGB image pixels, thereby establishing a mapping relationship between points in the point cloud and pixels in the image. The mapping relationship between points in the 3D point cloud and pixels in the 2D image is established through projection geometry, thus mapping the points in the LiDAR... Projecting onto an RGB image, the expression is:

[0010] [x,y,1] T =K(Rp+t)

[0011] Where (x,y) are the pixel coordinates of the 3D point p projected onto the RGB image, K is the camera's intrinsic calibration matrix, and R and t are the rotation matrix and translation vector that transform the 3D point from the LiDAR coordinate system to the camera coordinate system.

[0012] Preferably, the virtual error generation module includes:

[0013] Let C be the set of 3D bounding box categories, containing all possible categories of objects within the 3D bounding boxes. In the annotation results, count the frequency of each 3D bounding box category c, denoted as n. c ;

[0014] The following two methods are used to generate virtual error 3D boxes:

[0015] i) Change the category of the 3D bounding box within the initial annotation: When the actual object within the 3D bounding box does not match the category of the 3D bounding box's annotation, the 3D bounding box is considered incorrect. The category c′ of some of the 3D bounding boxes within the initial annotation is changed to category c to obtain the incorrect 3D bounding box, where c ≠ c′. When the 3D bounding box's annotation category is c′, according to the probability distribution... Where n ci For category c i The frequency of occurrence is used to select the modified category c;

[0016] ii) Fake 3D bounding box: When a non-target object in the background is bounded, the 3D bounding box is considered to be incorrect, and a portion of the background object is randomly bounded to create a fake 3D bounding box.

[0017] Preferably, the end-to-end error detector includes: a feature extractor and an error score generator;

[0018] The feature extractor extracts feature representations of 3D bounding boxes from the RGB image I and the LiDAR point cloud P. The feature extractor extracts three types of features for the 3D bounding boxes:

[0019] a) Point cloud features of the region of interest

[0020] To extract local features from the LiDAR point cloud within the 3D bounding box, we first obtain the point cloud within the region of interest of the 3D bounding box from the LiDAR point cloud, denoted as P. b The focus will be on the point cloud P within the region. b As a 3D point cloud feature extraction network The input and output are Where d b3D The feature dimension of the output;

[0021] b) Image features of the region of interest

[0022] Project the 3D bounding box onto the RGB image and find the minimum bounding rectangle of the corresponding 2D 3D bounding box. The region within this minimum bounding rectangle is taken as the region of interest of the image, and the image within the region of interest is denoted as I. b Image I within the area of ​​focus b As an image feature extraction network The input and output are Where d b2D The feature dimension of the output;

[0023] c) Focus on the integration characteristics of the region

[0024] To deeply fuse multimodal features, RGB images The input is fed into an image semantic segmentation network to generate high-level pixel semantic features with global semantic information. The point cloud P is input into a 3D segmentation network to generate high-level point-by-point features with global semantic information. Where w and h are the width and height of the input image I, respectively; N is the total number of points in the point cloud; d 2D and d 3D The feature dimension of the output;

[0025] By utilizing the mapping relationship between points in a point cloud and image pixels, semantic features of the image can be extracted. Convert to Image features H′ 2D With point cloud features H 3D By performing point-by-point connections, a high-level fused point cloud is obtained. Each point in the fused point cloud contains global information about both the point cloud and the image;

[0026] To obtain the fusion features of the region of interest, the point cloud of the 3D bounding box within the region of interest is obtained from the fused point cloud H, denoted as H. b The focus will be on the point cloud H within the region. b As a 3D point cloud feature extraction network The input and output are Where d bH The feature dimension of the output;

[0027] The features extracted by the feature extractor are concatenated with the inherent features of the 3D bounding box to generate the final features. The feature F is input into the error score generator, which adopts a standard multilayer perceptron structure. In each layer, the generator accepts the category features of the 3D bounding box as additional input, that is, the connection between the output of the previous layer and the category features is used as the input of the next layer.

[0028] In the error score generator, each layer is followed by a batch normalization layer and a dropout layer to reduce overfitting and enhance the model's generalization ability. The activation function used is LeakyReLU, which provides a gradient when the input is negative to avoid the gradient vanishing problem.

[0029] The output of the MLP is input into the Sigmoid activation function to obtain a scalar z between 0 and 1. This scalar represents the error score of the 3D bounding box, and the closer it is to 1, the greater the degree of error.

[0030] Preferably, during training, all 3D bounding boxes in the initial annotation are considered correct, and generated virtual erroneous 3D bounding boxes are considered incorrect. The training loss function is as follows:

[0031]

[0032] in, Represents the set of 3D bounding box annotations, y b ∈{0,1} indicates whether the 3D bounding box b is incorrect, y b =1 indicates that the 3D bounding box b is incorrect, z b This represents the error score of the 3D bounding box b.

[0033] The unsupervised multimodal 3D bounding box error detection method provided by the present invention includes:

[0034] Step 1: Generate a virtual 3D error box;

[0035] Step 2: Train an end-to-end error detector to regress the error score;

[0036] Step 3: Input the initial annotations into the error detector to obtain the error score of each 3D box. By using a threshold τ, the error score of the 3D box is mapped to 0 or 1, where 0 represents correct and 1 represents error.

[0037] Preferably, during the multimodal fusion process, points in the LiDAR point cloud are projected onto RGB image pixels, thereby establishing a mapping relationship between points in the point cloud and pixels in the image. The mapping relationship between points in the 3D point cloud and pixels in the 2D image is established through projection geometry, thus mapping the points in the LiDAR... Projecting onto an RGB image, the expression is:

[0038] [x,y,1] T =K(Rp+t)

[0039] Where (x,y) are the pixel coordinates of the 3D point p projected onto the RGB image, K is the camera's intrinsic calibration matrix, and R and t are the rotation matrix and translation vector that transform the 3D point from the LiDAR coordinate system to the camera coordinate system.

[0040] Preferably, step 1 includes:

[0041] Let C be the set of 3D bounding box categories, containing all possible categories of objects within the 3D bounding boxes. In the annotation results, count the frequency of each 3D bounding box category c, denoted as n. c ;

[0042] The following two methods are used to generate virtual error 3D boxes:

[0043] i) Change the category of the 3D bounding box within the initial annotation: When the actual object within the 3D bounding box does not match the category of the 3D bounding box's annotation, the 3D bounding box is considered incorrect. The category c′ of some of the 3D bounding boxes within the initial annotation is changed to category c to obtain the incorrect 3D bounding box, where c ≠ c′. When the 3D bounding box's annotation category is c′, according to the probability distribution... Where n ci For category c i The frequency of occurrence is used to select the modified category c;

[0044] ii) Fake 3D bounding box: When a non-target object in the background is bounded, the 3D bounding box is considered to be incorrect, and a portion of the background object is randomly bounded to create a fake 3D bounding box.

[0045] Preferably, the end-to-end error detector includes a feature extractor and an error score generator;

[0046] The feature extractor extracts feature representations of 3D bounding boxes from the RGB image I and the LiDAR point cloud P. The feature extractor extracts three types of features for the 3D bounding boxes:

[0047] a) Point cloud features of the region of interest

[0048] To extract local features from the LiDAR point cloud within the 3D bounding box, we first obtain the point cloud within the region of interest of the 3D bounding box from the LiDAR point cloud, denoted as P. b The focus will be on the point cloud P within the region.b As a 3D point cloud feature extraction network The input and output are Where d b3D The feature dimension of the output;

[0049] b) Image features of the region of interest

[0050] Project the 3D bounding box onto the RGB image and find the minimum bounding rectangle of the corresponding 2D 3D bounding box. The region within this minimum bounding rectangle is taken as the region of interest of the image, and the image within the region of interest is denoted as I. b Image I within the area of ​​focus b As an image feature extraction network The input and output are Where d b2D The feature dimension of the output;

[0051] c) Focus on the integration characteristics of the region

[0052] To deeply fuse multimodal features, RGB images The input is fed into an image semantic segmentation network to generate high-level pixel semantic features with global semantic information. The point cloud P is input into a 3D segmentation network to generate high-level point-by-point features with global semantic information. Where w and h are the width and height of the input image I, respectively; N is the total number of points in the point cloud; d 2D and d 3D The feature dimension of the output;

[0053] By utilizing the mapping relationship between points in a point cloud and image pixels, semantic features of the image can be extracted. Convert to Image features H′ 2D With point cloud features H 3D By performing point-by-point connections, a high-level fused point cloud is obtained. Each point in the fused point cloud contains global information about both the point cloud and the image;

[0054] To obtain the fusion features of the region of interest, the point cloud of the 3D bounding box within the region of interest is obtained from the fused point cloud H, denoted as H. b The focus will be on the point cloud H within the region. b As a 3D point cloud feature extraction network The input and output are Where d bH The feature dimension of the output;

[0055] The features extracted by the feature extractor are concatenated with the inherent features of the 3D bounding box to generate the final features. The feature F is input into the error score generator, which adopts a standard multilayer perceptron structure. In each layer, the generator accepts the category features of the 3D bounding box as additional input, that is, the connection between the output of the previous layer and the category features is used as the input of the next layer.

[0056] In the error score generator, each layer is followed by a batch normalization layer and a dropout layer to reduce overfitting and enhance the model's generalization ability. The activation function used is LeakyReLU, which provides a gradient when the input is negative to avoid the gradient vanishing problem.

[0057] The output of the MLP is input into the Sigmoid activation function to obtain a scalar z between 0 and 1. This scalar represents the error score of the 3D bounding box, and the closer it is to 1, the greater the degree of error.

[0058] Preferably, during training, all 3D bounding boxes in the initial annotation are considered correct, and generated virtual erroneous 3D bounding boxes are considered incorrect. The training loss function is as follows:

[0059]

[0060] in, Represents the set of 3D bounding box annotations, y b ∈{0,1} indicates whether the 3D bounding box b is incorrect, y b =1 indicates that the 3D bounding box b is incorrect, z b This represents the error score of the 3D bounding box b.

[0061] Compared with the prior art, the present invention has the following beneficial effects:

[0062] The framework proposed in this invention is an unsupervised method for detecting 3D bounding box annotation errors. It eliminates the need for manual labeling of 3D bounding box errors and requires no prior knowledge. By automatically generating virtual erroneous 3D bounding boxes, the framework transforms an unsupervised problem into a weakly supervised method for training deep models. Through the fusion of multimodal data, it can more effectively detect 3D bounding box annotation errors. Whether viewed from the perspective of RGB images or LiDAR, if the 3D bounding box annotations do not match the actual situation, the model will generate a high error score. Attached Figure Description

[0063] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0064] Figure 1 This is a schematic diagram of the framework for unsupervised and multimodal 3D bounding box annotation error detection according to the present invention;

[0065] Figure 2 This is a schematic diagram of the end-to-end error detector of the present invention;

[0066] Figure 3 This is a schematic diagram illustrating the mapping relationship between points in a 3D point cloud and image pixels according to the present invention.

[0067] Figure 4 This is a schematic diagram of the coordinate system for the data in this invention. Detailed Implementation

[0068] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0069] Example

[0070] like Figure 1 This invention provides an unsupervised multimodal 3D bounding box error detection system, comprising:

[0071] i) Virtual error generation module: mainly involves the generation of virtual error 3D boxes;

[0072] ii) Model training module: Trains an end-to-end error detector to regress error scores;

[0073] iii) Error detection module: Input the initial annotations into the model to obtain the error score of each 3D box. By using a threshold τ, the error score of the 3D box is directly mapped to 0 or 1, where 0 represents correct and 1 represents error.

[0074] In the multimodal fusion process, points in the LiDAR point cloud are projected onto RGB image pixels, thereby establishing a mapping relationship between points in the point cloud and pixels in the image. For example... Figure 3 As shown, the mapping relationship between points in a 3D point cloud and pixels in a 2D image is established through projection geometry. This maps the points in the LiDAR... Projecting onto an RGB image, the expression is:

[0075] [x,y,1] T =K(Rp+t)

[0076] Where (x,y) are the pixel coordinates of the 3D point p projected onto the RGB image, K is the camera's intrinsic calibration matrix, and R and t are the rotation matrix and translation vector that transform the 3D point from the LiDAR coordinate system to the camera coordinate system.

[0077] The 3D bounding box annotation error detection method of the present invention is as follows:

[0078] Step 1: Generating Virtual Errors

[0079] This step primarily involves generating virtual error 3D bounding boxes. Assume the set of 3D bounding box categories is C, containing all possible categories of objects within the 3D boxes. In the annotation results, the frequency of each 3D bounding box category c is counted, denoted as n. c .

[0080] The following two methods are used to generate virtual error 3D boxes: i) change the category of the 3D box in the initial annotation; ii) forge the 3D box.

[0081] i) Change the category of the 3D bounding boxes within the initial annotations: When the actual object within a 3D bounding box does not match the category of the 3D bounding box's annotation, the 3D bounding box is considered incorrect. Therefore, the category c′ of some 3D bounding boxes within the initial annotations is changed to category c to obtain some incorrect 3D bounding boxes, where c ≠ c′. Here, when the 3D bounding box's annotation category is c′, according to the probability distribution... Where n ci For category c i The frequency of occurrence is used to select the changed category c.

[0082] ii) Forged 3D bounding boxes: When a non-target object in the background is bounded, the 3D bounding box is considered incorrect. A portion of the background objects is randomly bounded to create a fake 3D bounding box. The algorithm for forging 3D bounding boxes is as follows:

[0083]

[0084] In the above algorithm, N pB , and These are all hyperparameters preset by the user. The algorithm guarantees that the number of points within the fake 3D bounding box is greater than... And it does not overlap with any other existing 3D bounding boxes. This is closer to the actual pattern of annotation errors.

[0085] The system assigns 3D bounding boxes to categories based on a specified probability distribution, ensuring that the distribution of 3D bounding box categories within the initial annotations is not disrupted after generating dummy errors. This makes it easier to guarantee that the number of 3D bounding boxes belonging to the same category in the initial annotations is the same as the number of fake dummy 3D bounding boxes after generating dummy errors.

[0086] Step 2, inherent features of the 3D frame

[0087] 3D bounding boxes have three inherent features, including a) motion features, b) shape features and c) category features.

[0088] a) Motion characteristics

[0089] Different objects exhibit different motion characteristics, necessitating the generation of motion features for objects within 3D bounding boxes to better detect erroneous 3D bounding box annotations. First, a 3D object tracking algorithm is applied to assign a unique identifier to each 3D bounding box across timestamps; 3D bounding boxes with the same identifier represent the same object. Assuming the identifier for 3D bounding box b is ID(b), at timestamp t, the center coordinates of the 3D bounding box with identifier ID(b) are (x...). t ,y t ,z t The motion feature extraction algorithm for the 3D bounding box b is as follows:

[0090]

[0091] In the above algorithm, v t δ represents the speed of the data acquisition vehicle at timestamp t; t This indicates the direction and angle of the data acquisition vehicle's movement at timestamp t; time_gap indicates the time difference between two adjacent timestamps. For example... Figure 4 As shown, in the algorithm, the front of the data acquisition vehicle points to the positive X-axis direction and to the left to the positive Y-axis direction. The counterclockwise direction of the positive X-axis direction is the positive direction of the motion angle.

[0092] In lines 1-8 of the algorithm, for timestamps t that are adjacent to each other... For points in a coordinate system containing timestamps, calculate the required changes in their X and Y coordinates to transform them from their respective coordinate systems to the coordinate system containing timestamp t. The calculation utilizes the concept of differentiation. Then, in lines 9 to 11 of the algorithm, the adjacent... The center coordinates of the 3D bounding box identified by ID(b) in each timestamp are transformed to the coordinate system of timestamp t. Finally, in line 12 of the algorithm, the motion features of the 3D bounding box can be obtained. This feature is defined as a vector consisting of the distance of position change of the 3D bounding box identified as ID(b) in consecutive adjacent timestamps in the same coordinate system.

[0093] Motion features can intuitively reflect the motion state of objects within a 3D bounding box. For example, when an object within a 3D bounding box is stationary, its motion feature is a d consisting of 0s. m A 3D vector. When the 3D bounding box appears by chance, its motion characteristics are d vectors composed of Nan. m 3D vector. For a fake 3D bounding box, there will be no 3D bounding boxes with the same identifier within adjacent timestamps. To make the fake 3D bounding boxes more realistic, the motion features of the fake 3D bounding boxes are set to the expected motion features of objects in their respective categories.

[0094] b) Shape characteristics

[0095] The inherent characteristics of the 3D bounding box also affect the accuracy of its annotation. For example, the volume of a pedestrian's 3D bounding box is much smaller than that of a bus's 3D bounding box. Therefore, the three-dimensional vector formed by the length, width, and height of the 3D bounding box is used as the shape feature of the 3D bounding box.

[0096] c) Category characteristics

[0097] To detect whether the 3D bounding box's labeled category matches the actual category of the object within the 3D bounding box, category features are also required. One-hot encoding is used to convert the 3D bounding box's category into a |C|-dimensional vector consisting of 0s and 1s. (Category features). Where |C| represents the number of categories. In vector F... bc In the table, all positions except the index position of the category, which is 1, are 0.

[0098] Step 3: End-to-end error detector

[0099] like Figure 2 As shown, the end-to-end error detector consists of two main modules: i) a feature extractor and ii) an error score generator.

[0100] i) Feature Extractor

[0101] The feature extractor extracts feature representations of 3D bounding boxes from the RGB image I and the LiDAR point cloud P. The feature extractor extracts three types of features for the 3D bounding box b: a) point cloud features of the region of interest, b) image features of the region of interest, and c) fusion features of the region of interest.

[0102] a) Point cloud features of the region of interest

[0103] To extract local features from the LiDAR point cloud within the 3D bounding box, we first obtain the point cloud within region b (the region of interest) of the 3D bounding box from the LiDAR point cloud, denoted as P. b Any advanced point cloud feature extraction network can be used to extract point cloud P. b Features, such as the classification network in PointNet and the feature encoding network in VoxelNet, will be used to focus on the point cloud P within the region of interest. b As a 3D point cloud feature extraction network The input and output are Where d b3D This refers to the feature dimension of the output.

[0104] b) Image features of the region of interest

[0105] Project the 3D bounding box onto the RGB image and find the smallest bounding rectangle of this 2D bounding box. The region within this smallest bounding rectangle is taken as the region of interest of the image, and the image within the region of interest is denoted as I. bAny advanced image feature extraction network can be used to extract image features. b Features such as VGG, GoogLeNet, ResNet, etc. The image I within the region of interest. b As an image feature extraction network The input and output are Where d b2D This refers to the feature dimension of the output.

[0106] c) Focus on the integration characteristics of the region

[0107] While LiDAR point clouds contain rich semantic information, 2D RGB images also contain rich semantic and color information, providing supplementary clues to the point cloud. To deeply fuse multimodal features, RGB images are... The input is fed into an image semantic segmentation network, such as MobelNetV3, HTC, Deeplabv3, FCN, etc., to generate high-level pixel semantic features with global semantic information. The point cloud P is input into a 3D segmentation network, such as RPVNet, AF2S3Net, PolarNet, PointNet, etc., to generate high-level pointwise features with global semantic information. Here, w and h are the width and height of the input image I, respectively; N is the total number of points in the point cloud; d 2D and d 3D This refers to the feature dimension of the output.

[0108] By utilizing the mapping relationship between points in a point cloud and image pixels, semantic features of the image can be extracted. Convert to Finally, the image features H′ 2D With point cloud features H 3D By performing point-by-point connections, a high-level fused point cloud is obtained. Each point in the fused point cloud contains global information about both the point cloud and the image.

[0109] To obtain the fusion features of the region of interest, the point cloud within the 3D bounding box region b (the region of interest) is extracted from the fused point cloud H, denoted as H. b The point cloud H within the focus area will be... b As a 3D point cloud feature extraction network The input and output are Where d bH This refers to the feature dimension of the output.

[0110] Both point cloud features and image features of the region of interest acquire information from local sources. However, the fusion features of the region of interest can better acquire information from the global perspective and deeply fuse the point cloud and image.

[0111] ii) Error score generator

[0112] The features extracted by the feature extractor are concatenated with the inherent features of the 3D bounding box (a total of six types of features) to generate the final features. The features F are input into an error score generator, which employs a standard Multilayer Perceptron (MLP) structure, but with a key difference: in each layer, the generator accepts the categorical features of the 3D bounding box as additional input. That is, the concatenation of the output of the previous layer with the categorical features serves as the input to the next layer. This allows for greater attention to the categorical features, as their match with other features determines the correctness of the 3D bounding box. Each layer in the error score generator is followed by a batch normalization layer and a dropout layer to reduce overfitting and enhance the model's generalization ability. The activation function used is LeakyReLU, which provides a small gradient when the input is negative, avoiding the vanishing gradient problem. Finally, the output of the MLP is input into a sigmoid activation function, resulting in a scalar z between 0 and 1. This scalar represents the error score of the 3D bounding box; the closer to 1, the greater the error.

[0113] Step 4: Model Training

[0114] Since the algorithm is unsupervised, it's impossible to determine whether each 3D bounding box in the initial annotations is correct or incorrect. Therefore, during training, all 3D bounding boxes in the initial annotations are simply considered correct. Generated dummy incorrect 3D bounding boxes are treated as incorrect. The training loss function is as follows:

[0115]

[0116] in, Represents the set of 3D bounding box annotations, y b ∈{0,1} indicates whether the 3D box b is incorrect (y b =1 indicates that the 3D box b is incorrect, z b Let represent the error score of 3D box b. The main purpose of this equation is to assign a higher loss to dummy erroneous 3D boxes. By incorporating a logarithmic function, the loss of these 3D boxes is effectively amplified, increasing their impact on the overall loss calculation. This approach emphasizes the importance of accurately identifying and penalizing dummy erroneous 3D boxes, as they are more critical and require more attention during training. Relatively speaking, this loss function can tolerate larger error scores for 3D boxes in the initial annotations.

[0117] Step 5, Error Detection

[0118] A higher error score for a 3D bounding box indicates that it is more likely to be incorrect. After the training phase, the trained model can be used to detect incorrect 3D bounding boxes in the initial annotations. By using a threshold τ, the error score can be mapped to 0 or 1, where 0 represents correct and 1 represents incorrect.

[0119] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0120] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. An unsupervised multimodal 3D bounding box error detection system, characterized in that, include: Virtual error generation module: Generates virtual 3D error boxes; Model training module: Trains an end-to-end error detector to regress error scores; Error detection module: Input the initial annotations into the error detector to obtain the error score of each 3D box. By using a threshold τ, the error score of the 3D box is mapped to 0 or 1, where 0 represents correct and 1 represents error. The end-to-end error detector includes: a feature extractor and an error score generator; The feature extractor extracts from RGB images and lidar point cloud The feature extractor extracts feature representations of 3D bounding boxes, and extracts three types of features for the 3D bounding boxes: a) Point cloud features of the region of interest To extract local features from the LiDAR point cloud within the 3D bounding box, we first obtain the point cloud within the region of interest of the 3D bounding box from the LiDAR point cloud, denoted as . The focus will be on point clouds within the region. As a 3D point cloud feature extraction network The input and output are ,in The feature dimension of the output; b) Image features of the region of interest Project the 3D bounding box onto the RGB image and find the minimum bounding rectangle of the corresponding 2D 3D bounding box. The region within this minimum bounding rectangle is taken as the region of interest of the image, and the image within the region of interest is denoted as . The image will be focused on within the area. As an image feature extraction network The input and output are ,in The feature dimension of the output; c) Focus on the integration characteristics of the region To deeply fuse multimodal features, RGB images The input is fed into an image semantic segmentation network to generate high-level pixel semantic features with global semantic information. ; to point clouds The input is fed into a 3D segmentation network to generate high-level pointwise features with global semantic information. ,in The input images are respectively Width and height; This represents the total number of points in the point cloud. as well as The feature dimension of the output; By utilizing the mapping relationship between points in a point cloud and image pixels, semantic features of the image can be extracted. Convert to Image features Point cloud features By performing point-by-point connections, a high-level fused point cloud is obtained. Each point in the fused point cloud contains global information about both the point cloud and the image; To obtain the fusion features of the region of interest, from the fusion point cloud The point cloud within the 3D bounding box of interest is obtained and denoted as... The focus will be on point clouds within the region. As a 3D point cloud feature extraction network The input and output are ,in The feature dimension of the output; The features extracted by the feature extractor are concatenated with the inherent features of the 3D bounding box to generate the final features. , will feature The input is fed into an error score generator, which employs a standard multilayer perceptron architecture. In each layer, the generator accepts the category features of the 3D bounding box as additional input; that is, the concatenation of the output of the previous layer and the category features serves as the input to the next layer. As a dimension of motion characteristics, For the dimension of shape features, Number of categories; In the error score generator, each layer is followed by a batch normalization layer and a dropout layer to reduce overfitting and enhance the model's generalization ability. The activation function used is LeakyReLU, which provides a gradient when the input is negative to avoid the gradient vanishing problem. The output of the MLP is input into the Sigmoid activation function to obtain a scalar between 0 and 1. This scalar represents the error score of the 3D bounding box; the closer it is to 1, the greater the degree of error.

2. The unsupervised multimodal 3D bounding box error detection system according to claim 1, characterized in that, In the multimodal fusion process, points in the LiDAR point cloud are projected onto RGB image pixels, thereby establishing a mapping relationship between points in the point cloud and pixels in the image. The mapping relationship between points in the 3D point cloud and pixels in the 2D image is established through projection geometry, thus transforming the points in the LiDAR... Projecting onto an RGB image, the expression is: in, It is a three-dimensional point Pixel coordinates projected onto the RGB image It is the camera's inherent calibration matrix. and It is the rotation matrix and translation vector that transforms a 3D point from the lidar coordinate system to the camera coordinate system.

3. The unsupervised multimodal 3D bounding box error detection system according to claim 2, characterized in that, The virtual error generation module includes: Let the set of 3D bounding box categories be This includes all possible categories of objects within the 3D bounding box. The annotation results summarize the category of each 3D bounding box. The frequency of occurrence is expressed as ; The following two methods are used to generate virtual error 3D boxes: i) Change the category of the 3D bounding box within the initial annotation: When the category of the actual object within the 3D bounding box does not match the category of the 3D bounding box annotation, the 3D bounding box is considered incorrect, and the category of the 3D bounding box within part of the initial annotation is changed. For category To obtain an incorrect 3D bounding box, where When the annotation category of the 3D bounding box is At that time, according to the probability distribution ,in For category The frequency of occurrence is used to select the changed category. ; ii) Fake 3D bounding box: When a non-target object in the background is bounded, the 3D bounding box is considered to be incorrect, and a portion of the background object is randomly bounded to create a fake 3D bounding box.

4. The unsupervised multimodal 3D bounding box error detection system according to claim 1, characterized in that, During training, all 3D bounding boxes in the initial annotation are considered correct, while generated virtual erroneous 3D bounding boxes are considered incorrect. The training loss function is as follows: in, Represents the set of 3D bounding box annotations. Represents 3D frame Is it wrong? Represents 3D frame That's wrong. Represents 3D frame The error score.

5. An unsupervised multimodal 3D bounding box error detection method, characterized in that, include: Step 1: Generate a virtual 3D error box; Step 2: Train an end-to-end error detector to regress the error score; Step 3: Input the initial annotations into the error detector to obtain the error score of each 3D box. By using a threshold τ, the error score of the 3D box is mapped to 0 or 1, where 0 represents correct and 1 represents error. The end-to-end error detector includes a feature extractor and an error score generator; The feature extractor extracts from RGB images and lidar point cloud The feature extractor extracts feature representations of 3D bounding boxes, and extracts three types of features for the 3D bounding boxes: a) Point cloud features of the region of interest To extract local features from the LiDAR point cloud within the 3D bounding box, we first obtain the point cloud within the region of interest of the 3D bounding box from the LiDAR point cloud, denoted as . The focus will be on point clouds within the region. As a 3D point cloud feature extraction network The input and output are ,in The feature dimension of the output; b) Image features of the region of interest Project the 3D bounding box onto the RGB image and find the minimum bounding rectangle of the corresponding 2D 3D bounding box. The region within this minimum bounding rectangle is taken as the region of interest of the image, and the image within the region of interest is denoted as . The image will be focused on within the area. As an image feature extraction network The input and output are ,in The feature dimension of the output; c) Focus on the integration characteristics of the region To deeply fuse multimodal features, RGB images The input is fed into an image semantic segmentation network to generate high-level pixel semantic features with global semantic information. ; to point clouds The input is fed into a 3D segmentation network to generate high-level pointwise features with global semantic information. ,in The input images are respectively Width and height; This represents the total number of points in the point cloud. as well as The feature dimension of the output; By utilizing the mapping relationship between points in a point cloud and image pixels, semantic features of the image can be extracted. Convert to Image features Point cloud features By performing point-by-point connections, a high-level fused point cloud is obtained. Each point in the fused point cloud contains global information about both the point cloud and the image; To obtain the fusion features of the region of interest, from the fusion point cloud The point cloud within the 3D bounding box of interest is obtained and denoted as... The focus will be on point clouds within the region. As a 3D point cloud feature extraction network The input and output are ,in The feature dimension of the output; The features extracted by the feature extractor are concatenated with the inherent features of the 3D bounding box to generate the final features. , will feature The input is fed into an error score generator, which employs a standard multilayer perceptron architecture. In each layer, the generator accepts the category features of the 3D bounding box as additional input; that is, the concatenation of the output of the previous layer and the category features serves as the input to the next layer. As a dimension of motion characteristics, For the dimension of shape features, Number of categories; In the error score generator, each layer is followed by a batch normalization layer and a dropout layer to reduce overfitting and enhance the model's generalization ability. The activation function used is LeakyReLU, which provides a gradient when the input is negative to avoid the gradient vanishing problem. The output of the MLP is input into the Sigmoid activation function to obtain a scalar between 0 and 1. This scalar represents the error score of the 3D bounding box; the closer it is to 1, the greater the degree of error.

6. The unsupervised multimodal 3D bounding box error detection method according to claim 5, characterized in that, In the multimodal fusion process, points in the LiDAR point cloud are projected onto RGB image pixels, thereby establishing a mapping relationship between points in the point cloud and pixels in the image. The mapping relationship between points in the 3D point cloud and pixels in the 2D image is established through projection geometry, thus transforming the points in the LiDAR... Projecting onto an RGB image, the expression is: in, It is a three-dimensional point Pixel coordinates projected onto the RGB image It is the camera's inherent calibration matrix. and It is the rotation matrix and translation vector that transforms a 3D point from the lidar coordinate system to the camera coordinate system.

7. The unsupervised multimodal 3D bounding box error detection method according to claim 6, characterized in that, Step 1 includes: Let the set of 3D bounding box categories be This includes all possible categories of objects within the 3D bounding box. The annotation results summarize the category of each 3D bounding box. The frequency of occurrence is expressed as ; The following two methods are used to generate virtual error 3D boxes: i) Change the category of the 3D bounding box within the initial annotation: When the category of the actual object within the 3D bounding box does not match the category of the 3D bounding box annotation, the 3D bounding box is considered incorrect, and the category of the 3D bounding box within part of the initial annotation is changed. For category To obtain an incorrect 3D bounding box, where When the annotation category of the 3D bounding box is At that time, according to the probability distribution ,in For category The frequency of occurrence is used to select the changed category. ; ii) Fake 3D bounding box: When a non-target object in the background is bounded, the 3D bounding box is considered to be incorrect, and a portion of the background object is randomly bounded to create a fake 3D bounding box.

8. The unsupervised multimodal 3D bounding box error detection method according to claim 5, characterized in that, During training, all 3D bounding boxes in the initial annotation are considered correct, while generated virtual erroneous 3D bounding boxes are considered incorrect. The training loss function is as follows: in, Represents the set of 3D bounding box annotations. Represents 3D frame Is it wrong? Represents 3D frame That's wrong. Represents 3D frame The error score.

Images