Active learning method for multi-view 3D target detection in autonomous driving scenarios

Abstract

The application discloses a kind of multi-view 3D target detection active learning method for automatic driving scene, mainly includes the following steps: obtaining automatic driving all-around view data set, pre-training detector with height prediction network, based on the height guided adversarial active learning method obtains the height distribution of two data sources in bird's eye view space and trains variation auto-encoder and discriminator, by the inference of trained discriminator, the representative score of data source is obtained;By two kinds of distance, the diversity score is calculated to obtain the spatiotemporal diversity score, according to the overall information amount score, the sample candidate set to be labeled is constructed by screening sample, and finally, the final labeled sample set is obtained according to the balanced sampling method based on class.The present application provides value judgment for the mass of samples to be labeled in the automatic driving scene, so as to adaptively select the sample with more information amount for training, and further improve the model performance.In addition, the present application has strong generalization, and has broad application prospect.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, and particularly relates to a multi-view 3D target detection active learning method for an automatic driving scene. BACKGROUND

[0002] The rise of autonomous driving technology marks a profound transformation in the field of transportation. Over the past few decades, with the rapid development of computer science and artificial intelligence, autonomous driving technology has evolved rapidly. The emergence of this technology is not only due to the improvement of computing power, but also benefits from breakthroughs in sensor technology, machine learning and deep learning.

[0003] The 3D target detection task aims to identify and locate traffic objects in 3D space, and perceive the surrounding target objects, which is a key task in the field of autonomous driving. According to the current mainstream method of sensor, it can be divided into LiDAR-based and vision-based detection methods. Compared with LiDAR-based methods, vision-based 3D object detection has the advantage of low deployment cost, and is a topic of concern in recent years.

[0004] Active learning is one of the important techniques to reduce the cost of data labeling, which iteratively selects the most valuable samples from the unlabeled data pool for labeling and training, and improves the model performance as much as possible under limited labeling cost. It has been widely used in traditional computer vision tasks such as image classification, object detection and semantic segmentation.

[0005] However, the current vision-based 3D target detection method relies on a large number of accurate 3D labels used in the training phase, which is extremely time-consuming and expensive compared to 2D detection. In contrast, it is very easy to obtain autonomous driving data, and a large number of video clips can be collected by a collection car during driving. The high cost of labeling greatly hinders the development of vision-based autonomous driving perception algorithms. Therefore, 3D target detection under limited labeling cost is a great challenge. SUMMARY

[0006] The purpose of the present application is to solve the problems existing in the prior art and provide a multi-view 3D target detection active learning method for an automatic driving scene.

[0007] In order to achieve the above-mentioned application purpose, the present application specifically adopts the following technical solutions:

[0008] A multi-view 3D target detection active learning method for an automatic driving scene, comprising the following steps:

[0009] S1, obtaining a labeled autonomous driving surround view dataset and an unlabeled autonomous driving surround view dataset;

[0010] S2, pre-training a detector on a labeled autonomous driving surround view dataset, the detector being obtained by additionally adding a height prediction network as an additional auxiliary task in a bird's eye view based 3D object detection model, the height prediction network being added after a bird's eye view encoder of the 3D object detection model and constituting a parallel processing branch with a bird's eye view decoder of the 3D object detection model, the height prediction network being used to predict a height distribution of each grid foreground part under a bird's eye view, the predicted height distribution being used for subsequent active learning;

[0011] S3, based on the height guided adversarial active learning method, first inputting samples in the labeled autonomous driving surround view dataset into the pre-trained detector to obtain first bird's eye view space height distributions, then inputting samples in the unlabeled autonomous driving surround view dataset into the pre-trained detector to obtain second bird's eye view space height distributions, training a variational autoencoder and a discriminator based on the obtained first bird's eye view space height distributions and the second bird's eye view space height distributions, the variational autoencoder and the discriminator each corresponding to a loss function for parameter updating; when training the discriminator, if the judgment result output by the discriminator is that the current sample belongs to the unlabeled autonomous driving surround view dataset, the current sample is taken as a representative sample;

[0012] S4, after the training of the variational autoencoder and the discriminator is completed, inputting the representative sample into the trained variational autoencoder and discriminator, and taking the output of the trained discriminator as a data source representative score;

[0013] S5, for samples in the unlabeled autonomous driving surround view dataset, calculating a temporal distance between the sample and the sample in the labeled autonomous driving surround view dataset closest in distance according to the frame number of the sample in the video sequence, taking the temporal distance calculation result as a temporal diversity score, calculating a spatial distance between the sample and the sample in the labeled autonomous driving surround view dataset closest in distance according to the shooting location of the sample in the high-precision map, taking the spatial distance calculation result as a spatial diversity score, and adding and averaging the temporal diversity score and the spatial diversity score to obtain a spatio-temporal diversity score;

[0014] S6, weighting and summing the data source representative score and the spatio-temporal diversity score to obtain a total information amount score of the sample, sorting the samples in the unlabeled autonomous driving surround view dataset according to the total information amount score, and screening out a batch of samples in the unlabeled autonomous driving surround view dataset with the highest total information amount score to construct a candidate set of samples to be labeled;

[0015] S7, according to the category-based balanced sampling method, first, the instance-level category score of the detection frame is predicted by the trained detector, then the instance-level category score is dimensionally reduced, the dimensional reduction method adopts class-by-class summation, the sample-level category score is obtained after dimensional reduction, a selected sample score vector is constructed in advance, the selected sample score vector is iteratively updated until the pre-set labeling upper limit is reached, and the final sample set to be labeled is obtained; in each labeling, the category with the least number of samples in the selected sample score vector is taken as the category to be updated, the selected sample is selected from the candidate set of samples to be labeled, and the selected sample satisfies that the selected sample is the sample with the highest score corresponding to the category to be updated in the sample-level category score, and the sample-level category score of the selected sample is added to the selected sample score vector as a new selected sample score vector.

[0016] On the basis of the above scheme, each step can be implemented in the following preferred specific manner.

[0017] As a preferred, in step S1, the labeled autonomous driving surround view data set and the unlabeled autonomous driving surround view data set each contain multiple multi-view videos of autonomous driving scenes, each frame in the multi-view video is a sample, each sample has a multi-view image, an internal parameter of a camera, an external parameter of the camera, a timestamp of the sample in the multi-view video, and position information of the sample in a high-precision map, the labeled autonomous driving surround view data set additionally has data labeling, each data labeling has a spatial position, a size, a direction, a speed and a category of a 3D detection frame.

[0018] As a preferred, in step S2, the height prediction network is a multi-layer perception (MLP).

[0019] Preferably, in step S3, in each iterative round of the variational autoencoder and the discriminator training process, the first bird's-eye view spatial height distribution is input into the encoder of the variational autoencoder to obtain a first latent vector, the first latent vector is input into the decoder of the variational autoencoder for reconstruction, and a reconstructed first height distribution is obtained, the posterior distribution of the first latent vector is input into the discriminator to obtain a first judgment result, the second bird's-eye view spatial height distribution is input into the encoder of the variational autoencoder to obtain a second latent vector, the second latent vector is input into the decoder of the variational autoencoder for reconstruction, and a reconstructed second height distribution is obtained, the posterior distribution of the second latent vector is input into the discriminator to obtain a second judgment result, and the first latent vector is calculated based on the posterior distribution of the first latent vector and the unit Gaussian prior distribution. A KL divergence loss is used. A second KL divergence loss is calculated based on the posterior distribution of the second latent vector and the unit Gaussian prior distribution. A reconstruction learning loss of the variational autoencoder is constructed according to the first KL divergence loss, the second KL divergence loss, the reconstructed first height distribution, and the reconstructed second height distribution. An adversarial learning loss of the variational autoencoder is constructed based on the first judgment result and the second judgment result. The reconstruction learning loss and the adversarial learning loss are added as the final loss of the variational autoencoder. The final loss of the discriminator is constructed based on the first judgment result and the second judgment result. The final loss of the variational autoencoder is minimized to update the parameters of the variational autoencoder. The final loss of the discriminator is minimized to update the parameters of the discriminator. Until a preset number of iterations is reached, a trained variational autoencoder and discriminator are obtained.

[0020] As a preference, the final loss L of the variational autoencoder VAE for:

[0021]

[0022] Where, represents the spatial height distribution of the first bird's-eye view, represents the spatial height distribution of the second bird's-eye view; z L represents the first latent vector, z U represents the second latent vector; p θ represents the decoder of the variational autoencoder, q φ represents the encoder of the variational autoencoder; Disc represents the discriminator; represents the reconstructed first height distribution, represents the reconstructed second height distribution; β represents the preset first hyperparameter; p(z) represents a unit Gaussian prior distribution; represents the posterior distribution of the first latent vector, represents the posterior distribution of the second latent vector; represents the first KL divergence loss, represents the second KL divergence loss; represents the first judgment result, represents the second judgment result; represents the reconstruction learning loss of the variational autoencoder; represents the adversarial learning loss of the variational autoencoder.

[0023] As preferred, the final loss of the discriminator is:

[0024]

[0025] As preferred, if the first judgment result is 0, it indicates that the sample corresponding to the first hidden vector is from the labeled autonomous driving surround view dataset, and if the first judgment result is 1, it indicates that the sample corresponding to the first hidden vector is from the unlabeled autonomous driving surround view dataset; if the second judgment result is 0, it indicates that the sample corresponding to the second hidden vector is from the labeled autonomous driving surround view dataset, and if the second judgment result is 1, it indicates that the sample corresponding to the second hidden vector is from the unlabeled autonomous driving surround view dataset.

[0026] As preferred, in step S5, for the sample X U from the unlabeled autonomous driving surround view dataset, the spatiotemporal diversity score is calculated as:

[0027]

[0028]

[0029] In the formula, represents the spatial diversity score, represents the temporal diversity score; X L′ represents the sample X U closest to the sample X s from the labeled autonomous driving surround view dataset; d U (X L′ , X U ) represents the spatial distance between the sample X L′ from the unlabeled autonomous driving surround view dataset and the sample X t from the labeled autonomous driving surround view dataset; d U (X L′ , X U ) represents the temporal distance between the sample X L′ from the unlabeled autonomous driving surround view dataset and the sample X U from the labeled autonomous driving surround view dataset; id(X Uframe number in the video sequence; id(X L′ represents a sample X in the annotated autonomous driving surround view dataset L′ frame number in the video sequence; λ is a preset second hyperparameter.

[0030] Compared with the prior art, the present application has the following beneficial effects:

[0031] 1) The present application is the first vision-based surround view 3D target detection active learning framework. By decoupling the sample information of autonomous driving surround view camera data, a two-stage active learning algorithm is designed from three aspects of data prior, bird's eye view features and detection results. By introducing data source representative score and data spatiotemporal diversity score, the present application realizes diversity sampling in sample space and bird's eye view space. By introducing class balanced sampling, balance in class space is realized.

[0032] 2) The present application provides value judgment for massive samples to be labeled in the autonomous driving scene, so as to adaptively select samples with more information for training, thereby improving the model performance. In the nuScenes dataset, 50% of the samples can achieve comparable results to the full dataset training. In addition, the present application has strong generalization, and good performance can also be achieved on different datasets and different 3D detection models. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 is a schematic diagram of the steps of the method of the present application;

[0034] Figure 2 is a schematic diagram of the overall architecture of the method of the present application;

[0035] Figure 3 is a network architecture diagram of the detector of the present application. DETAILED DESCRIPTION

[0036] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the scope of the present application, so the present application is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present application can be combined accordingly without conflict.

[0037] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0038] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned multi-view 3D object detection active learning method for autonomous driving scenarios includes the following steps S1 to S7. The specific implementation process is described below.

[0039] In a preferred embodiment of the present invention, a multi-view 3D object detection active learning method for autonomous driving scenarios is provided. This method helps the 3D object detection model achieve better performance with limited labeling costs by selecting the most informative samples in the autonomous driving scene. Specifically, the present invention improves the performance of the 3D object detection model under data labeling constraints by decoupling the information content of multi-view samples in the autonomous driving scene and selecting unlabeled samples for labeling in stages. It should be noted that the 3D object detection model in the present invention can be any detection model based on vision and using bird's-eye view features.

[0040] like Figure 1 As shown, in a preferred embodiment of the present invention, the active learning method for multi-view 3D object detection in an autonomous driving scenario specifically includes the following steps:

[0041] S1. Obtain a labeled autonomous driving surround view dataset and an unlabeled autonomous driving surround view dataset.

[0042] It should be noted that in step S1 of the present invention, both the labeled autonomous driving surround view dataset and the unlabeled autonomous driving surround view dataset contain multi-perspective videos of multiple autonomous driving scenes. Each frame in the multi-perspective video is a sample. Each sample carries a multi-perspective image, the intrinsic parameters of the camera, the extrinsic parameters of the camera, the timestamp of the sample in the multi-perspective video, and the location information of the sample in the high-precision map. The labeled autonomous driving surround view dataset additionally carries data annotations, and each data annotation carries the spatial position, size, direction, speed, and category of the 3D detection box.

[0043] In this embodiment, for a labeled autonomous driving surround view dataset, which contains multiple-view videos of several autonomous driving scenes, each frame in the multi-view video is a sample. A sample in the labeled autonomous driving surround view dataset can be represented as X L ={I,K,E,t L ,s Lwherein I is a multi-view image, K is an internal parameter of a camera, E is an external parameter of the camera, t L is a sample X L is a timestamp in a multi-view video, s L is a sample X L is position information in a high-definition map. In addition, the annotated autonomous driving surround view dataset additionally has data annotation. For the data annotation Y, the following contents should be included: Y = {b, y}, wherein the spatial position, size, direction and velocity of the 3D detection box are stored in the vector b = [x, y, z, l, w, h, θ, v x ,v y ]. (x, y, z) is the spatial position coordinate of the 3D detection box, x represents the X-axis coordinate in the spatial position coordinate, y represents the Y-axis coordinate in the spatial position coordinate, and z represents the Z-axis coordinate in the spatial position coordinate; l, w and h represent the size of the 3D detection box, l represents the length of the 3D detection box, w represents the width of the 3D detection box, and h represents the height of the 3D detection box; v x ,v y represent the velocities of the 3D detection box along the X-axis and the Y-axis respectively; θ represents the direction of the 3D detection box; and y represents the category of the 3D detection box, including motor vehicles, pedestrians, bicycles and the like.

[0044] In the embodiment, for the unannotated autonomous driving surround view dataset, wherein a plurality of multi-view videos of autonomous driving scenes are included, each frame in the multi-view video is a sample. A sample in the unannotated autonomous driving surround view dataset can be represented as X U = {I, K, E, t U ,s U}, wherein t U is a sample X U is a timestamp in a multi-view video, s U is a sample X U is position information in a high-definition map.

[0045] S2, pre-training a detector on the annotated autonomous driving surround view dataset, the detector being obtained by additionally adding a height prediction network as an additional auxiliary task in the bird's eye view based 3D target detection model, the height prediction network being added after the bird's eye view encoder of the 3D target detection model and constituting a parallel processing branch with the bird's eye view decoder of the 3D target detection model, the height prediction network being used for predicting the height distribution of each grid foreground part under the bird's eye view, and the predicted height distribution being used for subsequent active learning.

[0046] It should be noted that, in step S2 of the present application, as Figure 3As shown, the detector of the application is constructed by additionally adding a height prediction network in the form of a prediction head at the aerial view encoder. The detector is pre-trained on the grid height under the aerial view perspective, obtaining the height distribution of the foreground part in each grid, and the prediction value output by the height prediction network is H BEV = [μ h′ , μ l′ , σ h′ , σ l′ ], where μ h′ and σ h′ correspond to the mean and standard deviation of the object center height h' in each grid, respectively, and μ l′ and σ l′ correspond to the mean and standard deviation of the object height value l' in each grid, respectively.

[0047] In addition, it should be noted that the structure of the height prediction network can be selected according to actual needs, and can obtain the above prediction value form, and can be added after the aerial view encoder and constitute a parallel processing branch with the aerial view decoder. In the embodiment, the structure of the height prediction network is a multi-layer perceptron (MLP), and since the predicted standard deviation is non-negative, the height prediction network predicts the logarithmic value of the standard deviation.

[0048] S3, based on the height guided adversarial active learning method, the samples in the labeled autonomous driving surround view data set are input into the pre-trained detector to obtain the first aerial view space height distribution, and the samples in the unlabeled autonomous driving surround view data set are input into the pre-trained detector to obtain the second aerial view space height distribution, the first aerial view space height distribution and the second aerial view space height distribution are used to train the variational autoencoder and the discriminator, the variational autoencoder and the discriminator each correspond to a loss function for parameter updating, when training the discriminator, if the judgment result output by the discriminator is that the current sample belongs to the unlabeled autonomous driving surround view data set, the current sample is taken as a representative sample.

[0049] It should be noted that in step S3 of the application, the height guided adversarial active learning method includes reconstruction learning and adversarial learning. For samples X L from the labeled autonomous driving surround view data set and samples X U from the unlabeled autonomous driving surround view data set, the pre-trained detector is used for inference respectively to obtain the aerial view space height distribution of the two data sources: the first aerial view space height distribution and the second aerial view space height distribution During the reconstruction learning phase, β-VAE is used to learn implicit representations. The spatial height distributions of the bird's-eye view images from two data sources are used as input. The encoder of the variational autoencoder converts each of these spatial height distributions into latent variables. The decoder of the variational autoencoder then decodes the latent variables and reconstructs the image. During the adversarial learning phase, adversarial training is used with the latent variables as input. β-VAE is used to model the height distributions of the different data sources as similar latent variable distributions. The discriminator is used to determine which data source the latent variables come from (either a labeled autonomous driving surround view dataset or an unlabeled autonomous driving surround view dataset). Adversarial training of the variational autoencoder and the discriminator enables the discriminator to robustly discriminate the source of data.

[0050] It should be noted that in each iteration of the variational autoencoder and the discriminator training process in step S3 of the present invention, the first bird's-eye view spatial height distribution Input into the encoder of the variational autoencoder to obtain the first latent vector z L , input the first latent vector into the decoder of the variational autoencoder for reconstruction, and obtain the reconstructed first height distribution The first latent vector z L The posterior distribution of Input into the discriminator to get the first judgment result The second bird's-eye view spatial height distribution Input into the encoder of the variational self-encoder to obtain the second latent vector z U , input the second latent vector into the decoder of the variational autoencoder for reconstruction, and obtain the reconstructed second height distribution The second latent vector z U The posterior distribution of Input into the discriminator to get the second judgment result Posterior distribution based on the first latent vector And the unit Gaussian prior distribution p(z) calculates the first KL divergence loss, based on the posterior distribution of the second latent vector And the unit Gaussian prior distribution p(z) calculates the second KL divergence loss, and constructs the reconstruction learning loss of the variational autoencoder according to the first KL divergence loss, the second KL divergence loss, the reconstructed first height distribution and the reconstructed second height distribution Constructing the adversarial learning loss of the variational autoencoder based on the first judgment result and the second judgment result The reconstruction learning loss and the adversarial learning loss are added as the final loss of the variational autoencoder, and the final loss of the discriminator is constructed based on the first judgment result and the second judgment result. The final loss of the variational autoencoder is updated to minimize parameters of the variational autoencoder, the final loss of the discriminator is updated to minimize parameters of the discriminator until a preset iteration round is reached, and a trained variational autoencoder and discriminator are obtained.

[0051] Further, if the first judgment result is 0, it indicates that the sample corresponding to the first latent vector is from the labeled autonomous driving surround view dataset, if the first judgment result is 1, it indicates that the sample corresponding to the first latent vector is from the unlabeled autonomous driving surround view dataset; if the second judgment result is 0, it indicates that the sample corresponding to the second latent vector is from the labeled autonomous driving surround view dataset, if the second judgment result is 1, it indicates that the sample corresponding to the second latent vector is from the unlabeled autonomous driving surround view dataset.

[0052] Further, the reconstruction learning loss of the variational autoencoder is

[0053]

[0054] In the formula, z represents the first latent vector, z represents the second latent vector, p(z) represents a unit Gaussian prior distribution, and p represents a preset first hyperparameter. represents the first bird's eye view space height distribution, represents the second bird's eye view space height distribution; z represents the first latent vector, and z represents the second latent vector. L represents the first latent vector, z U represents the second latent vector; p θ represents the decoder of the variational autoencoder, q φ represents the encoder of the variational autoencoder; represents the reconstructed first height distribution, represents the reconstructed second height distribution; β represents a preset first hyperparameter; and p(z) represents a unit Gaussian prior distribution. represents the posterior distribution of the first latent vector, represents the posterior distribution of the second latent vector. represents the first KL divergence loss, represents the second KL divergence loss.

[0055] The adversarial learning loss of the variational autoencoder is

[0056]

[0057] In the formula, Disc represents the discriminator, represents the first judgment result, represents the second judgment result.

[0058] Therefore, the final loss of the variational autoencoder is

[0059] The final loss of the discriminator ​​for:

[0060]

[0061] It should also be noted that, in the present invention, the first hyperparameter β can be set by those skilled in the art according to actual needs. In this embodiment, the first hyperparameter β is set to 1.

[0062] S4: After the variational autoencoder and the discriminator are trained, the representative samples are input into the trained variational autoencoder and discriminator, and the output of the trained discriminator is used as the data source representative score.

[0063] It should be noted that in step S4 of the present invention, the data source representativeness score is calculated as follows:

[0064]

[0065] S5. For samples in the unlabeled autonomous driving surround view dataset, calculate the temporal distance between them and the nearest labeled sample in the autonomous driving surround view dataset based on their frame number in the video sequence, and use the calculated temporal distance as the temporal diversity score. Calculate the spatial distance between them and the nearest labeled sample in the autonomous driving surround view dataset based on their shooting location in the high-precision map, and use the calculated spatial distance as the spatial diversity score. Add the temporal diversity score and the spatial diversity score and then take the average to obtain the spatiotemporal diversity score.

[0066] It should be noted that, in the present invention, a sample sampling method based on spatiotemporal diversity is proposed, such as Figure 2 As shown in Figure 2, two distance calculation methods are used to construct the diversity scores corresponding to each distance, namely the temporal diversity score and the spatial diversity score, and then the spatiotemporal diversity score is constructed. Specifically, for the sample X from the unlabeled autonomous driving surround view dataset, U , spatiotemporal diversity score The calculation method is:

[0067]

[0068] Where, represents the spatial diversity score, In this embodiment, both the spatial diversity score and the temporal diversity score represent the sample X in the unlabeled autonomous driving surround view dataset. U To distance X U Sample X from the most recent labeled autonomous driving surround view dataset L′ The normalized distance, spatial diversity score, and temporal diversity score are calculated as follows.

[0069] spatial diversity score The calculation method is:

[0070]

[0071] In the formula, X L′ represents the spatial distance between the sample X U in the unlabeled autonomous driving surround view data set and the nearest sample in the labeled autonomous driving surround view data set; d s (X U ,X L′ ) represents the sample X U in the unlabeled autonomous driving surround view data set and the sample X L′ in the labeled autonomous driving surround view data set.

[0072] In the present application, d s (·,·) represents a method of measuring the distance between samples in space. In the present embodiment, the spatial distance between samples is calculated by using a manifold distance based on a KNN graph, and the KNN graph is constructed by selecting the K nearest samples in the labeled autonomous driving surround view data set to each sample in the unlabeled autonomous driving surround view data set, and the edge weight is the L2 distance. The KNN graph is constructed offline, and the spatial distance d s (X U ,X L′ ) between samples is represented by the shortest path on the KNN graph, and is set to infinity if it is unreachable.

[0073] temporal diversity score The calculation method is:

[0074]

[0075] In the formula, d t (X U ,X L′ ) represents the temporal distance between the sample X U in the unlabeled autonomous driving surround view data set and the sample X L′ in the labeled autonomous driving surround view data set, and the calculation method is:

[0076]

[0077] In the formula, id(X U ) represents the frame number of the sample X U in the video sequence; and id(X L′ ) represents the sample XL′ frame number in the video sequence; and λ is a preset second hyper-parameter.

[0078] In the embodiment, d t (·,·) represents a distance measurement manner between samples in time sequence, and the calculation manner of the time sequence distance is based on L2 distance.

[0079] In addition, it should be noted that in the present application, the second hyper-parameter λ can be set by those skilled in the art according to actual needs. In the embodiment, the second hyper-parameter λ is set to

[0080] S6, the data source representative score and the spatiotemporal diversity score are weighted and summed as the overall information amount score of the sample, the samples in the unlabeled automatic driving surround view data set are sorted according to the overall information amount score, and a batch of samples in the unlabeled automatic driving surround view data set with the highest overall information amount score are selected to construct a candidate set of samples to be labeled.

[0081] It should be noted that in step S6 of the present application, the data source representative score and the spatiotemporal diversity score are weighted as the overall information amount score of the sample, which is used as the basis for screening the samples in the unlabeled automatic driving surround view data set. The specific calculation manner of the overall information amount score is as follows:

[0082]

[0083] In the formula, α i = α0- δi represents the weighted hyper-parameter when the sample in the unlabeled automatic driving surround view data set is selected and screened in the i-th round; α0 represents a preset third hyper-parameter, and δi represents a preset fourth hyper-parameter.

[0084] In the present application, the weighted hyper-parameter α i can be adaptively adjusted based on the current amount of labeled sample data. In the embodiment, the third hyper-parameter α0 and the fourth hyper-parameter δi can be set by those skilled in the art according to actual needs. Correspondingly, the third hyper-parameter α0 is set to 0.4, and the fourth hyper-parameter δi is set to 0.1.

[0085] S7, according to the balanced sampling method based on the category, the instance level category score of the detection frame is first predicted by the trained detector, then the instance level category score is dimensionally reduced, the dimension reduction method adopts class-by-class summation, the sample level category score is obtained after dimension reduction, a selected sample score vector is constructed in advance, the selected sample score vector is iteratively updated until a pre-set labeling upper limit is reached, and a final to-be-labeled sample set is obtained; in each labeling, the category with the least number of samples in the selected sample score vector is taken as the to-be-updated category, the selected sample is selected from the to-be-labeled sample candidate set, and the selected sample satisfies that the selected sample is the sample with the highest score corresponding to the to-be-updated category in the sample level category score, and the sample level category score of the selected sample is added to the selected sample score vector as a new selected sample score vector.

[0086] The application will be described below by a specific example to show the application effect of the multi-view 3D target detection active learning method for automatic driving scene described in S1-S7 of the above embodiment on a specific data set, so as to understand the essence of the application.

[0087] Embodiment

[0088] The implementation process of the multi-view 3D target detection active learning method for automatic driving scene in this embodiment has been described before, and the specific steps will not be described in detail, and only the effect of the case data will be described below. The application is implemented on an automatic driving scene data set nuScenes with true value labeling, and the following is an introduction to the automatic driving scene data set nuScenes.

[0089] nuScenes data set: The data set is composed of 1000 video clips, with a total of 40158 annotated key frames, and the training set, validation set and test set are divided as follows: the training set contains 700 video clips with a total of 28130 frames, the validation set contains 150 video clips with a total of 6019 frames, and the test set contains 150 video clips with a total of 6009 frames. Since there is no public annotation data in the test set, the test set is not used in the implementation process. By dividing the annotation set and the unannotated set in the training set to simulate the unannotated condition, the annotation of the unannotated sample is invisible. The nuScenes data set contains more than 1.4 million annotated 3D frame data.

[0090] This embodiment is trained on the annotated set in the nuScenes training set and evaluated on the nuScenes validation set. The evaluation index mainly considers the NDS index and mAP index provided by the nuScenes data set, and the calculation method of the index is:

[0091]

[0092] In the formula, mAP is one of the evaluation indexes commonly used in target detection, used to measure the accuracy and performance of the model. The calculation method of mAP is to calculate the arithmetic mean of the AP (Average Precision) of all categories at different thresholds, where AP is the precision calculation of a single category. Unlike the IoU threshold in 2D detection, the threshold here is the offset distance of the center point of the 3D detection box, which is also one of the main evaluation indexes. In the formula, mTP includes mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE), and mean Average Attribute Error (mAAE).

[0093] The 3D target detection model selected in this embodiment is BEVFormer, which can be specifically referred to in the prior art document: Li, Zhiqi, et al. "Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

[0094] The iterative training process of this embodiment is as follows:

[0095] (1) Randomly select 10% of the samples (2813) as the initial annotation set based on the nuScenes dataset;

[0096] (2) Use the annotation set to train the detector, and according to the trained detector, use the method of the present application to select the most informative samples that need to be annotated from the remaining unannotated data. Here, the number of selected samples is set to 10% (2813);

[0097] (3) Label the selected samples and add them to the annotation set to obtain a new annotation set;

[0098] (4) Repeat operations (2) and (3) until the amount of annotated data reaches the upper limit, which is set to 50% in this embodiment.

[0099] The detection accuracy of the above iterative training process in this embodiment is shown in Table 1. The higher the indicators NDS and mAP, the better, and the lower the indicators mATE, mASE, mAOE, mAVE and mAAE, the better. In this task, the indicators of more concern are NDS and mAP.

[0100] Table 1

[0101]

[0102] In the detection results of the above iterative training, the method of the present application can significantly improve the model performance under limited data labeling. In addition, it can be observed that the method under 50% labeling restriction achieves comparable performance to that under 100% full data training.

[0103] The above-described embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical solutions obtained by equivalent replacement or equivalent transformation shall fall within the protection scope of the present application.

Claims

1. An active learning method for multi-view 3D object detection in autonomous driving scenarios, characterized by: The following steps are involved: S1. Obtain a labeled autonomous driving surround view dataset and an unlabeled autonomous driving surround view dataset; S2. Pre-training a detector on a labeled autonomous driving surround view dataset. The detector is obtained by adding a height prediction network as an additional auxiliary task to a 3D object detection model based on a bird's-eye view. The height prediction network is added after the bird's-eye view encoder of the 3D object detection model and forms a parallel processing branch with the bird's-eye view decoder of the 3D object detection model. The height prediction network is used to predict the height distribution of each grid foreground part in the bird's-eye view. The predicted height distribution is used for subsequent active learning. S3. A height-guided adversarial active learning method first inputs samples from a labeled autonomous driving surround view dataset into a pre-trained detector to obtain a first bird's-eye view spatial height distribution. Samples from an unlabeled autonomous driving surround view dataset are then input into the pre-trained detector to obtain a second bird's-eye view spatial height distribution. A variational autoencoder and a discriminator are trained based on the obtained first and second bird's-eye view spatial height distributions. The variational autoencoder and the discriminator each correspond to a loss function for parameter updating. When training the discriminator, if the discriminator outputs a judgment result that the current sample belongs to the unlabeled autonomous driving surround view dataset, the current sample is used as a representative sample. S4. After the variational autoencoder and the discriminator are trained, the representative samples are input into the trained variational autoencoder and the discriminator, and the output of the trained discriminator is used as the data source representative score; S5. For samples in the unlabeled autonomous driving surround view dataset, calculate the temporal distance between them and the nearest labeled sample in the autonomous driving surround view dataset based on their frame number in the video sequence. Use the calculated temporal distance as the temporal diversity score. Calculate the spatial distance between them and the nearest labeled sample in the autonomous driving surround view dataset based on their shooting location in the high-precision map. Use the calculated spatial distance as the spatial diversity score. Add the temporal diversity score and the spatial diversity score and then take the average to obtain the spatiotemporal diversity score. S6. Take the weighted sum of the data source representativeness score and the spatiotemporal diversity score as the overall information content score of the sample, sort the samples in the unlabeled autonomous driving surround view dataset according to the overall information content score, and select a batch of samples in the unlabeled autonomous driving surround view dataset with the highest overall information content score to construct a candidate set of samples to be labeled; S7. First, the trained detector predicts the instance-level category score of the detection box according to the category-based balanced sampling method. Then, the instance-level category score is reduced in dimension by summing up the scores class by class. After the dimension reduction, the sample-level category score is obtained. A selected sample score vector is pre-constructed. The selected sample score vector is continuously iterated and updated until a pre-set labeling upper limit is reached, thereby obtaining the final set of samples to be labeled. In each labeling, the category with the least number of samples in the selected sample score vector is taken as the category to be updated, and the selected samples are selected from the candidate set of samples to be labeled, and the selected samples are the samples with the highest scores corresponding to the category to be updated in the sample-level category scores. The sample-level category scores of the selected samples are added to the selected sample score vector as the new selected sample score vector.

2. The active learning method for multi-view 3D object detection in autonomous driving scenarios according to claim 1, characterized in that: In step S1, both the labeled autonomous driving surround view dataset and the unlabeled autonomous driving surround view dataset contain multi-view videos of multiple autonomous driving scenes. Each frame in the multi-view video is a sample. Each sample carries a multi-view image, the camera's intrinsic parameters, the camera's extrinsic parameters, the timestamp of the sample in the multi-view video, and the location information of the sample in the high-precision map. The labeled autonomous driving surround view dataset additionally carries data annotations, and each data annotation carries the spatial position, size, direction, speed, and category of the 3D detection box.

3. The active learning method for multi-view 3D object detection in autonomous driving scenarios according to claim 1, characterized in that: In step S2, the height prediction network is a multi-layer perceptron MLP.

4. The active learning method for multi-view 3D object detection in autonomous driving scenarios according to claim 1, wherein: In step S3, in each iterative round of the variational autoencoder and the discriminator training process, the first bird's-eye view spatial height distribution is input into the encoder of the variational autoencoder to obtain a first latent vector, the first latent vector is input into the decoder of the variational autoencoder for reconstruction, and a reconstructed first height distribution is obtained. The posterior distribution of the first latent vector is input into the discriminator to obtain a first judgment result, the second bird's-eye view spatial height distribution is input into the encoder of the variational autoencoder to obtain a second latent vector, the second latent vector is input into the decoder of the variational autoencoder for reconstruction, and a reconstructed second height distribution is obtained. The posterior distribution of the second latent vector is input into the discriminator to obtain a second judgment result, and the first KL is calculated based on the posterior distribution of the first latent vector and the unit Gaussian prior distribution. Divergence loss, calculate the second KL divergence loss based on the posterior distribution of the second latent vector and the unit Gaussian prior distribution, construct the reconstruction learning loss of the variational autoencoder according to the first KL divergence loss, the second KL divergence loss and the reconstructed first height distribution and the reconstructed second height distribution, construct the adversarial learning loss of the variational autoencoder based on the first judgment result and the second judgment result, add the reconstruction learning loss and the adversarial learning loss as the final loss of the variational autoencoder, construct the final loss of the discriminator based on the first judgment result and the second judgment result, minimize the final loss of the variational autoencoder to update the parameters of the variational autoencoder, minimize the final loss of the discriminator to update the parameters of the discriminator, until the preset iteration rounds are reached, and the trained variational autoencoder and discriminator are obtained.

5. The active learning method for multi-view 3D object detection in autonomous driving scenarios according to claim 4, characterized in that: The final loss L of the variational autoencoder VAE for: Where, represents the spatial height distribution of the first bird's-eye view, represents the spatial height distribution of the second bird's-eye view; z L represents the first latent vector, z U represents the second latent vector; p θ represents the decoder of the variational autoencoder, q φ represents the encoder of the variational autoencoder; Disc represents the discriminator; represents the reconstructed first height distribution, represents the reconstructed second height distribution; β represents the preset first hyperparameter; p(z) represents a unit Gaussian prior distribution; represents the posterior distribution of the first latent vector, represents the posterior distribution of the second latent vector; represents the first KL divergence loss, represents the second KL divergence loss; Indicates the first judgment result, Indicates the second judgment result; represents the reconstruction learning loss of the variational autoencoder; represents the adversarial learning loss of the variational autoencoder.

6. The active learning method for multi-view 3D object detection in autonomous driving scenarios according to claim 5, characterized in that: The final loss of the discriminator for:

7. The active learning method for multi-view 3D object detection in autonomous driving scenarios according to claim 4, characterized in that: If the first judgment result is 0, it means that the sample corresponding to the first latent vector comes from a labeled autonomous driving surround view dataset. If the first judgment result is 1, it means that the sample corresponding to the first latent vector comes from an unlabeled autonomous driving surround view dataset. If the second judgment result is 0, it means that the sample corresponding to the second latent vector comes from a labeled autonomous driving surround view dataset. If the second judgment result is 1, it means that the sample corresponding to the second latent vector comes from an unlabeled autonomous driving surround view dataset.

8. The active learning method for multi-view 3D object detection in autonomous driving scenarios according to claim 1, characterized in that: In step S5, for the sample X from the unlabeled autonomous driving surround view dataset U , spatiotemporal diversity score The calculation method is: Where, represents the spatial diversity score, represents the temporal diversity score; X L′ Represents the sample X in the unlabeled autonomous driving surround view dataset U Samples from the most recent annotated autonomous driving surround view dataset; d s (X U ,X L′ ) represents the sample X in the unlabeled autonomous driving surround view dataset U and samples X in the labeled autonomous driving surround view dataset L′ The spatial distance between t (X U ,X L′ ) represents the sample X in the unlabeled autonomous driving surround view dataset U and samples X in the labeled autonomous driving surround view dataset L′ The temporal distance between id(X U ) represents the sample X in the unlabeled autonomous driving surround view dataset U Frame number in the video sequence; id(X L′ ) represents the sample X in the labeled autonomous driving surround view dataset L′ The frame number in the video sequence; λ is the second preset hyperparameter.

Images