A point cloud object detection domain adaptation method from simulated data to real data

Abstract

The application provides a point cloud target detection domain adaptation method from simulation data to real data, which comprises the following steps: obtaining a point cloud from a target domain; inputting the point cloud into a trained two-stage point cloud target detection model, wherein the two-stage point cloud target detection model outputs a final prediction result of each target object in the point cloud, and the final prediction result comprises a category result and a positioning result of the target object; wherein the two-stage point cloud target detection model comprises a one-stage detection model and a two-stage detection model, the one-stage detection model is used for processing the point cloud to output a feature map and a one-stage candidate box, and the two-stage detection model is used for processing the feature map and the candidate box to obtain the category of each target object in the point cloud and a corresponding two-stage detection box; a size parameter in a candidate box parameter of the one-stage candidate box is a fixed set of numerical values, and the two-stage detection box is obtained based on the one-stage candidate box.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and specifically to a point cloud target detection domain adaptation method from simulation data to real data. Background Technology

[0002] In recent years, autonomous driving technology has gradually become a hot research area across various industries. In autonomous driving, point clouds, as an important source of perception input, are widely used for environmental perception and obstacle detection. Among these, point cloud object detection is a key task in autonomous driving, aiming to accurately identify and locate different types of target objects on the road, such as vehicles and pedestrians.

[0003] In point cloud object detection tasks, a large amount of labeled data is typically required to train models and achieve good performance. However, acquiring large-scale labeled point cloud data is a time-consuming and expensive task. To address this issue, autonomous driving simulators have become an effective solution. An autonomous driving simulator is a tool based on computer graphics and virtual reality technology that can simulate real-world traffic scenarios and various environmental conditions. By generating and labeling large amounts of point clouds within the simulator, the labeled data needed to train point cloud object detection models can be obtained quickly and efficiently. However, due to the limitations of simulators in simulating real-world scenarios, there are significant distributional differences between simulated data and data collected in real-world scenarios. Therefore, directly applying models trained on simulated data to real-world scenarios may lead to a substantial performance degradation.

[0004] Unsupervised Domain Adaptation (UDA) is a technique in machine learning that aims to address the problem of distribution differences between the source and target domains by transferring a model trained on a labeled source domain to an unlabeled target domain. It can solve the problem of point cloud distribution differences between the source and target domains in autonomous driving, thus overcoming the difficulties caused by the lack of labeled target domain data. Existing point cloud object detection UDA methods are mainly divided into the following two categories:

[0005] 1) Self-Training-Based Approach: This approach achieves domain adaptation through a pre-trained model and pseudo-label generation. First, the method trains a pre-trained model using labeled point clouds from the source domain. Then, pseudo-labels are generated in the target domain using this pre-trained model; that is, unlabeled data from the target domain is predicted using the pre-trained model, and the prediction results are used as pseudo-labels. Finally, the labeled data from the source domain and the pseudo-labeled data from the target domain are merged, and the model is retrained to achieve domain adaptation.

[0006] 2) Adversarial Training-Based Methods: Adversarial training-based methods introduce the training of a domain classifier. Adversarial training is a method that learns feature representations by maximizing the similarity between samples of the same class and minimizing the similarity between samples of different classes. In domain adaptation, adversarial training-based methods train an additional domain classifier to distinguish features between the target and source domains. By optimizing a binary classification loss function, this method makes the features of the target and source domains more similar in the feature space, thus achieving the effect of domain adaptation.

[0007] However, the drawbacks of these existing technologies are as follows: Existing point cloud object detection UDA methods such as MLCNet (Multi-level Conditional Network) and ST3D (Self-training for Weakly Supervised Domain Adaptation in 3D Object Detection) mainly focus on cross-domain methods between two real datasets, such as from the Nusense dataset (a large, publicly available dataset for autonomous driving) to the Kitti dataset (currently the largest international dataset for evaluating computer vision algorithms in autonomous driving scenarios), and from the Lyft dataset (a large, publicly available dataset for autonomous driving) to the Kitti dataset, etc., and have achieved usable results; however, these methods have not yet been experimented with domain adaptation tasks from simulation data to real data, lack targeted integration of the characteristics of simulation data, and exhibit performance degradation issues in cross-domain results from simulation data to real data.

[0008] Therefore, it is necessary to improve the aforementioned existing technologies. Summary of the Invention

[0009] The main technical problem solved by this invention is to provide a point cloud target detection domain adaptation method from simulation data to real data. This method can adapt to cross-domain tasks from simulation data to real data and can solve the problem of detector performance degradation caused by excessive differences between point clouds in the simulation domain and the real domain.

[0010] According to a first aspect, one embodiment provides a point cloud target detection domain adaptation method from simulated data to real data. The method includes:

[0011] Obtain the point cloud from the target domain;

[0012] The point cloud is input into a trained two-stage point cloud target detection model, and the two-stage point cloud target detection model outputs the final prediction result for each target object in the point cloud. The final prediction result includes the category result and localization result of the target object.

[0013] The two-stage point cloud target detection model includes a one-stage detection model and a two-stage detection model. The one-stage detection model is used to process the point cloud to output a feature map and a one-stage candidate box. The two-stage detection model is used to process the feature map and the one-stage candidate box to obtain the category of each target object in the point cloud and the corresponding two-stage detection box.

[0014] Wherein, the size parameter in the candidate box parameters of the first-stage candidate box is a set of fixed values ​​corresponding to the source domain dataset, and the second-stage detection box is obtained based on the first-stage candidate box;

[0015] The first-stage detection model is trained on a source domain dataset from the source domain, and the second-stage detection model is trained on a target domain dataset from the target domain. The source domain and the target domain are different domains.

[0016] In one embodiment, the source domain is a simulation domain, and the source domain dataset is a simulation dataset; the set of fixed values ​​corresponding to the source domain dataset is determined based on the data distribution on the source domain dataset.

[0017] In one embodiment, the two-stage detection model includes a teacher sub-model; the teacher sub-model is obtained through iterative training of a preset number of training rounds.

[0018] In one embodiment, the training process for one training round of the teacher sub-model includes:

[0019] After the training iteration begins, unlabeled target domain data is input into the teacher sub-model. The teacher sub-model predicts the target domain data and outputs the category prediction result, regression prediction result, and uncertainty prediction result of the two-stage detection box corresponding to the target domain data. The target domain data comes from the target domain dataset.

[0020] Based on the magnitude of the uncertainty prediction result, determine whether the unlabeled target domain data is a low uncertainty sample;

[0021] If the target domain data is a low-uncertainty sample, then it is further determined whether the low-uncertainty sample is a high-confidence sample; if the target domain data is not a low-uncertainty sample, then the current iteration of training ends.

[0022] If the low uncertainty sample is a high confidence sample, the category prediction result, regression prediction result, and uncertainty prediction result corresponding to the high confidence sample are used as pseudo-labels for the high confidence sample. The student sub-model corresponding to the teacher sub-model predicts the high confidence sample and outputs the category prediction result, regression prediction result, and uncertainty prediction result of the two-stage detection box corresponding to the high confidence sample. If the low uncertainty sample is not a high confidence sample, the current iteration of training ends.

[0023] The classification loss function, regression loss function, and uncertainty loss function of the student sub-model are calculated based on the category prediction result, regression prediction result, and uncertainty prediction result of the two-stage detection box corresponding to the high-confidence sample, as well as the pseudo-label of the high-confidence sample.

[0024] The student sub-model is updated using the student sub-model loss function via backpropagation;

[0025] The teacher sub-model is updated using an exponential moving average expression. After updating the teacher sub-model, the current iteration of training ends.

[0026] In one embodiment, the student sub-model loss function includes the classification loss function, regression loss function, and uncertainty loss function of the student sub-model;

[0027] Wherein, the uncertainty loss function of the student sub-model is equal to the mean of the overall uncertainty of the multiple two-stage detection boxes corresponding to the high-confidence samples, and the overall uncertainty of the two-stage detection boxes is equal to the mean of the uncertainty of all detection box parameters of the two-stage detection boxes.

[0028] In one embodiment, the detection box parameters follow a Gaussian distribution N(μ,σ). 2 μ is the mean of the detection box parameters, σ is the standard deviation of the detection box parameters, and the standard deviation of the detection box parameters is used as the uncertainty of the detection box parameters.

[0029] In one embodiment, the uncertainty loss function of the student sub-model includes a loss function for the uncertainty of each of the two-stage detection boxes' parameters.

[0030] The expression for the loss function of the uncertainty of the detection box parameters is:

[0031]

[0032] Where x is the input sample (i.e., any sample); θ is the network weight of the student sub-model; and fμ f is the predicted value of the student sub-model for the mean μ of the Gaussian distribution corresponding to the detection box parameters. σ Let be the predicted value of the student sub-model for the standard deviation σ of the Gaussian distribution corresponding to the detection box parameters, and y be the label of the sample. In one embodiment, the expression for the exponential moving average is:

[0033]

[0034] Among them, the The network weights of the teacher sub-model at time t are represented by the following: The student sub-model represents the network weights at time t. The value represents the network weight of the teacher sub-model at time (t-1), and α is the update coefficient.

[0035] In one embodiment, all the detection box parameters of the two-stage detection box are the coordinates of the eight vertices of the two-stage detection box.

[0036] According to a second aspect, one embodiment provides a computer-readable storage medium. The computer-readable storage medium includes a program. The program is executable by a processor to implement the point cloud target detection domain adaptation method as described in any embodiment herein.

[0037] The beneficial effects of this application are:

[0038] The point cloud target detection domain adaptation method of this application first acquires a point cloud from the target domain; then, it inputs the point cloud into a trained two-stage point cloud target detection model. The two-stage point cloud target detection model outputs the final prediction result for each target object in the point cloud, including the target object's category and localization result. The two-stage point cloud target detection model includes a one-stage detection model and a two-stage detection model. The one-stage detection model processes the point cloud to output a feature map and a one-stage candidate box. The two-stage detection model processes the feature map and candidate boxes to obtain the category of each target object in the point cloud and its corresponding two-stage detection box. The size parameter in the candidate box parameters of the one-stage candidate box is a fixed set of values, and the two-stage detection box is obtained based on the one-stage candidate box. The point cloud target detection domain adaptation method of this application can adapt to cross-domain tasks from simulation data to real data and can solve the problem of detector performance degradation caused by excessive differences between the point cloud and the real domain. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a point cloud target detection domain adaptation method according to one embodiment.

[0040] Figure 2This is a flowchart illustrating the training process of one training round of a teacher sub-model in one embodiment.

[0041] Figure 3 This is a schematic diagram of all the detection box parameters for a two-stage detection box in one embodiment. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0043] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0044] The serial numbers assigned to components in this document, such as "first" and "second," are used only to distinguish the described objects and have no sequential or technical meaning. The terms "connection" and "linkage" used in this application, unless otherwise specified, include both direct and indirect connections (linkages).

[0045] The technical solution of this application will be described in detail below with reference to the embodiments.

[0046] Please refer to Figure 1 This application provides a point cloud target detection domain adaptation method, including:

[0047] Step S100: Obtain the point cloud from the target domain;

[0048] Step S200: Input the point cloud into the trained two-stage point cloud object detection model. The two-stage point cloud object detection model outputs the final prediction result for each target object in the point cloud. The final prediction result includes the category result and localization result of the target object.

[0049] The two-stage point cloud target detection model includes a one-stage detection model and a two-stage detection model. The one-stage detection model is used to process the point cloud and output a feature map and a one-stage candidate box. The two-stage detection model is used to process the feature map and the candidate box to obtain the category of each target object in the point cloud and the corresponding two-stage detection box.

[0050] Among them, the size parameter in the candidate box parameters of the first-stage candidate box is a fixed set of values, and the second-stage detection box is obtained based on the first-stage candidate box.

[0051] The first-stage detection model is trained on the source domain dataset from the source domain, and the second-stage detection model is trained on the target domain dataset from the target domain. The source domain and the target domain are different domains.

[0052] It should be noted that, firstly, the one-stage and two-stage detection models are integrated and trained in a supervised manner on the source domain dataset; then, the two-stage detection model is trained on the target domain. That is, the two-stage detection model was also trained on the source domain dataset.

[0053] In some embodiments, in step S100 above, the point cloud from the target domain is a point cloud frame-level data sample, and each point cloud frame-level data sample may include one or more target objects.

[0054] In some embodiments, the two-stage point cloud object detection model in step S200 above can use an existing point cloud object detection model as its basic framework. Then, the basic framework is improved by applying the various improvement points discussed below to obtain the two-stage point cloud object detection model of this application. For example, the existing point cloud object detection model mentioned above can be the PointRCNN point cloud object detector. PointRCNN is a two-stage object detection algorithm. The first stage generates candidate boxes and corresponding feature maps based on the input data; the second stage corrects the generated candidate boxes, etc. The existing point cloud object detection model mentioned above can also be other two-stage point cloud detectors, and this application does not impose specific limitations on them. The PointRCNN point cloud object detector includes two key modules: the RPN module and the RCNN module. The RPN module is responsible for generating candidate boxes and regressing their accurate locations. It consists of three sub-networks: a feature encoder, a candidate box regression network, and a candidate box classification network. The RCNN module is responsible for further classifying and regressing the candidate boxes generated by the RPN module.

[0055] It should be noted that the specific details of the target object category and localization results mentioned above are common knowledge in this field and will not be elaborated here. The specific process by which the one-stage detection model processes the point cloud to output feature maps and one-stage candidate boxes is existing technology in this field and will not be elaborated here.

[0056] In step S200 above, the source domain is the simulation domain, and the source domain dataset is the simulation dataset; a set of fixed values ​​is determined based on the data distribution on the source domain dataset.

[0057] In some embodiments, this application uses the simulation domain as the source domain described above.

[0058] In some embodiments, the target domain can be the autonomous driving domain. The target domain dataset can be a real-world dataset, such as one from the autonomous driving domain. The target domain can also be other domains.

[0059] In some embodiments, the source domain dataset can be a simulation dataset from the field of autonomous driving.

[0060] In some embodiments, the source domain dataset can also be a simulation dataset from other domains. No limitation is made here regarding the domain of the simulation dataset.

[0061] In existing common two-stage point cloud object detection models, the two-stage network (i.e., the two-stage detection model) used to predict the final 3D bounding box (as described in the two-stage detection model above) relies not only on the features of the one-stage network (i.e., the feature map output by the one-stage detection model above) as input, but also on the candidate boxes of the one-stage network (i.e., the candidate boxes of the one-stage network above) as output. The two-stage network only needs to predict the deviation between the two-stage detection boxes and the label values. Typically, this method can enhance the robustness of the network. However, in cross-domain tasks, due to the difference in target size distribution between the source and target domains, the one-stage detection model trained under supervision on the source domain dataset may have significant errors in the candidate boxes of the one-stage network in cross-domain tasks, thus affecting the accuracy of the two-stage localization.

[0062] Therefore, to reduce reliance on the output of the first-stage network and simplify the second-stage prediction task, this application employs a simplified point cloud detector mechanism using anchor boxes in the second stage. Specifically, it uses fixed-size anchor boxes (i.e., the aforementioned second-stage detection boxes) to calculate the regression loss. This simple substitution allows the second-stage network to predict its difference from the final labeled box (as described above) based solely on this fixed set of geometric parameters. While this reduces the overall robustness of the two-stage point cloud object detection model to some extent, it improves the consistency of the two-stage point cloud object detection model across the source and target domains, and reduces the complexity of the second-stage prediction task.

[0063] Specifically, this application will include all candidate box parameters for the first-stage candidate boxes (please refer to...). Figure 3 On the left, the dimensions of the candidate box include the three-dimensional coordinates (x, y, z) of the center point, the length (l), width (w), height (h), and deflection angle (α) of the candidate box (see reference). Figure 3 On the left, the size parameters here specifically refer to the candidate box parameters related to the size or dimensions of the candidate box, such as the length (l), width (w), and height (h) of the candidate box in the first stage mentioned above, which are replaced with a set of fixed values ​​that have been set in advance.

[0064] In some embodiments, the size parameter values ​​in the candidate box parameters of the aforementioned first-stage candidate box can be set to the statistical results on the source domain. This is because the actual labels in the source domain are visible; that is, the length, width, and height of the candidate boxes in all labels can be counted first, and then their average value can be taken as the aforementioned set of fixed values. It is understood that the setting of the aforementioned set of fixed values ​​is acceptable as long as it is reasonable. For example, the length (l), width (w), and height (h) of the aforementioned first-stage candidate box can be set to 3.9, 1.6, and 1.56, respectively.

[0065] It's important to note that existing two-stage networks typically rely on the output of the first-stage network. For example, based on the ROI (Region of Interest) box output by the first-stage network, a final predicted box is generated by calculating the difference between the predicted box and the ROI box. The ROI parameters are then added to the prediction result to obtain the final predicted box. However, this method only performs well when training and testing are on the same domain. If the labeled boxes from the source domain are used to train the aforementioned one-stage detection model, the model will be biased towards predicting the size of the output ROI box as the size in the source domain, leading to a significant deviation from the actual target domain. If the ROI boxes output by the first-stage detection model are then used as the basis for predicting the difference between the final predicted box and the ROI box, the final predicted box will also be affected by the estimation bias of the ROI box from the first-stage detection model, thus impacting the accuracy of the final predicted box. In this application, the size of the ROI box (i.e., the candidate box parameters of the first-stage candidate box) of the aforementioned one-stage detection model is replaced with a set of fixed values ​​during the source domain training. The aforementioned two-stage detection model only needs to learn to predict the difference between the set of fixed values ​​and the final size (i.e., the detection box parameters of the second-stage detection box). Similarly, when testing on the target domain, the aforementioned two-stage detection model only needs to consider predicting the difference based on the set of fixed values. This unifies the consistency of the two-stage point cloud object detection model in the source and target domains, thereby transferring the knowledge learned from the source domain to the target domain and reducing the task complexity of the two-stage point cloud object detection model in cross-domain tasks.

[0066] It should be noted that while the aforementioned set of fixed values ​​can be adjusted by the user, these values ​​still need to be within a reasonable range to prevent them from deviating too much from reality. The reason for stating that "a set of fixed values ​​is determined based on the data distribution in the source domain dataset" is as follows: A reasonable range can usually be obtained by averaging these attributes across all labels in a dataset. This can be done using either the target domain dataset or the source domain dataset. However, using label data from the target domain dataset (which is typically unavailable in cross-domain tasks) would be considered an information leak, meaning it cannot be considered a fully unsupervised cross-domain method. Therefore, the setting of the aforementioned set of fixed values ​​usually depends on the data distribution in the source domain, not the data distribution in the target domain.

[0067] It should be noted that the aforementioned one-stage detection model is a source-domain specific model obtained through fully supervised training on source-domain labeled data. The aforementioned source-domain labeled data refers to the simulated data and its corresponding ground truth labels. These simulated data and corresponding ground truth labels can generally be obtained through a simulator, and the corresponding ground truth labels can also be obtained through precise manual annotation (i.e., real data). Since the training process of the aforementioned one-stage detection model is existing technology and common knowledge in this field, the details of how to obtain the trained one-stage detection model will not be elaborated here.

[0068] In step S200 above, the two-stage detection model includes a teacher sub-model; the teacher sub-model is obtained through iterative training of a preset number of training rounds.

[0069] It should be noted that both the teacher sub-model and the student sub-model were pre-trained. The training method for the average teacher (i.e., the self-supervised learning domain adaptation framework of the average teacher-student model mentioned below) is roughly as follows: 1) The two-stage detection model (such as the teacher sub-model and the student sub-model) is first trained on the source domain (such as the source domain dataset mentioned above) for a certain number of rounds, and then the network parameters W are retained at this time; 2) After transferring to the target domain dataset for training, at the beginning of training, the teacher sub-model and the student sub-model read the same network parameters W as their initial parameters; 3) Since the inputs of the teacher sub-model and the student sub-model are not the same (this is because the same set of data has undergone different data augmentations), it is necessary to ensure that the output of the student sub-model is consistent with the output of the teacher sub-model during training. That is, it is necessary to construct a corresponding consistency loss function to update the network parameters of the student sub-model; 4) For the teacher sub-model, it does not directly backpropagate from the loss function to update the parameters, but obtains the update from the parameters of the student sub-model according to the exponential moving average method mentioned below. The average teacher training method (not proposed in this application) is existing technology in this field, and this application applies it to the cross-domain task of point cloud object detection.

[0070] In some embodiments, those skilled in the art can flexibly set the above-mentioned preset training rounds according to actual needs, and there is no limitation on the above-mentioned preset training rounds here. In each iteration training round, the student sub-model is updated once, and then the teacher sub-model is updated once.

[0071] In some embodiments, please refer to Figure 2 The training process for one training round of the teacher sub-model includes:

[0072] Step S10: After the training of the current iteration begins, the unlabeled target domain data is input into the teacher sub-model. The teacher sub-model predicts the target domain data and outputs the category prediction result, regression prediction result, and uncertainty prediction result of the two-stage detection box corresponding to the target domain data; wherein, the target domain data comes from the target domain dataset.

[0073] Step S20: Based on the magnitude of the uncertainty prediction result, determine whether the unlabeled target domain data is a low uncertainty sample;

[0074] If the target domain data is a low-uncertainty sample, then it continues to determine whether the low-uncertainty sample is a high-confidence sample; if the target domain data is not a low-uncertainty sample, then the current iteration of training ends.

[0075] If a low-uncertainty sample is a high-confidence sample, the category prediction result, regression prediction result, and uncertainty prediction result corresponding to the high-confidence sample are used as pseudo-labels for the high-confidence sample. The student sub-model corresponding to the teacher sub-model predicts the high-confidence sample and outputs the category prediction result, regression prediction result, and uncertainty prediction result of the two-stage detection box corresponding to the high-confidence sample. If the low-uncertainty sample is not a high-confidence sample, the current iteration of training ends.

[0076] Step S30: Calculate the classification loss function, regression loss function, and uncertainty loss function of the Student sub-model based on the category prediction results, regression prediction results, and uncertainty prediction results of the two-stage detection boxes corresponding to the high-confidence samples, as well as the pseudo-labels of the high-confidence samples.

[0077] Step S40: Update the student sub-model using the student sub-model loss function through backpropagation;

[0078] Step S50: Update the teacher sub-model using the exponential moving average expression. After updating the teacher sub-model, end the current iteration of training.

[0079] In step S10 above, the input unlabeled target domain data is a point cloud frame-level sample, and each point cloud frame may include one or more target objects.

[0080] In some embodiments, in step S20 above, the low uncertainty samples can be target domain data whose uncertainty prediction results are below a preset threshold. The preset threshold can be flexibly set according to actual needs. In some embodiments, the process of determining the low uncertainty samples is relative to the entire target domain dataset. In this application, the uncertainty of all samples in the target domain dataset is obtained by performing network inference on the entire target domain dataset once before the start of iterative training, and then samples within a preset proportion range are used as low uncertainty samples. For example, all samples in the target domain dataset can be sorted from low to high uncertainty, and then samples within a preset proportion threshold (such as the top 20%) are selected as low uncertainty samples.

[0081] Both the aforementioned teacher sub-model and the aforementioned student sub-model are complete two-stage detector models, and their prediction results include the category of the detection box, the location of the detection box, and the uncertainty of the detection box (derived from the "additional network structure" mentioned in this application).

[0082] In some embodiments, in step S20 above, the confidence level of the high-confidence samples refers to a classification confidence value given by the classification network of the detector (such as the teacher sub-model mentioned above). This value is used in general self-learning methods and the Mean Teachers framework mentioned above. This application still needs to use the above-mentioned confidence level to improve the accuracy of the selected pseudo-labels. It should be noted that in step S30 above, the pseudo-labels of the high-confidence samples are obtained by reasoning from the teacher network (i.e., the teacher sub-model mentioned above) and used for supervised training of the student network (i.e., the student sub-model mentioned above). The pseudo-labels of the high-confidence samples and the supervised training of the student network both belong to the above-mentioned prior art (i.e., the Mean Teachers mentioned above), so they will not be described in detail here.

[0083] It should be noted that the specific content of updating the student sub-model using the student sub-model loss function through backpropagation in step S40 above is also existing technology in this field, so it will not be elaborated here.

[0084] In some embodiments, the anchor box simplified point cloud detector mechanism can be used after the steps of "the teacher sub-model predicts the target domain data and outputs the category prediction result, regression prediction result, and uncertainty prediction result of the two-stage detection box corresponding to the target domain data" and "the student sub-model predicts the high-confidence samples and outputs the category prediction result, regression prediction result, and uncertainty prediction result of the two-stage detection box corresponding to the high-confidence samples." In other words, the anchor box simplified point cloud detector mechanism can be considered as a post-processing step for the regression prediction results.

[0085] The training method of the two-stage detection model in this application uses self-supervised learning, specifically a self-supervised learning domain adaptation framework based on average teacher-student interaction. During training, the two-stage detection model uses unlabeled target domain data and employs two independent two-stage neural networks: a "student network" (i.e., the student sub-model) and a "teacher network" (i.e., the teacher sub-model). The "student network" and "teacher network" have the same structure but different parameter values, both initially derived from networks trained in the source domain. During training, the teacher network infers from the unlabeled target domain data, using the inferred predictions as pseudo-labels. The "student network" is then trained using both labeled and pseudo-labeled data. The goal of the "student network" is to minimize the discrepancies between the predictions and the pseudo-labels and labeled data, ensuring that the predictions of the student sub-model are as consistent as possible with those of the teacher sub-model. In step S50, the teacher sub-model is obtained using the Exponential Moving Average (EMA) method, which can be considered the average value of the student sub-model over time.

[0086] It should be noted that the student and teacher sub-models involved in this application belong to the prior art in this field, but they differ from the teacher and student models in existing conventional transfer learning: the teacher and student sub-models in this application are based on the same network structure and number of parameters, but use different parameters and training methods. The aforementioned training method is called Mean Teachers (an unsupervised self-training method). The training method of the teacher sub-model in this application makes some improvements to the Mean Teachers method in the self-training stage (see the training method of the teacher sub-model in this application for details).

[0087] In step S50 above, the expression for the exponential moving average is:

[0088]

[0089] in, This represents the network weights of the teacher sub-model at time t. This represents the network weights of the student sub-model at time t. Let α represent the network weight of the teacher sub-model at time (t-1), and α be the update coefficient.

[0090] In some embodiments, the update coefficient α ranges between 0 and 1, but α cannot be equal to 0 or 1. The update coefficient α is typically a large value such as 0.999 or 0.99.

[0091] After the aforementioned teacher sub-model completes the training of the preset training rounds, the aforementioned teacher sub-model serves as the second-stage detection model in the aforementioned trained two-stage point cloud object detection model.

[0092] In step S40 above, the student sub-model loss function includes the classification loss function, regression loss function, and uncertainty loss function of the student sub-model.

[0093] The uncertainty loss function of the student sub-model is equal to the mean of the overall uncertainty of the multiple two-stage detection boxes corresponding to the high-confidence samples.

[0094] The overall uncertainty of the two-stage detection frame is equal to the mean of the uncertainties of all detection frame parameters of the two-stage detection frame.

[0095] This application also employs a localization noise index based on uncertainty and a noise sample suppression mechanism, which are discussed in detail below: A typical point cloud detector (such as the two-stage detection model mentioned above) needs to simultaneously complete both classification and localization tasks. For a single regressor y in the localization task, assuming y follows a Gaussian distribution N(μ,σ) with parameters μ and σ. 2The "individual regression quantity y" mentioned above can refer to any one of the following regression quantities: the candidate box center point position, the candidate box length, width, height, and deflection angle. Alternatively, the "individual regression quantity y" can be any one of the eight detection box parameters of the two-stage detection box, where the eight detection box parameters are the coordinates of the eight vertices of the two-stage detection box. This application can directly regress the parameter σ through an additional network structure, thus serving as an indicator of the uncertainty of the current regression result (i.e., the aforementioned "individual regression quantity y"). Specifically, the "additional network structure" refers to another multi-layer fully connected network connected after the two-stage network (such as the aforementioned two-stage detection model). This "additional network structure" is independent of the existing classification and localization networks in a typical detector. By performing maximum likelihood estimation (MLE) on the network weights of the model (such as the aforementioned student sub-model), a negative log-likelihood loss function can be used to optimize the model's estimation of σ. Generally, high uncertainty in the input target domain data (such as the target domain data input in step S10 above) means that the model estimation error may be large. Since the self-training process requires using the prediction results of the teacher sub-model as pseudo-labels, the larger the estimation error of these pseudo-labels, the greater the noise introduced into the entire self-learning process of the teacher sub-model. Therefore, the aforementioned high-noise samples refer to high noise introduced during the self-learning process, which may lead to poor self-learning performance. For the above reasons, the training process of the teacher sub-model in this application terminates the current iteration of training when it is determined that the input target domain data is not a low-uncertainty sample. Furthermore, after determining that the input target domain data is a low-uncertainty sample, it is necessary to further determine whether the low-uncertainty sample is a high-confidence sample. That is, only the determined high-confidence samples are used for the subsequent self-training process of the teacher sub-model, in order to eliminate the adverse effects of high-noise samples (such as all samples in the input target domain data other than high-confidence samples) on the self-training process of the teacher sub-model.

[0096] It's important to note that conventional neural networks typically require a loss function to calculate the deviation between the estimated and true values, then backpropagation is performed to optimize the model's parameters, thereby improving the accuracy of the model's estimates. The negative log-likelihood loss function described above serves as such a loss function to optimize the accuracy of the model's estimation of uncertainty. The "network weights of the model" mentioned above refer to the weight parameters of a generalized regression model; that is, the negative log-likelihood loss function can be obtained by performing maximum likelihood estimation on the weight parameters of any regression model.

[0097] As mentioned above, the detection box parameters follow a Gaussian distribution N(μ,σ). 2 μ is the mean of the detection box parameters, and σ is the standard deviation of the detection box parameters. The standard deviation of the detection box parameters is used as the uncertainty of the detection box parameters.

[0098] In step S30 above, the uncertainty loss function of the student sub-model includes the loss function of the uncertainty of each detection box parameter of all two-stage detection boxes, and the loss function of the uncertainty of the detection box parameters (i.e., the negative log-likelihood loss function mentioned above). nll The expression for ) is:

[0099]

[0100] Where x is the input sample; θ is the network weight of the student sub-model; f μ f is the predicted value of the student sub-model for the mean μ of the Gaussian distribution corresponding to the parameters of the detection box. σ Let σ be the predicted standard deviation σ of the Gaussian distribution corresponding to the parameters of the detection box for the student sub-model, and y be the label of the above sample.

[0101] It should be noted that the same method was used to obtain the loss function for the uncertainty of the detection box parameters when training in the source domain without using the average teacher method.

[0102] It should be noted that the uncertainty prediction results corresponding to the detection frame parameters are expressed as the square of the predicted standard deviation (i.e., variance), that is, using the above f... σ It is expressed as the square of .

[0103] It should be noted that the sample labels here are sample labels in a broad sense. Specifically, for source domain data, the sample labels here refer to truth labels (usually manually labeled); for target domain data (which does not contain truth labels), the sample labels here refer to the aforementioned pseudo-labels.

[0104] It should be noted that the above negative log-likelihood loss function l nll As a loss function, backpropagation is used during training to update the parameters of the entire network (such as the student sub-model), thereby optimizing the network's estimation of the uncertainty of the detection box parameters.

[0105] The overall loss function of the two-stage point cloud object detection model described in this application includes five loss functions: a one-stage classification loss function, a one-stage localization loss function, a two-stage classification loss function, a two-stage localization loss function, and the aforementioned uncertainty loss function. In some embodiments, the five loss functions can be weighted according to a preset set of weights, and then the weighted five loss functions can be added together to obtain the final overall loss function of the two-stage point cloud object detection model. That is, the two-stage point cloud object detection model is obtained by training using this final overall loss function.

[0106] This application further considers expressing the uncertainty of multiple regression quantities as the uncertainty of the entire detection box. Conventional detectors directly regress each regression quantity for each detection box (such as the candidate box parameters of a one-stage candidate box or the detection box parameters of a two-stage detection box as described above). Please refer to [reference needed]. Figure 3 On the left, however, each regressor in this representation has a different physical meaning and distribution range, and the process of reducing it to an independent constant representation usually requires a carefully designed set of weights. Please refer to... Figure 3 On the right side, starting from the corner point of the detection box, the overall uncertainty of the two-stage detection box is represented by 8 corner points (i.e., the eight detection box parameters of the two-stage detection box are the eight vertices of the two-stage detection box (such as...) Figure 3 The average uncertainty of the coordinates (shown by numbers 1 to 8 on the right) can avoid adjusting and designing this set of weights. After expressing the uncertainty of the two-stage detection box as the uncertainty of the eight vertices of the two-stage detection box, it can be obtained through the negative log-likelihood loss function l. nll The estimation of the uncertainty of the coordinates of each vertex of the two-stage detection box is optimized. That is, during the prediction process, the mean of the uncertainties of the eight vertices is directly used to represent the uncertainty of the entire two-stage detection box.

[0107] In some embodiments, all the detection box parameters of the two-stage detection box are the coordinates of the eight vertices of the two-stage detection box. Of course, in other embodiments, all the detection box parameters of the two-stage detection box can still use the original seven detection box parameters (such as the three-dimensional coordinates of the center point of the detection box, the length, width, height and deflection angle of the detection box).

[0108] It should be noted that the reason for expressing the overall uncertainty of the two-stage detection box as the average of the uncertainties of the coordinates of the eight vertices of the two-stage detection box in this application is as follows: In conventional regression methods, the parameters of a predicted 3D detection box are represented as seven regression quantities (such as the three-dimensional coordinates of the center point of the first-stage candidate box, the length, width, height, and deflection angle of the first-stage candidate box). The above representation of regression quantities results in each detection box having seven uncertainty values, corresponding to the attributes of these seven boxes. However, in this representation, the units and meanings of parameters such as the three-dimensional coordinates of the center point of the first-stage candidate box, the length, width, height, and deflection angle of the first-stage candidate box are different (for example, the unit of length is meters, while the unit of deflection angle is radians). This leads to significant differences in the numerical distribution of these regression quantities, and consequently, significant differences in the uncertainties corresponding to these regression quantities. Therefore, for ease of comparison, this application represents the uncertainty of a detection box (such as the two-stage detection box mentioned above) with a single scalar, instead of having a corresponding uncertainty for each attribute of the detection box. For example, the uncertainties of the seven regression quantities correspond to the three-dimensional coordinates of the center point of the detection box, the length, width, height, and deflection angle of the first-stage candidate box. This requires performing a method similar to averaging on the above seven regression quantities. However, due to the reasons mentioned above (i.e., the range and unit of each value of the above seven regression quantities may be different), this will lead to a structurally unavoidable large difference in these seven uncertainties. For example, in the experiment, the uncertainty of the regression quantity corresponding to the length is generally around 0.01, which is nearly a hundred times higher than the uncertainty of the corresponding deflection angle. Directly averaging the uncertainties of the seven regression quantities cannot well reflect the uncertainty of the deflection angle (because the uncertainty of the deflection angle is masked by the uncertainty of the length during the averaging process). Therefore, a set of weights needs to be designed to balance this difference, but this set of weights cannot be obtained directly through mathematical derivation and can only be obtained through continuous experiments to obtain an approximate best setting.

[0109] Please refer to Figure 3 On the right side, when the eight vertices of the two-stage detection box are used to represent the two-stage detection box, that is, the eight detection box parameters of the two-stage detection box are the coordinates of the eight vertices of the two-stage detection box, the uncertainties of the eight detection box parameters of the two-stage detection box basically conform to the same range (i.e., based on the same unit), and thus the difference in the uncertainties of the eight detection box parameters of the two-stage detection box can be significantly reduced; in this way, the uncertainty of the entire two-stage detection box can be obtained by directly calculating the mean of the uncertainties of the eight detection box parameters, without the need to design a weight for each regression variable.

[0110] Since the input of a point cloud detector (such as the two-stage point cloud target detection model of this application) is usually a frame of point cloud, and its output is the category and location of all possible target objects in the point cloud of that frame, according to the above method, this application can obtain the uncertainty of each detection box (such as the two-stage detection box), and then obtain the uncertainty of all detection boxes in the current point cloud frame.

[0111] It can be seen that the uncertainty obtained by the above method can generally be considered as an attribute of the data (such as the target domain data) and is independent of the model (such as the two-stage detection model mentioned above), i.e., data uncertainty. Data uncertainty can capture noise present in sensor measurements. In the prediction process, it is generally accepted that data noise is closely related to prediction accuracy, i.e., data uncertainty can be used to measure the noise level of pseudo-labels. Based on data uncertainty as a noise measure of pseudo-labels, this application adopts two different learning strategies to suppress the influence of noisy samples on the self-training process of the above teacher sub-model:

[0112] 1) The first strategy is to select point cloud frames by predicting the uncertainty of the detection boxes of all possible target objects in a frame of point cloud data, and taking the average of the uncertainty of all detection boxes in a frame of point cloud data as the uncertainty of the current frame. In this way, the point cloud frames with lower noise (i.e. the high confidence samples mentioned above) can be selected from all target domain data input to the above teacher sub-model.

[0113] 2) The second strategy is at the pseudo-label level. For each pseudo-label in the same frame of point cloud data in the target domain, the teacher sub-model can predict the uncertainty of the high-confidence sample corresponding to that pseudo-label. This application uses an inverse weighting method to assign the above negative log-likelihood loss function l nll The portion generated by high-noise samples is given a smaller weight, thereby reducing the impact of high-noise samples.

[0114] It should be noted that the teacher sub-model here refers to the teacher model in the Mean Teachers architecture mentioned above, which is used to generate pseudo-labels and use these pseudo-labels to guide the student model in the self-learning process.

[0115] It should be noted that the "uncertainty of the current label" mentioned above refers to the average of the uncertainties of the eight corner points of the entire detection frame (i.e., the uncertainties of the eight detection frame parameters in the two-stage detection frame). The uncertainties of the eight detection frame parameters cannot be simply compared. It is necessary to express this set of values ​​(i.e., the uncertainties of the eight detection frame parameters) as a scalar that can reflect the average of the set of values, that is, the average of the set of values. Then, this average is used as the uncertainty of the pseudo label mentioned above, and is used for subsequent comparison with the uncertainties of other labels.

[0116] The above is a description of a point cloud target detection domain adaptation method. Some embodiments of this application also disclose a computer-readable storage medium. This computer-readable storage medium includes a program that can be executed by a processor to implement the point cloud target detection domain adaptation method as described in any of the embodiments herein.

[0117] This document describes various exemplary embodiments with reference to them. However, those skilled in the art will recognize that changes and modifications can be made to the exemplary embodiments without departing from the scope of this document. For example, various operational steps and components for performing operational steps can be implemented in different ways depending on the specific application or considering any number of cost functions associated with the operation of the system (e.g., one or more steps can be deleted, modified, or combined with other steps).

[0118] In the above embodiments, implementation can be achieved, in whole or in part, by software, hardware, firmware, or any combination thereof. Furthermore, as those skilled in the art will understand, the principles herein can be reflected in a computer program product on a computer-readable storage medium pre-loaded with computer-readable program code. Any tangible, non-transitory computer-readable storage medium may be used, including magnetic storage devices (hard disks, floppy disks, etc.), optical storage devices (CD-ROMs, DVDs, Blu-ray discs, etc.), flash memory, and / or the like. These computer program instructions can be loaded onto a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to form a machine, such that instructions executing on the computer or other programmable data processing apparatus can generate means for implementing a specified function. These computer program instructions can also be stored in a computer-readable storage medium that can instruct the computer or other programmable data processing apparatus to operate in a particular manner, such that instructions stored in the computer-readable storage medium can form an article of manufacture including means for implementing the specified function. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to perform a series of operational steps on the computer or other programmable apparatus to produce a computer-implemented process, such that instructions executing on the computer or other programmable apparatus can provide steps for implementing the specified function.

[0119] While the principles herein have been illustrated in various embodiments, numerous modifications to the structure, arrangement, proportions, elements, materials, and components, particularly suited to specific environmental and operational requirements, may be used without departing from the principles and scope of this disclosure. These modifications and other alterations or alterations will be included within the scope of this document.

[0120] The foregoing specific descriptions have been described with reference to various embodiments. However, those skilled in the art will recognize that various modifications and changes can be made without departing from the scope of this disclosure. Therefore, considerations for this disclosure are to be illustrative rather than restrictive, and all such modifications are to be included within its scope. Similarly, advantages, other advantages, and solutions to problems with respect to various embodiments have been described above. However, benefits, advantages, solutions to problems, and any elements that produce these, or make them more explicit, should not be construed as critical, essential, or necessary. The term “comprising” and any other variations thereof as used herein are non-exclusive inclusion, meaning that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed or not part of the process, method, system, article, or apparatus. Furthermore, the term “coupled” and any other variations thereof as used herein refer to physical connections, electrical connections, magnetic connections, optical connections, communication connections, functional connections, and / or any other connections.

[0121] Those skilled in the art will recognize that many changes can be made to the details of the above embodiments without departing from the basic principles of the invention. Therefore, the scope of the invention should be determined only by the claims.

Claims

1. A method for adapting the target detection domain from simulated data to real data in point cloud data, characterized in that, include: Obtain the point cloud from the target domain; The point cloud is input into a trained two-stage point cloud target detection model, and the two-stage point cloud target detection model outputs the final prediction result for each target object in the point cloud. The final prediction result includes the category result and localization result of the target object. The two-stage point cloud target detection model includes a one-stage detection model and a two-stage detection model. The one-stage detection model is used to process the point cloud to output a feature map and a one-stage candidate box. The two-stage detection model is used to process the feature map and the one-stage candidate box to obtain the category of each target object in the point cloud and the corresponding two-stage detection box. Wherein, the size parameter in the candidate box parameters of the first-stage candidate box is a set of fixed values ​​corresponding to the source domain dataset, and the second-stage detection box is obtained based on the first-stage candidate box; The first-stage detection model is trained on a source domain dataset from the source domain, and the second-stage detection model is trained first on the source domain dataset and then on a target domain dataset from the target domain. The source domain and the target domain are different domains. The two-stage detection model includes a teacher sub-model, which is obtained through iterative training of a preset number of training rounds. The training process for one training round of the teacher sub-model includes: After the training iteration begins, unlabeled target domain data is input into the teacher sub-model. The teacher sub-model predicts the target domain data and outputs the category prediction result, regression prediction result, and uncertainty prediction result of the two-stage detection box corresponding to the target domain data. The target domain data comes from the target domain dataset. Based on the magnitude of the uncertainty prediction result, determine whether the unlabeled target domain data is a low uncertainty sample; If the target domain data is a low-uncertainty sample, then it is further determined whether the low-uncertainty sample is a high-confidence sample; if the target domain data is not a low-uncertainty sample, then the current iteration of training ends. If the low uncertainty sample is a high confidence sample, the category prediction result, regression prediction result, and uncertainty prediction result corresponding to the high confidence sample are used as pseudo-labels for the high confidence sample. The high confidence sample is then predicted by the student sub-model corresponding to the teacher sub-model, and the category prediction result, regression prediction result, and uncertainty prediction result of the two-stage detection box corresponding to the high confidence sample are output. If the low uncertainty sample is not a high confidence sample, the current iteration of training ends.

2. The point cloud target detection domain adaptation method as described in claim 1, characterized in that, The source domain is a simulation domain, and the source domain dataset is a simulation dataset; the set of fixed values ​​corresponding to the source domain dataset is determined based on the data distribution on the source domain dataset.

3. The point cloud target detection domain adaptation method as described in claim 1, characterized in that, The training process for one training round of the teacher sub-model also includes: The classification loss function, regression loss function, and uncertainty loss function of the student sub-model are calculated based on the category prediction result, regression prediction result, and uncertainty prediction result of the two-stage detection box corresponding to the high-confidence sample, as well as the pseudo-label of the high-confidence sample. The student sub-model is updated using the student sub-model loss function via backpropagation; The teacher sub-model is updated using an exponential moving average expression. After updating the teacher sub-model, the current iteration of training ends.

4. The point cloud target detection domain adaptation method as described in claim 3, characterized in that, The student sub-model loss function includes the classification loss function, regression loss function, and uncertainty loss function of the student sub-model; Wherein, the uncertainty loss function of the student sub-model is equal to the mean of the overall uncertainty of the multiple two-stage detection boxes corresponding to the high-confidence samples, and the overall uncertainty of the two-stage detection boxes is equal to the mean of the uncertainty of all detection box parameters of the two-stage detection boxes.

5. The point cloud target detection domain adaptation method as described in claim 4, characterized in that, The detection box parameters follow a Gaussian distribution. The The mean value of the detection box parameters, the Let the standard deviation of the detection box parameters be used as the uncertainty of the detection box parameters.

6. The point cloud target detection domain adaptation method as described in claim 5, characterized in that, The expression for the loss function of the uncertainty of the detection box parameters is: ， Where x is the input sample; θ is the network weight of the student sub-model; The f μ The mean of the Gaussian distribution corresponding to the detection box parameters for the student sub-model. The predicted value, The f σ The standard deviation of the Gaussian distribution corresponding to the parameters of the detection box for the student sub-model. The predicted value is y, where y is the label of the sample.

7. The point cloud target detection domain adaptation method as described in claim 3, characterized in that, The expression for the exponential moving average is: ， Among them, the The network weights of the teacher sub-model at time t are represented by the following: The student sub-model represents the network weights at time t. The value represents the network weight of the teacher sub-model at time (t-1), and α is the update coefficient.

8. The point cloud target detection domain adaptation method as described in claim 4, characterized in that, All the parameters of the two-stage detection box are the coordinates of the eight vertices of the two-stage detection box.

9. A computer-readable storage medium, characterized in that, The method includes a program that can be executed by a processor to implement the point cloud target detection domain adaptation method as described in any one of claims 1 to 8.

Images