Model training method, target detection method, device, equipment, medium and product

Abstract

The embodiment of the invention provides a model training method and device, a target detection method and device, electronic equipment, a computer storage medium and a computer program product. The method comprises the steps that a first point cloud set and a second point cloud set are acquired; the first point cloud set comprises a plurality of marked point clouds, and the second point cloud set comprises a plurality of unmarked point clouds; determining a pseudo label of each unlabeled point cloud in the second point cloud set by using the first point cloud set; obtaining a third point cloud set based on the unmarked point clouds and the pseudo labels of the unmarked point clouds; performing mixing processing on the first point cloud set and the third point cloud set to obtain a mixed point cloud set; and training a first detection model by using the first point cloud set and the mixed point cloud set to obtain a target detection model.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a model training method, a target detection method, a device, an electronic device, a computer storage medium, and a computer program product. Background Art

[0002] In the field of autonomous driving, the amount of data that needs to be manually labeled is huge, so there is a problem of high labeling costs. In this case, making full use of unlabeled data has become an important strategy to improve the performance of autonomous driving detection models.

[0003] In the related art, a model is used to train both unlabeled data and labeled data. However, in practical applications, the pseudo-labels of unlabeled data are obtained by model detection, and these detections are not always completely accurate. In addition, the amount of unlabeled data is huge, so the model may rely more on these inaccurate pseudo-labels rather than the real labeled data during training, which in turn causes the model to be biased during training and overfit to the pseudo-labels instead of learning the distribution of the real data, which reduces the model detection performance. Summary of the invention

[0004] In view of the above problems, embodiments of the present application are proposed to provide a model training method, target detection method, device, electronic device, computer storage medium and computer program product that overcome the above problems or at least partially solve the above problems.

[0005] The technical solution of this application is implemented as follows:

[0006] The present application embodiment provides a model training method, the method comprising:

[0007] Acquire a first point cloud set and a second point cloud set; the first point cloud set includes a plurality of annotated point clouds, and the second point cloud set includes a plurality of unannotated point clouds;

[0008] Determine the pseudo labels of each unlabeled point cloud in the second point cloud set by using the first point cloud set; obtain a third point cloud set based on each unlabeled point cloud and the pseudo labels of each unlabeled point cloud;

[0009] Mixing the first point cloud set and the third point cloud set to obtain a mixed point cloud set;

[0010] The first point cloud set and the mixed point cloud set are used to train a first detection model to obtain a target detection model.

[0011] The present application provides a target detection method, the method comprising:

[0012] Acquire a detection point cloud set; the detection point cloud set includes at least one unlabeled point cloud to be detected;

[0013] The target detection model is used to predict the at least one unlabeled point cloud to be detected to obtain a prediction result; the prediction result includes a predicted label of the at least one unlabeled point cloud to be detected; wherein the target detection model is trained according to the model training method provided by one or more of the aforementioned technical solutions.

[0014] The present application also provides a model training device, which includes:

[0015] A first acquisition module, used to acquire a first point cloud set and a second point cloud set; the first point cloud set includes a plurality of annotated point clouds, and the second point cloud set includes a plurality of unannotated point clouds;

[0016] A determination module, configured to determine the pseudo labels of each unlabeled point cloud in the second point cloud set by using the first point cloud set; and obtain a third point cloud set based on each unlabeled point cloud and the pseudo labels of each unlabeled point cloud;

[0017] A mixing module, used for mixing the first point cloud set and the third point cloud set to obtain a mixed point cloud set;

[0018] A training module is used to train a first detection model using the first point cloud set and the mixed point cloud set to obtain a target detection model.

[0019] The present application also provides a target detection device, which includes:

[0020] A second acquisition module is used to acquire a detection point cloud set; the detection point cloud set includes at least one unlabeled point cloud to be detected;

[0021] A prediction module is used to use a target detection model to predict the at least one unlabeled point cloud to be detected to obtain a prediction result; the prediction result includes a predicted label of the at least one unlabeled point cloud to be detected; wherein the target detection model is trained according to the method provided by one or more of the aforementioned technical solutions.

[0022] An embodiment of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the model training method or target detection method provided by one or more of the aforementioned technical solutions is implemented.

[0023] An embodiment of the present application provides a computer storage medium, which stores a computer program; after the computer program is executed by a processor, it can implement the model training method or target detection method provided by one or more of the aforementioned technical solutions.

[0024] The present application provides a computer program product, including a computer program, which, when executed by a processor, implements the model training method or target detection method provided by one or more of the aforementioned technical solutions.

[0025] The embodiment of the present application provides a model training method, a target detection method, an apparatus, an electronic device, a computer storage medium and a computer program product. The model training method includes: obtaining a first point cloud set and a second point cloud set; the first point cloud set includes a plurality of annotated point clouds, and the second point cloud set includes a plurality of unannotated point clouds;

[0026] Determine the pseudo labels of each unlabeled point cloud in the second point cloud set by using the first point cloud set; obtain a third point cloud set based on each unlabeled point cloud and the pseudo labels of each unlabeled point cloud;

[0027] Mixing the first point cloud set and the third point cloud set to obtain a mixed point cloud set;

[0028] The first point cloud set and the mixed point cloud set are used to train a first detection model to obtain a target detection model.

[0029] It can be seen that in the embodiment of the present application, after determining the pseudo-labels of each unlabeled point cloud, the pseudo-labels of these unlabeled point clouds are not directly used for model training, but the first point cloud set and the third point cloud set with pseudo-labels are mixed, so that the point clouds in the mixed point cloud set after mixing include not only the information of the unlabeled point cloud, but also the information of the labeled point cloud. In this way, the inconsistency between the unlabeled point cloud and the labeled point cloud can be reduced; further, if the first point cloud set and the mixed point cloud set are used to jointly train the detection model, the excessive reliance on pseudo-labels in the model training process can be reduced, the model training effect can be ensured, and the model detection performance can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flowchart of a model training method in an embodiment of the present application;

[0031] Figure 2 It is a flowchart of another model training method in an embodiment of the present application;

[0032] Figure 3 It is a structural schematic diagram corresponding to a model training method in an embodiment of the present application;

[0033] Figure 4 A schematic diagram of a target detection method according to an embodiment of the present application;

[0034] Figure 5 It is a schematic diagram of the structure of the model training device in the embodiment of the present application;

[0035] Figure 6 is a schematic diagram of the structure of the target detection device in an embodiment of the present application;

[0036] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The present application will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the embodiments provided herein are only used to explain the present application and are not intended to limit the present application. In addition, the embodiments provided below are partial embodiments for implementing the present application, rather than providing all embodiments for implementing the present application. In the absence of conflict, the technical solutions recorded in the embodiments of the present application can be implemented in any combination.

[0038] It should be noted that, in the embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus including a series of elements includes not only the elements explicitly recorded, but also includes other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other related elements (such as steps in a method or units in a device, for example, a unit may be a portion of a circuit, a portion of a processor, a portion of a program or software, etc.) in the method or apparatus including the element.

[0039] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, I and / or J can represent the three situations that I exists alone, I and J exist at the same time, and J exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of I, J, and R can represent including any one or more elements selected from the set consisting of I, J, and R.

[0040] For example, the model training method provided in the embodiment of the present application includes a series of steps, but the model training method provided in the embodiment of the present application is not limited to the recorded steps. Similarly, the model training device provided in the embodiment of the present application includes a series of modules, but the model training device provided in the embodiment of the present application is not limited to including the modules explicitly recorded, and may also include modules required to obtain relevant data or perform processing based on relevant data.

[0041] In some embodiments of the present application, the model training method can be implemented using a processor in a model training device, and the above-mentioned processor can be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, and a microprocessor.

[0042] In an embodiment of the present application, the model training method can be applied to three-dimensional target detection scenarios in the field of unmanned autonomous driving. The method can be executed by a model training device, which can be configured in an electronic device.

[0043] Figure 1 is a flow chart of a model training method in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0044] Step 100: Obtain a first point cloud set and a second point cloud set.

[0045] In an embodiment of the present application, the first point cloud set includes multiple labeled point clouds, and the second point cloud set includes multiple unlabeled point clouds; wherein the labeled point clouds are point clouds labeled with labels, and the unlabeled point clouds are point clouds without labels; in actual scenarios, the number of unlabeled point clouds in the second point cloud set is much larger than the number of labeled point clouds in the first point cloud set.

[0046] Exemplarily, the label refers to the category to which the point cloud belongs. Here, the point cloud is a point cloud of a three-dimensional target, which is a set of points consisting of the spatial coordinates of each sampling point on the surface of the three-dimensional target. In an autonomous driving scenario, the three-dimensional target can be a pedestrian or vehicle on the road, etc.

[0047] Exemplarily, the point cloud can be obtained based on the principle of laser measurement or photogrammetry. Furthermore, it can also be obtained through acquisition equipment such as laser radar, stereo camera, etc., or it can also be obtained through other methods, which is not specifically limited in the embodiments of the present application.

[0048] Step 101: using the first point cloud set, determining the pseudo labels of each unlabeled point cloud in the second point cloud set, and obtaining a third point cloud set based on each unlabeled point cloud and the pseudo labels of each unlabeled point cloud.

[0049] Exemplarily, the pseudo-label of an unlabeled point cloud refers to the category of the unlabeled point cloud; wherein the category represents a category to which the unlabeled point cloud belongs.

[0050] In an embodiment of the present application, after obtaining a first point cloud set including multiple labeled point clouds, the first point cloud set can be used to determine pseudo labels for each unlabeled point cloud in the second point cloud set; the process is exemplarily described below.

[0051] In some embodiments, using the first point cloud set to determine the pseudo labels of each unlabeled point cloud in the second point cloud set can include: using the first point cloud set to train a second detection model to obtain a trained second detection model; using the trained second detection model to detect each unlabeled point cloud in the second point cloud set to obtain a pseudo label for each unlabeled point cloud.

[0052] Exemplarily, in an autonomous driving scenario, the second detection model can be a cloud-based model for target detection; exemplary, the cloud-based model is a large backbone model with more parameters and a larger network capacity. Therefore, using this model to detect each unlabeled point cloud in the second point cloud set can ensure the detection effect.

[0053] In an embodiment of the present application, the first point cloud set can be used to train the second detection model first; exemplarily, before model training, the first point cloud set can be divided into a training set and a validation set, and appropriate training parameters can be set, such as learning rate, batch size, and number of training iterations, etc., by iteratively training the second detection model to obtain a trained second detection model.

[0054] Furthermore, after obtaining the trained second detection model, the trained second detection model can be used to detect multiple unlabeled point clouds in the second point cloud set to obtain detection results. Here, the detection results include pseudo labels of the multiple unlabeled point clouds.

[0055] In an embodiment of the present application, after obtaining the pseudo-labels of each unlabeled point cloud in the second point cloud set, a third point cloud set can be obtained based on each unlabeled point cloud and the pseudo-labels of each unlabeled point cloud; that is, the third point cloud set includes multiple unlabeled point clouds with pseudo-labels.

[0056] Step 102: Mix the first point cloud set and the third point cloud set to obtain a mixed point cloud set.

[0057] In an embodiment of the present application, after obtaining the first point cloud set and the third point cloud set, the first point cloud set and the third point cloud set can be mixed to obtain a mixed point cloud set; here, the mixed point cloud set includes multiple mixed point clouds.

[0058] In some embodiments, the first point cloud set and the third point cloud set are mixed to obtain a mixed point cloud set, including: filtering the third point cloud set to obtain a fourth point cloud set; randomly sampling multiple points included in each labeled point cloud in the first point cloud set to obtain multiple first sampling point sets corresponding to the first point cloud set; randomly sampling multiple points included in each pseudo-labeled point cloud in the fourth point cloud set to obtain multiple second sampling point sets corresponding to the fourth point cloud set; mixing every two sampling point sets in the multiple first sampling point sets and the multiple second sampling point sets to obtain multiple mixed point clouds; and determining a mixed point cloud set based on the multiple mixed point clouds.

[0059] In an embodiment of the present application, in order to ensure the reliability of the pseudo-labels corresponding to each point cloud in the third point cloud set, after obtaining the third point cloud set, it can be filtered to obtain a fourth point cloud set; and then mixed processing can be performed based on the fourth point cloud set.

[0060] In some embodiments, filtering the third point cloud set to obtain a fourth point cloud set may include: obtaining pseudo-label confidences of multiple pseudo-labeled point clouds in the third point cloud set; filtering the pseudo-labeled point clouds in the third point cloud set whose pseudo-label confidences are less than or equal to a confidence threshold to obtain a fourth point cloud set.

[0061] It should be noted that the detection results obtained using the trained second detection model include, in addition to the pseudo-labels of multiple unlabeled point clouds in the second point cloud set, also the pseudo-label confidences of multiple unlabeled point clouds; here, the pseudo-label confidence is used to characterize the category probability that the unlabeled point cloud belongs to a certain category.

[0062] For example, for ease of understanding, the pseudo-label of each unlabeled point cloud can be expressed as The pseudo-label confidence is expressed as i∈(1,N), where N is the number of unlabeled point clouds, that is, the number of second detection boxes detected by the model.

[0063] In an embodiment of the present application, an unlabeled point cloud with pseudo-labels may be referred to as a point cloud with pseudo-labels; after obtaining the pseudo-label confidences of multiple point clouds with pseudo-labels in the third point cloud set, the pseudo-label confidence of each unlabeled point cloud may be compared with a confidence threshold, and the unlabeled point clouds that are less than or equal to the confidence threshold may be filtered to retain each unlabeled point cloud whose pseudo-label confidence is greater than the confidence threshold; wherein, the implementation of the filtering process may refer to formula (1).

[0064]

[0065] Among them, Score is the confidence threshold of a certain unlabeled point cloud; it should be noted that filtering a certain unlabeled point cloud refers to filtering the unlabeled point cloud, the pseudo-labels of the unlabeled point cloud, and the second detection box together.

[0066] Here, the value of the confidence threshold can be set according to actual conditions, and the embodiments of the present application do not limit this; for example, the value of the confidence threshold can be 0.7 or 0.8, etc.

[0067] Exemplarily, it is assumed that the confidence threshold is 0.8, and the pseudo-label confidence of a certain unlabeled point cloud is 0.5. Since the pseudo-label confidence is less than the confidence threshold, the unlabeled point cloud needs to be filtered.

[0068] It can be understood that by filtering the unlabeled point clouds with low pseudo-label confidence in the second point cloud set, the quality of the pseudo-labels of the remaining unlabeled point clouds can be guaranteed, thereby ensuring the training effect of the subsequent model.

[0069] Exemplarily, according to the above content, the pseudo-label quality of the filtered fourth point cloud set is better than that of the unfiltered third point cloud set. Therefore, in order to ensure the model training effect, after obtaining the fourth point cloud set, the labeled point cloud in the first point cloud set and the pseudo-label point cloud in the fourth point cloud set can be randomly sampled to obtain multiple mixed point clouds.

[0070] For example, for ease of understanding, each unlabeled point cloud with a pseudo label in the filtered fourth point cloud set can be called a pseudo-labeled point cloud, and the corresponding expression is (x U ,y U ), where x U Indicates that there is a pseudo-label point cloud, y U Represents the pseudo-label corresponding to the pseudo-label point cloud; each labeled point cloud in the first point cloud set is represented as (x L ,y L ), where x L Indicates that there is a labeled point cloud, y L Indicates the label corresponding to the annotated point cloud; further, for each pseudo-labeled point cloud (xU ,y U ) includes multiple points and each annotated point cloud (x L ,y L ) are randomly sampled to obtain a first sampling point set and a second sampling point set, and the first sampling point set and the second sampling point set are mixed to obtain a corresponding mixed point cloud, which is represented by The expressions corresponding to the random sampling process can be found in formula (2) and formula (3):

[0071]

[0072] in, represents one of the mixed point clouds, represents the pseudo label corresponding to the mixed point cloud; where P and Q are two binary masks used to randomly select sampling points and satisfy the constraints shown in formula (4):

[0073]

[0074] In the embodiment of the present application, by performing the above processing on both the annotated point cloud in the first point cloud set and the pseudo-labeled point cloud in the fourth point cloud set, a plurality of mixed point clouds can be obtained, which are represented as Here, N LU Indicates the number of mixed point clouds.

[0075] Further, after obtaining a plurality of mixed point clouds, a mixed point cloud set may be determined based on the plurality of mixed point clouds.

[0076] Step 103: Use the first point cloud set and the mixed point cloud set to train the first detection model to obtain a target detection model.

[0077] In the embodiment of the present application, after obtaining the mixed point cloud set according to the above steps, the first detection model can be trained using the first point cloud set and the mixed point cloud set to obtain the target detection model. For example, in the autonomous driving scenario, the first detection model can be a detection model deployed on the vehicle side.

[0078] In some embodiments, training the detection model using the first point cloud set and the mixed point cloud set may include: acquiring training point clouds from the first point cloud set and the mixed point cloud set according to a set ratio; and training the detection model using the training point clouds.

[0079] Here, the value of the set ratio can be set according to actual conditions, and the embodiments of the present application are not limited to this; for example, the value of the set ratio can be 1:1 or 1:1.5, etc.

[0080] Exemplarily, assuming that the ratio is set to 1:1, the same amount of training point clouds can be obtained from the first point cloud set and the mixed point cloud set according to the ratio. Then, the detection model is trained using the training point clouds to obtain a target detection model, which can then be used for target recognition.

[0081] It can be seen that in the embodiment of the present application, after determining the pseudo-labels of each unlabeled point cloud, the pseudo-labels of these unlabeled point clouds are not directly used for model training, but the first point cloud set and the third point cloud set with pseudo-labels are mixed, so that the point clouds in the mixed point cloud set after mixing include not only the information of the unlabeled point cloud, but also the information of the labeled point cloud. In this way, the inconsistency between the unlabeled point cloud and the labeled point cloud can be reduced; further, if the first point cloud set and the mixed point cloud set are used to jointly train the detection model, the excessive reliance on pseudo-labels in the model training process can be reduced, the model training effect can be ensured, and the model detection performance can be improved.

[0082] In the embodiment of the present application, after obtaining multiple hybrid point clouds according to the above steps, the multiple hybrid point clouds can also be updated using the first point cloud set, and the hybrid point cloud set is determined according to each updated hybrid point cloud, and then the first point cloud set and the hybrid point cloud set are used to jointly train the first detection model to obtain the target detection model. The following is an exemplary description of the update process of the hybrid point cloud.

[0083] In some embodiments, determining a mixed point cloud set based on multiple mixed point clouds may include: when determining that the first point cloud set includes a target point cloud that has an overlapping area with any mixed point cloud, using the target point cloud to update the multiple mixed point clouds to obtain updated mixed point clouds; and determining the mixed point cloud set based on the updated mixed point clouds.

[0084] Here, the target point cloud is one of the labeled point clouds in the first point cloud set. According to the above content, the point cloud is the point cloud of a three-dimensional target. Therefore, it can be determined whether the first point cloud set includes a target point cloud that has an overlapping area with any mixed point cloud. If it is determined to be so, the target point cloud is used to update multiple mixed point clouds to obtain updated mixed point clouds.

[0085] Exemplarily, determining whether the first point cloud set includes a target point cloud that has an overlapping area with any mixed point cloud may include: obtaining a first detection frame of each labeled point cloud in the first point cloud set and a second detection frame of each mixed point cloud; and determining whether the first point cloud set includes a target point cloud that has an overlapping area with any mixed point cloud based on the overlap between the first detection frame of each labeled point cloud and the second detection frame of each mixed point cloud.

[0086] Exemplarily, the first detection frame of each labeled point cloud can be obtained during the training of the second detection model; in addition, the detection results obtained by using the trained second detection model also include the second detection frame of each unlabeled point cloud in the second point cloud set; therefore, the second detection frame of each mixed point cloud can be obtained therefrom; here, the first detection frame and the second detection frame are both three-dimensional detection frames, which may include position information and size information of the corresponding point cloud.

[0087] In an embodiment of the present application, after obtaining the first detection frame of each labeled point cloud and the second detection frame of each mixed point cloud, it can be determined whether the first point cloud set includes a target point cloud that has an overlapping area with any mixed point cloud based on the overlap between the first detection frame of each labeled point cloud and the second detection frame of each mixed point cloud.

[0088] It can be understood that since the first detection frame of each labeled point cloud and the second detection frame of each mixed point cloud include the position information and size information of the point cloud, it is possible to determine, based on the first detection frame of each labeled point cloud and the second detection frame of each mixed point cloud, whether the first point cloud set includes a target point cloud that has an overlapping area with any mixed point cloud. For example, it is possible to determine whether there is an overlap between the first detection frame of each labeled point cloud and the second detection frame of each mixed point cloud, and then determine whether the first point cloud set includes a target point cloud that has an overlapping area with any mixed point cloud based on the judgment result.

[0089] Exemplarily, the number of target point clouds may be one or more. When there are multiple target point clouds, the process of updating the mixed point cloud using each target point cloud is similar. For ease of description, one target point cloud will be used as an example for explanation.

[0090] Exemplarily, if the above judgment result indicates that there is an overlap between the first detection frame of a labeled point cloud and the second detection frame of a mixed point cloud, it means that the labeled point cloud is the target point cloud, that is, whether there is a target point cloud in the first point cloud set that overlaps with any mixed point cloud. At this time, the target point cloud can be used to update multiple mixed point clouds to obtain updated mixed point clouds.

[0091] In some embodiments, updating multiple hybrid point clouds using a target point cloud to obtain updated hybrid point clouds may include: determining each target hybrid point cloud to be replaced from the hybrid point cloud set; and replacing each target hybrid point cloud with the target point cloud to obtain updated hybrid point clouds.

[0092] In an embodiment of the present application, according to the above steps, each mixed point cloud that has an overlapping area with the annotated point cloud can be determined from multiple mixed point clouds, and then, the target mixed point clouds to be replaced can be determined from the mixed point cloud set based on the determined mixed point clouds.

[0093] Exemplarily, the target mixed point clouds to be replaced may include mixed point clouds having overlapping areas with the target point cloud, and mixed point clouds whose distance from the second detection frame of the mixed point cloud is within a set range; here, the value of the set range can be set according to actual conditions, for example, it can be 0.2m or 0.5m, etc., and is not specifically limited here.

[0094] Here, the target mixed point cloud can be one or more; further, after determining the target mixed point clouds to be replaced, the target point cloud can be used to replace each target mixed point cloud to obtain a mixed point cloud set; here, replacing each target mixed point cloud with the target point cloud means replacing each target mixed point cloud and the corresponding second detection frame with the target point cloud and the corresponding first detection frame.

[0095] It can be understood that when there is overlap between the annotated points and the mixed point cloud, the mixed point cloud and the corresponding second detection frame are replaced by the annotated point cloud and the corresponding first detection frame, which can ensure that the mixed point cloud corresponds to the first detection frame with the annotated points, thereby improving the data reliability of the mixed point cloud set.

[0096] Exemplarily, after obtaining the updated mixed point clouds, a mixed point cloud set can be determined based on the updated mixed point clouds, and then the detection model can be trained together using the first point cloud set and the mixed point cloud set to obtain a target detection model; here, the mixed point cloud set includes the updated mixed point clouds.

[0097] It can be understood that training the detection model using point clouds trained from annotated point cloud sets and mixed point cloud sets can ensure that the model learns the real labeled point cloud while learning the pseudo-labeled point cloud improved by using the real labeled point cloud. In this way, the robustness of the detection model for pseudo-labeled point clouds is improved, and the model performance is improved.

[0098] In order to better reflect the purpose of this application, based on the above-mentioned embodiments of this application, further examples are given in combination with the autonomous driving scenario.

[0099] Figure 2 is a flow chart of another model training method in an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0100] Step 200: Train the cloud model.

[0101] Exemplarily, a first point cloud set including a plurality of annotated point clouds may be obtained, and a cloud model may be trained using the first point cloud set to obtain a trained cloud model; here, the cloud model corresponds to the above-mentioned first detection model.

[0102] Step 201: Use the trained cloud model to detect each unlabeled point cloud in the second point cloud set.

[0103] Exemplarily, a second point cloud set including multiple unlabeled point clouds can be obtained, and each unlabeled point cloud in the second point cloud set can be detected using a trained cloud model to obtain a third point cloud set; here, the third point cloud set can include pseudo labels for each unlabeled point cloud, a second detection box, and pseudo label confidence.

[0104] Step 202: Filter pseudo labels using pseudo label confidence.

[0105] Exemplarily, the pseudo-label confidence of each unlabeled point cloud in the third point cloud set may be compared with a confidence threshold, and the unlabeled point clouds with a confidence threshold less than or equal to the confidence threshold may be filtered to obtain a fourth point cloud set.

[0106] Step 203: Randomly sample the marked point cloud and the pseudo-labeled point cloud to obtain multiple mixed point clouds.

[0107] Step 204: Update the multiple mixed point clouds using the target point cloud in the first point cloud set.

[0108] Exemplarily, a target point cloud that has an overlapping area with any mixed point cloud can be determined from the first point cloud set, and then the target point cloud can be used to update multiple mixed point clouds; here, whether the point clouds overlap is determined based on whether the first detection frame of the annotated point cloud overlaps the second detection frame of the mixed point cloud.

[0109] Step 205: Obtain training point clouds from the first point cloud set and the mixed point cloud set according to a set ratio to train the vehicle-side model.

[0110] Here, the vehicle-side model corresponds to the first detection module, which is deployed in the autonomous driving vehicle. For example, after obtaining the mixed point cloud set, training point clouds can be obtained from the first point cloud set and the mixed point cloud set according to a set ratio, and the vehicle-side model can be trained together to improve the generalization ability of the model.

[0111] It can be seen that in the embodiment of the present application, using the first point cloud set to update multiple mixed point clouds can reduce the inconsistency between the pseudo-label distribution and the true label distribution, so that the model can fully utilize the pseudo-annotated point cloud while avoiding overfitting of the pseudo-annotated point cloud, thereby improving the model training effect.

[0112] For ease of understanding, Figure 2 Based on Figure 3 Further explanation of the model training process, Figure 3 is a structural diagram corresponding to a model training method in an embodiment of the present application, such as Figure 3 As shown, the vehicle-side model may include a three-dimensional detector 31 and a three-dimensional prediction head 32, and the three-dimensional prediction head 32 is located at the output end of the detection model.

[0113] Exemplarily, after obtaining a mixed point cloud set based on a labeled point cloud set (corresponding to the first point cloud set mentioned above) and a pseudo-annotated point cloud set (corresponding to the fourth point cloud set mentioned above), a training point cloud can be obtained from the labeled point cloud set and the mixed point cloud set according to a set ratio, and feature detection can be performed on the training point cloud using a three-dimensional detector 31. The feature detection result is converted into a final detection result and output using a three-dimensional prediction head 32; wherein the detection result includes a pseudo-label of the training point cloud and a pseudo-label confidence.

[0114] It can be seen that the embodiment of the present application, by mixing the real label point cloud and the pseudo label point cloud, uses the information of the real label point cloud to improve the distribution of the pseudo label point cloud, thereby improving the model training effect in the absence of a large number of labeled point clouds.

[0115] Figure 4 A flow chart of a target detection method according to an embodiment of the present application is shown in FIG. Figure 4 As shown, the method comprises the following steps:

[0116] Step 300: Obtain a detection point cloud set;

[0117] Step 301: Use the target detection model to predict at least one unlabeled point cloud to be detected to obtain a prediction result.

[0118] Exemplarily, the detection point cloud set may include at least one unlabeled point cloud to be detected; here, the unlabeled point cloud to be detected refers to the unlabeled point cloud of the three-dimensional target to be detected, and each three-dimensional target to be detected corresponds to an unlabeled point cloud.

[0119] In an embodiment of the present application, after obtaining a detection point cloud set, the detection point cloud set can be input into a target detection model for prediction to obtain a prediction result; here, the prediction result may include a prediction label of at least one unlabeled point cloud to be detected, wherein the prediction label is the category to which the unlabeled point cloud to be detected belongs.

[0120] In an embodiment of the present application, the target detection model can be obtained according to the above steps 100 to 103. To avoid repeated description, it will not be repeated here.

[0121] It is understandable that since the target detection model uses a pseudo-label point cloud improved by using the real label point cloud in addition to the labeled data set during the training process, this can reduce the model's dependence on a large number of pseudo-label point clouds and ensure the target detection performance of the detection model.

[0122] Figure 5 It is a schematic diagram of the structure of the model training device in the embodiment of the present application. Figure 5 As shown, the device includes: a first acquisition module 400, a determination module 401, a mixing module 402 and a training module 403, wherein:

[0123] A first acquisition module 400 is used to acquire a first point cloud set and a second point cloud set; the first point cloud set includes a plurality of annotated point clouds, and the second point cloud set includes a plurality of unannotated point clouds;

[0124] A determination module 401 is used to determine the pseudo labels of each unlabeled point cloud in the second point cloud set by using the first point cloud set; and obtain a third point cloud set based on the each unlabeled point cloud and the pseudo labels of the each unlabeled point cloud;

[0125] A mixing module 402, configured to mix the first point cloud set and the third point cloud set to obtain a mixed point cloud set;

[0126] The training module 403 is used to train the first detection model using the first point cloud set and the mixed point cloud set to obtain a target detection model.

[0127] In some embodiments, the mixing of the first point cloud set and the third point cloud set to obtain a mixed point cloud set includes:

[0128] Performing filtering processing on the third point cloud set to obtain a fourth point cloud set;

[0129] Randomly sampling a plurality of points included in each labeled point cloud in the first point cloud set to obtain a plurality of first sampling point sets corresponding to the first point cloud set;

[0130] Randomly sampling a plurality of points included in each pseudo-labeled point cloud in the fourth point cloud set to obtain a plurality of second sampling point sets corresponding to the fourth point cloud set;

[0131] Performing a mixing process on every two sampling point sets among the plurality of first sampling point sets and the plurality of second sampling point sets to obtain a plurality of mixed point clouds;

[0132] Based on the multiple mixed point clouds, the mixed point cloud set is determined.

[0133] In some embodiments, determining the hybrid point cloud set based on the plurality of hybrid point clouds comprises:

[0134] When it is determined that the first point cloud set includes a target point cloud having an overlapping area with any mixed point cloud, the plurality of mixed point clouds are updated using the target point cloud to obtain updated mixed point clouds;

[0135] The mixed point cloud set is determined according to the updated mixed point clouds.

[0136] In some embodiments, the method further comprises:

[0137] Obtain a first detection frame for each annotated point cloud and a second detection frame for each mixed point cloud in the first point cloud set;

[0138] According to the overlapping conditions of the first detection frames of the respective annotated point clouds and the second detection frames of the respective mixed point clouds, it is determined whether the first point cloud set includes the target point cloud having an overlapping area with any mixed point cloud.

[0139] In some embodiments, the updating of the plurality of mixed point clouds using the target point cloud to obtain updated mixed point clouds includes:

[0140] Determining target mixed point clouds to be replaced from the mixed point cloud set; the target mixed point clouds include mixed point clouds with overlapping areas with the target point cloud, and mixed point clouds with a distance from a second detection frame of the mixed point cloud within a set range;

[0141] The target point cloud is used to replace the target mixed point clouds to obtain the updated mixed point clouds.

[0142] In some embodiments, using the first point cloud set to determine pseudo labels for each unlabeled point cloud in the second point cloud set includes:

[0143] Using the first point cloud set to train a second detection model to obtain a trained second detection model;

[0144] The trained second detection model is used to detect each unlabeled point cloud in the second point cloud set to obtain a pseudo label for each unlabeled point cloud.

[0145] In some embodiments, filtering the third point cloud set to obtain a fourth point cloud set includes:

[0146] Obtaining pseudo-label confidences of a plurality of pseudo-labeled point clouds in the third point cloud set;

[0147] The pseudo-label point clouds whose pseudo-label confidences in the third point cloud set are less than or equal to the confidence threshold are filtered to obtain the fourth point cloud set.

[0148] In some embodiments, the training of the first detection model using the first point cloud set and the mixed point cloud set includes:

[0149] Acquire a training point cloud from the first point cloud set and the mixed point cloud set according to a set ratio;

[0150] The first detection model is trained using the training point cloud.

[0151] Figure 6 is a schematic diagram of the structure of the target detection device in the embodiment of the present application. Figure 6 As shown, the device includes: a second acquisition module 500 and a prediction module 501, wherein:

[0152] The second acquisition module 500 is used to acquire a predicted point cloud set; the predicted point cloud set includes at least one unlabeled point cloud of a target to be detected;

[0153] The prediction module 501 is used to use the target detection model to predict the unlabeled point cloud of the at least one target to be detected to obtain a prediction result; the prediction result includes the predicted label of the at least one target to be detected; wherein the target detection model is trained according to the method provided by one or more of the aforementioned technical solutions.

[0154] In actual applications, the above-mentioned first acquisition module 400, determination module 401, mixing module 402, training module 403, second acquisition module 500 and prediction module 501 can all be implemented by a processor located in an electronic device, and the processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0155] In addition, each functional module in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software functional modules.

[0156] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent commodity, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the relevant technology or the whole or part of the technical solution can be embodied in the form of a software commodity. The computer software commodity is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc. Various media that can store program codes.

[0157] Specifically, the computer program instructions corresponding to a model training method or a target detection method in this embodiment can be stored on a storage medium such as a CD, a hard disk, or a U disk. When the computer program instructions corresponding to a model training method or a target detection method in the storage medium are read or executed by an electronic device, any model training method or target detection method in the aforementioned embodiments is implemented.

[0158] Based on the same technical concept as the above embodiments, see Figure 7 , which shows an electronic device 600 provided in an embodiment of the present application, which may include: a memory 601 and a processor 602; wherein,

[0159] Memory 601, used for storing computer programs and data;

[0160] Processor 602 is used to execute the computer program stored in the memory to implement any one of the model training methods or target detection methods in the aforementioned embodiments.

[0161] In practical applications, the memory 601 may be a volatile memory, such as RAM; or a non-volatile memory, such as ROM, flash memory, hard disk drive (HDD) or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 602.

[0162] The processor 602 may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understandable that for different electronic devices, the electronic device used to implement the processor function may also be other, which is not specifically limited in the embodiments of the present application.

[0163] The present application provides a computer program product, including a computer program, which, when executed by a processor, implements any one of the model training methods or target detection methods of the aforementioned embodiments.

[0164] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0165] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0166] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0167] The features disclosed in the various commodity embodiments provided in this application can be arbitrarily combined without conflict to obtain new commodity embodiments.

[0168] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0169] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program codes.

[0170] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0172] The above are only preferred embodiments of the present application and are not intended to limit the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a first point cloud set and a second point cloud set; the first point cloud set includes a plurality of annotated point clouds, and the second point cloud set includes a plurality of unannotated point clouds; Determine the pseudo labels of each unlabeled point cloud in the second point cloud set by using the first point cloud set, and obtain a third point cloud set based on each unlabeled point cloud and the pseudo labels of each unlabeled point cloud; Mixing the first point cloud set and the third point cloud set to obtain a mixed point cloud set; The first point cloud set and the mixed point cloud set are used to train a first detection model to obtain a target detection model.

2. The method according to claim 1, characterized in that The mixing process of the first point cloud set and the third point cloud set to obtain a mixed point cloud set includes: Performing filtering processing on the third point cloud set to obtain a fourth point cloud set; Randomly sampling a plurality of points included in each labeled point cloud in the first point cloud set to obtain a plurality of first sampling point sets corresponding to the first point cloud set; Randomly sampling a plurality of points included in each pseudo-labeled point cloud in the fourth point cloud set to obtain a plurality of second sampling point sets corresponding to the fourth point cloud set; Performing a mixing process on every two sampling point sets among the plurality of first sampling point sets and the plurality of second sampling point sets to obtain a plurality of mixed point clouds; Based on the multiple mixed point clouds, the mixed point cloud set is determined.

3. The method according to claim 2, characterized in that The step of determining the mixed point cloud set based on the plurality of mixed point clouds comprises: When it is determined that the first point cloud set includes a target point cloud having an overlapping area with any mixed point cloud, the plurality of mixed point clouds are updated using the target point cloud to obtain updated mixed point clouds; The mixed point cloud set is determined according to the updated mixed point clouds.

4. The method according to claim 3, characterized in that The method further comprises: Obtain a first detection frame for each annotated point cloud and a second detection frame for each mixed point cloud in the first point cloud set; According to the overlapping conditions of the first detection frames of the respective annotated point clouds and the second detection frames of the respective mixed point clouds, it is determined whether the first point cloud set includes the target point cloud having an overlapping area with any mixed point cloud.

5. The method according to claim 4, characterized in that The step of updating the plurality of mixed point clouds by using the target point cloud to obtain updated mixed point clouds includes: Determining target mixed point clouds to be replaced from the mixed point cloud set; the target mixed point clouds include mixed point clouds with overlapping areas with the target point cloud, and mixed point clouds with a distance from a second detection frame of the mixed point cloud within a set range; The target point cloud is used to replace the target mixed point clouds to obtain the updated mixed point clouds.

6. The method according to claim 1, characterized in that The step of using the first point cloud set to determine pseudo labels of each unlabeled point cloud in the second point cloud set includes: Using the first point cloud set to train a second detection model to obtain a trained second detection model; The trained second detection model is used to detect each unlabeled point cloud in the second point cloud set to obtain a pseudo label for each unlabeled point cloud.

7. The method according to claim 2, characterized in that The filtering process on the third point cloud set to obtain a fourth point cloud set includes: Obtaining pseudo-label confidences of a plurality of pseudo-labeled point clouds in the third point cloud set; The pseudo-label point clouds whose pseudo-label confidences in the third point cloud set are less than or equal to the confidence threshold are filtered to obtain the fourth point cloud set.

8. The method according to claim 1, characterized in that: The using the first point cloud set and the mixed point cloud set to train the first detection model includes: Acquire a training point cloud from the first point cloud set and the mixed point cloud set according to a set ratio; The first detection model is trained using the training point cloud.

9. A target detection method, characterized in that: The method comprises: Acquire a detection point cloud set; the detection point cloud set includes at least one unlabeled point cloud to be detected; The target detection model is used to predict the at least one unlabeled point cloud to be detected to obtain a prediction result; the prediction result includes a predicted label of the at least one unlabeled point cloud to be detected; wherein the target detection model is trained according to the method described in any one of claims 1 to 7.

10. A model training device, characterized in that: The device comprises: A first acquisition module, used to acquire a first point cloud set and a second point cloud set; the first point cloud set includes a plurality of annotated point clouds, and the second point cloud set includes a plurality of unannotated point clouds; A determination module, configured to determine the pseudo labels of each unlabeled point cloud in the second point cloud set by using the first point cloud set; and obtain a third point cloud set based on each unlabeled point cloud and the pseudo labels of each unlabeled point cloud; A mixing module, used for mixing the first point cloud set and the third point cloud set to obtain a mixed point cloud set; A training module is used to train a first detection model using the first point cloud set and the mixed point cloud set to obtain a target detection model.

11. A target detection device, characterized in that: The device comprises: A second acquisition module is used to acquire a detection point cloud set; the detection point cloud set includes at least one unlabeled point cloud to be detected; A prediction module is used to use a target detection model to predict the at least one unlabeled point cloud to be detected to obtain a prediction result; the prediction result includes a predicted label of the at least one unlabeled point cloud to be detected; wherein the target detection model is trained according to the method provided by one or more of the aforementioned technical solutions.

12. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the method according to any one of claims 1 to 9 when executing the program.

13. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images