BEV 3D target detection model fine tuning method based on 2D supervision

Abstract

The invention relates to the technical field of automatic driving, in particular to a BEV 3D target detection model fine tuning method based on 2D supervision, and the method comprises the steps: inputting a target multi-view image into a pre-trained 3D target detection model and a pre-trained 2D target detection model, and obtaining a 3D detection frame prediction output result and a 2D detection frame prediction output result respectively, in combination with a preset detection frame projection strategy, obtaining a 2D projection detection frame corresponding to the 3D detection frame prediction output result, and matching the 2D projection detection frame with a preset 2D detection frame true value to obtain a matching result; and optimizing a target loss function of the pre-trained 3D target detection model by utilizing a matching result and preset 2D annotation data so as to generate a 3D target detection fine-tuning model for supervising and fine-tuning the 2D annotation data. According to the method, when accurate depth information cannot be obtained, the data annotation cost can be saved through a 2D information supervision fine tuning mode, the model deployment speed is improved, and model updating is efficiently realized at low cost.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a BEV 3D target detection model fine-tuning method based on 2D supervision. Background Art

[0002] The rapid development of autonomous vehicle technology has brought many conveniences to people's lives. In autonomous driving technology, accurate and real-time environmental perception is a crucial link in achieving autonomous navigation, which directly affects the driving safety of the vehicle and the comfort of the passengers. Autonomous vehicles need to accurately understand the vehicles, pedestrians and obstacles in the surrounding environment, and infer their positions and motion states in three-dimensional space to ensure that autonomous vehicles can drive safely and accurately. Bird's-Eye-View (BEV) perception algorithm has gradually attracted attention as an important means of environmental perception. It presents the environment around the vehicle in the form of a bird's-eye view, intuitively showing the position of the target in three-dimensional space, so that the vehicle can better perceive the surrounding environment.

[0003] Among the related technologies, many advanced algorithms have been widely used in autonomous driving target detection, such as BEVFormer (a purely visual autonomous driving perception algorithm) and BEVDet (a 3D detection technology based on Bird's Eye View). These algorithms are trained with a large amount of 3D labeled data and conduct targeted learning for specific task scenarios, achieving the effect of automatically extracting image features and accurately identifying various targets on the road. At the same time, in order to improve the accuracy and real-time performance of detection, research on algorithm optimization and hardware acceleration is also continuing to move forward.

[0004] However, the autonomous driving target detection algorithms in related technologies have low robustness in complex scenarios. Some target detection algorithms cannot take into account cross-domain generalization capabilities while improving accuracy. Most target detection algorithms rely on a large amount of 3D labeled data to train the models. Data acquisition and labeling are time-consuming and costly, but they are difficult to adapt to the updates and iterations of vehicle models or mission scenarios in real life, which needs to be solved urgently. Summary of the invention

[0005] The present application provides a BEV 3D target detection model fine-tuning method based on 2D supervision to solve the problems that the autonomous driving target detection algorithms in related technologies have low robustness in complex scenarios, some target detection algorithms cannot take into account cross-domain generalization capabilities while improving accuracy, and most target detection algorithms rely on a large amount of labeled data to train the models. Data acquisition and labeling are time-consuming and costly, but are difficult to adapt to the update and iteration of vehicle models or mission scenarios in real life.

[0006] In a first aspect, an embodiment of the present application provides a BEV 3D target detection model fine-tuning method based on 2D supervision, comprising the following steps: inputting a target multi-view image into a pre-trained 3D target detection model and a pre-trained 2D target detection model to obtain a 3D detection box prediction output result and a 2D detection box prediction output result, respectively; based on the 3D detection box prediction output result, the 2D detection box prediction output result and a preset detection box projection strategy, obtaining a 2D projection detection box corresponding to the 3D detection box prediction output result, and matching the 2D projection detection box with a preset 2D detection box true value to obtain a matching result; and optimizing a target loss function of the pre-trained 3D target detection model using the matching result and preset 2D annotation data to generate a 3D target detection fine-tuning model supervised by the 2D annotation data.

[0007] Optionally, in one embodiment of the present application, after fine-tuning the target loss function using the matching result and the preset 2D annotation data, it also includes: fine-tuning the pre-trained 2D target detection model according to the 2D annotation data, and determining whether the fine-tuned pre-trained 2D target detection model meets the preset detection performance requirements; when the fine-tuned pre-trained 2D target detection model meets the preset detection performance requirements, inputting the preset unlabeled multi-view image into the pre-trained 3D target detection model and the pre-trained 2D target detection model respectively to output a second 3D detection box prediction output result and a second 2D detection box prediction output result; obtaining a second 2D projection detection box corresponding to the second 3D detection box prediction output result, and matching the second 2D projection detection box with the second 2D detection box prediction output result to obtain an unlabeled matching result; based on the unlabeled matching result, optimizing the target loss function to generate a 3D target detection fine-tuning model corresponding to the unlabeled multi-view image.

[0008] Optionally, in one embodiment of the present application, based on the 3D detection frame prediction output result, the 2D detection frame prediction output result and the preset detection frame projection strategy, the 2D projection detection frame corresponding to the 3D detection frame prediction output result is obtained, and the 2D projection detection frame and the preset 2D detection frame true value are matched to obtain a matching result, including: performing coordinate conversion on the 3D detection frame prediction output result in the BEV space coordinate system to the image coordinate system of each camera to obtain a 3D detection frame in the image coordinate system; converting the 3D detection frame in the image coordinate system into a 2D detection frame to obtain the 2D projection detection frame corresponding to the 3D detection frame prediction output result; based on the image coordinate system, matching the 2D projection detection frame with the preset 2D detection frame true value to obtain the matching result.

[0009] Optionally, in one embodiment of the present application, the 3D detection frame prediction output result in the BEV space coordinate system is converted to the image coordinate system of each camera, and the expression formula of the 3D detection frame in the image coordinate system is obtained as follows:

[0010] P img =K·T·P BEV

[0011] Among them, K is the camera internal parameter matrix, T is the camera external parameter matrix, P img and P BEV They are the coordinates of the 3D detection box corner points in the image coordinate system and the BEV space coordinate system respectively.

[0012] The second aspect of the present application provides a BEV 3D target detection model fine-tuning device based on 2D supervision, including: a processing module, used to input the target multi-view image into a pre-trained 3D target detection model and a pre-trained 2D target detection model, and obtain a 3D detection box prediction output result and a 2D detection box prediction output result respectively; a projection module, used to obtain a 2D projection detection box corresponding to the 3D detection box prediction output result based on the 3D detection box prediction output result, the 2D detection box prediction output result and a preset detection box projection strategy, and match the 2D projection detection box with a preset 2D detection box true value to obtain a matching result; a fine-tuning module, used to optimize the target loss function of the pre-trained 3D target detection model using the matching result and preset 2D annotation data to generate a 3D target detection fine-tuning model supervised by the 2D annotation data.

[0013] Optionally, in one embodiment of the present application, it also includes: a judgment module, which is used to fine-tune the pre-trained 2D target detection model according to the 2D annotation data after fine-tuning the target loss function using the matching result and the preset 2D annotation data, and judge whether the fine-tuned pre-trained 2D target detection model meets the preset detection performance requirements; an output module, which is used to input the preset unlabeled multi-view image into the pre-trained 3D target detection model and the pre-trained 2D target detection model respectively when the fine-tuned pre-trained 2D target detection model meets the preset detection performance requirements, so as to output a second 3D detection box prediction output result and a second 2D detection box prediction output result; a matching module, which is used to obtain a second 2D projection detection box corresponding to the second 3D detection box prediction output result, and match the second 2D projection detection box with the second 2D detection box prediction output result to obtain an unlabeled matching result; a generation module, which is used to optimize the target loss function based on the unlabeled matching result, and generate a 3D target detection fine-tuning model corresponding to the unlabeled multi-view image.

[0014] Optionally, in one embodiment of the present application, the projection module includes: a first conversion unit, used to convert the 3D detection frame prediction output result in the BEV space coordinate system to the image coordinate system of each camera to obtain the 3D detection frame in the image coordinate system; a second conversion unit, used to convert the 3D detection frame in the image coordinate system into a 2D detection frame to obtain a 2D projection detection frame corresponding to the 3D detection frame prediction output result; a matching unit, used to match the 2D projection detection frame with a preset 2D detection frame true value based on the image coordinate system to obtain the matching result.

[0015] Optionally, in one embodiment of the present application, the 3D detection frame prediction output result in the BEV space coordinate system is converted to the image coordinate system of each camera, and the expression formula of the 3D detection frame in the image coordinate system is obtained as follows:

[0016] P img =K·T·P BEV

[0017] Among them, K is the camera internal parameter matrix, T is the camera external parameter matrix, P img and P BEV They are the coordinates of the 3D detection box corner points in the image coordinate system and the BEV space coordinate system respectively.

[0018] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the BEV 3D target detection model fine-tuning method based on 2D supervision as described in the above embodiment.

[0019] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the program is executed by a processor, it implements the above-mentioned BEV 3D target detection model fine-tuning method based on 2D supervision.

[0020] The fifth aspect embodiment of the present application provides a computer program product, including a computer program, which, when executed, is used to implement the above-mentioned BEV 3D target detection model fine-tuning method based on 2D supervision.

[0021] The embodiments of the present application can use 2D annotation data to achieve synchronous fine-tuning of the pre-trained 2D object detection model and the pre-trained 3D object detection model based on a pre-trained 3D object detection model and a pre-trained 2D object detection model, so as to obtain a 2D object detection fine-tuning model and a 3D object detection fine-tuning model, and can use the 2D object detection fine-tuning model to generate 2D pseudo labels to achieve automatic fine-tuning of the 3D object detection fine-tuning model. The fine-tuning framework based on 2D supervision solves the problems of difficulty and high cost in obtaining 3D annotation data during fine-tuning in actual vehicle deployment, reduces dependence on large-scale annotation data, reduces fine-tuning costs, and accelerates the deployment of the BEV 3D object detection model and subsequent iterative optimization. At the same time, the 3D detection frame projection is independent of the BEV 3D object detection model, and the 2D detection frame is quickly obtained by simple coordinate transformation. The pluggable feature enables the fine-tuning framework of the present application to be quickly applied to a variety of 3D detection models and other BEV perception models. This solves the problem that the autonomous driving target detection algorithms in related technologies have low robustness in complex scenarios, and that some target detection algorithms cannot take into account cross-domain generalization capabilities while improving accuracy. In addition, most target detection algorithms rely on a large amount of 3D annotated data to train the models. Data acquisition and annotation are time-consuming and costly, and are difficult to adapt to the updates and iterations of vehicle models or mission scenarios in real life.

[0022] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0024] Figure 1 A flowchart of a BEV 3D target detection model fine-tuning method based on 2D supervision according to an embodiment of the present application;

[0025] Figure 2 A schematic diagram of a fine-tuning framework based on 2D label supervision according to an embodiment of the present application;

[0026] Figure 3 This is a schematic diagram of a 3D detection frame projection according to an embodiment of the present application;

[0027] Figure 4 Schematic diagram of the structure of a BEV 3D target detection model fine-tuning device based on 2D supervision according to an embodiment of the present application;

[0028] Figure 5 Schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0029] Reference numerals:

[0030] 10-BEV 3D target detection model fine-tuning device based on 2D supervision: 100-processing module, 200-training module and 300-fine-tuning module; 501-memory, 502-processor and 503-communication interface. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0032] The following describes a BEV 3D object detection model fine-tuning method based on 2D supervision according to an embodiment of the present application with reference to the accompanying drawings. In view of the low robustness of the autonomous driving target detection algorithms in the related technologies mentioned in the above background technology in complex scenarios, the fact that some target detection algorithms cannot take into account cross-domain generalization capabilities while improving accuracy, and most target detection algorithms rely on a large amount of labeled data to train the model. Data acquisition and annotation take a long time and have high economic costs, but are difficult to adapt to the update and iteration of vehicle models or mission scenarios in real life. The present application provides a BEV3D target detection model fine-tuning method based on 2D supervision. In this method, based on a pre-trained 3D target detection model and a pre-trained 2D target detection model, 2D annotation data can be used to achieve synchronous fine-tuning of the pre-trained 2D target detection model and the pre-trained 3D target detection model to obtain a 2D target detection fine-tuning model and a 3D target detection fine-tuning model, and the 2D target detection fine-tuning model can be used to generate 2D pseudo-labels to achieve automatic fine-tuning of the 3D target detection fine-tuning model. Through the fine-tuning framework based on 2D supervision, the problem of difficulty and high cost in obtaining 3D annotation data during fine-tuning of actual vehicle deployment is solved, the dependence on large-scale annotation data is reduced, the fine-tuning cost is reduced, and the BEV is accelerated. The deployment and subsequent iterative optimization of the 3D target detection model are independent of the BEV 3D target detection model through 3D detection frame projection, and the 2D detection frame is quickly obtained using simple coordinate transformation. The pluggable feature allows the fine-tuning framework of this application to be quickly applied to a variety of 3D detection models and other BEV perception models. This solves the problems of low robustness of autonomous driving target detection algorithms in complex scenarios in related technologies, and that some target detection algorithms cannot take into account cross-domain generalization capabilities while improving accuracy, and that most target detection algorithms rely on a large amount of 3D annotated data to train the model. Data acquisition and annotation take a long time and have high economic costs, but are difficult to adapt to the update and iteration of vehicle models or mission scenarios in real life.

[0033] Specifically, Figure 1A flowchart of a BEV 3D object detection model fine-tuning method based on 2D supervision provided in an embodiment of the present application.

[0034] like Figure 1 As shown, the BEV 3D object detection model fine-tuning method based on 2D supervision includes the following steps:

[0035] In step S101, the target multi-view image is input into a pre-trained 3D target detection model and a pre-trained 2D target detection model to obtain a 3D detection box prediction output result and a 2D detection box prediction output result, respectively.

[0036] It can be understood that the target multi-view image here can be understood as multiple images of the same scene captured from different perspectives of the target object to be detected, for example: parallel perspective: this is the most common perspective, bird's-eye view, upward view, wide-angle view and telescopic view, etc.

[0037] In some embodiments, when the vehicle is driving automatically, it is necessary to accurately understand the vehicles, pedestrians and obstacles in the surrounding environment, and infer their positions and motion states in three-dimensional space to ensure that the autonomous driving vehicle can drive safely and accurately. Therefore, when detecting the target object, the embodiment of the present application needs to first input the target multi-view image, that is, the multi-view image of the object in the surrounding target scene, into the pre-trained 3D target detection model and the pre-trained 2D target detection model.

[0038] It should be noted that the pre-trained 3D target detection model and pre-trained 2D target detection model here can be understood as a 3D target detection model and a 2D target detection model pre-trained by professional and technical personnel in this field, and may be different according to actual conditions. They are only used for illustrative purposes here without any specific limitations.

[0039] After inputting the target multi-view image into the pre-trained 3D target detection model and the pre-trained 2D target detection model, the 3D detection box prediction output result and the 2D detection box prediction output result can be obtained. For example, the 2D detection box prediction output result may contain information such as the positioning of the target on the plane, and the 3D detection box prediction output result may contain depth information such as the size, position and movement direction of the target.

[0040] The embodiment of the present application can input the target multi-view image into a pre-trained 3D target detection model and a pre-trained 2D target detection model, and obtain the 3D detection box prediction output result and the 2D detection box prediction output result respectively, which helps to improve the accuracy of target detection.

[0041] Step S102, based on the 3D detection frame prediction output result, the 2D detection frame prediction output result and the preset detection frame projection strategy, obtain the 2D projection detection frame corresponding to the 3D detection frame prediction output result, and match the 2D projection detection frame with the preset 2D detection frame true value to obtain a matching result.

[0042] It is understandable that the preset detection frame projection strategy here can be understood as a preset projection strategy applicable to the detection frame, and the preset 2D detection frame ground truth here can be understood as some data truth values ​​of the preset 2D detection frame. For example, the 2D detection frame ground truth may include, but is not limited to, the center point coordinates, the width and height of the detection frame, etc.

[0043] As a possible implementation method, when fine-tuning the 3D object detection model based on 2D supervision, the embodiment of the present application needs to use 3D detection frame projection to implement 2D supervision. Among them, when performing 2D supervision training, it is necessary to first obtain the 2D projection detection frame corresponding to the 3D detection frame prediction output result, and then match the 2D projection detection frame with the preset 2D detection frame true value to obtain the matching result.

[0044] Among them, matching the 2D projected detection frame with the preset 2D detection frame truth value can be understood as a step in 2D supervised training. For example, the 3D detection model may predict 100 detection frames, but there are only 10 objects in the actual picture, that is, 10 detection frame truth values. Through matching, the prediction frame closest to the 10 detection frame truth values ​​can be found, which facilitates the subsequent loss training and further supervised training of the 10 detection frames.

[0045] Specifically, the matching can calculate the distance between all predicted boxes and the true value (including regression (coordinates, etc.), classification (object type) IoU), and match each true value with the predicted box with the smallest distance. For example, there is a true value and two predictions, and the regression part is:

[0046] True value 1: coordinates = (0,0), width and height = [1,2];

[0047] Prediction 1: coordinates = (0, 1), width and height = [1, 3];

[0048] Prediction 2: coordinates = (0, 1), width and height = [1, 2];

[0049] For the L1 distance between these two predictions,

[0050] Distance 1 = |0-0|+|1-0|+|1-1|+|3-2| = 2;

[0051] Distance 2 = |0-0|+|1-0|+|1-1|+|2-2| = 1;

[0052] This means that for the regression part, distance 2 < distance 1. Similarly, the classification and IoU parts are calculated. If distance 2 is still less than distance 1, prediction 2 is matched to true value 1, and the prediction 2 and true value 1 can be used for loss calculation. It should be noted that the calculation of this process will be more complicated in actual operation. For example, classification and regression will be multiplied by coefficients respectively, and the center coordinates and width and height of the regression will be normalized according to the image size. For example: λ_cls*(cost of the category)+λ_reg*(normalized coordinates, width and height L1 distance), coordinates, width and height normalization: if the image size is 1600*900, the detection box width and height = [1,2], and the normalized width is 1 / 1600, height 2 / 900, etc. The embodiments of this application are only for illustrative purposes and are not specifically limited.

[0053] This step will be further explained below.

[0054] Optionally, in one embodiment of the present application, based on the 3D detection frame prediction output result and the preset detection frame projection strategy, a 2D projection detection frame corresponding to the 3D detection frame prediction output result is obtained, and the 2D projection detection frame and the preset 2D detection frame true value are matched to obtain a matching result, including: performing coordinate conversion on the 3D detection frame prediction output result in the BEV space coordinate system to the image coordinate system of each camera to obtain a 3D detection frame in the image coordinate system; converting the 3D detection frame in the image coordinate system into a 2D detection frame to obtain a 2D projection detection frame corresponding to the 3D detection frame prediction output result; based on the image coordinate system, matching the 2D projection detection frame with the preset 2D detection frame true value to obtain a matching result.

[0055] Optionally, in one embodiment of the present application, the 3D detection frame prediction output result in the BEV space coordinate system is converted to the image coordinate system of each camera, and the expression formula of the 3D detection frame in the image coordinate system can be:

[0056] P img =K·T·P BEV

[0057] Among them, K is the camera internal parameter matrix, T is the camera external parameter matrix, P img and P BEV They are the coordinates of the 3D detection box corner points in the image coordinate system and the BEV space coordinate system respectively.

[0058] It is understandable that the embodiment of the present application needs to be based on 2D supervision when realizing the automatic fine-tuning of the 3D target detection model. The key step of realizing 2D supervision may be, but is not limited to, 3D detection frame projection. The key point is to project the prediction result in the BEV space coordinate system back to the coordinate system of the surround camera, such as Figure 3 As shown, it is a schematic diagram of the 3D detection frame projection of an embodiment of the present application, and the process is further described in detail below.

[0059] Taking BEVFormer as an example, the BEV space coordinate system it uses is the LiDAR coordinate system. First, the 3D detection frame prediction output results in the LiDAR coordinate system are converted to the image coordinate system of each camera through the LiDAR's extrinsic matrix, the camera's extrinsic matrix, and the camera's intrinsic matrix, and the coordinates of all 3D detection frame prediction output results in each image coordinate system are obtained. It can be expressed as follows:

[0060]

[0061] Among them, K, T cam 、T lidar They are camera intrinsic parameter, camera extrinsic parameter, and lidar extrinsic parameter matrix respectively. They are the coordinates of the corner points of the detection box in the laser radar and image coordinate systems respectively.

[0062] Then Normalize to the same plane (z=1 plane), determine whether all corner points of the detection frame fall within the picture according to the image size, and then assign the detection frame to each camera, and take the circumscribed rectangle of the 3D detection frame in the image coordinate system to obtain the 2D projection detection frame.

[0063] Then, the 2D projected detection frame is matched with a certain 2D detection frame true value to obtain the matching result for loss calculation. Repeat the operation until the 3D detection frame is projected onto all view image planes to achieve fine-tuning of the 3D detection model based on 2D annotation data.

[0064] The projection process in the embodiment of the present application only involves coordinate transformation, and the projection part is independent of the BEV perception model and is plug-and-play. Therefore, the present application can be applied to a variety of 3D detection models and other BEV perception models without changing the structure of the 3D detection part. Based on the same coordinate transformation principle, it is not only limited to 3D target detection, but can also be applied to 2D projection of lane lines, sidewalks, etc. of the real-time perception model of the map, effectively expanding the scope of application of the present application.

[0065] Step S103, using the matching results and the preset 2D annotation data to optimize the target loss function of the pre-trained 3D object detection model, so as to generate a 3D object detection fine-tuning model supervised by the 2D annotation data.

[0066] It can be understood by those skilled in the art that the preset 2D annotation data here can be understood as data containing 2D detection box true value (2D ground truth) that is manually annotated in advance.

[0067] In some embodiments, after obtaining the true value of the 2D detection box that matches the 3D detection box prediction output projection, the target loss function of the pre-trained 3D target detection model is optimized using certain 2D annotated data, that is, the loss calculation of the pre-trained 3D target detection model is performed using certain 2D annotated data, so as to fine-tune the 3D target detection model, thereby obtaining a 3D target detection fine-tuning model under 2D supervised training.

[0068] Among them, the loss calculation is the same as the matching calculation, which can be divided into three parts: classification, regression, and IoU, which can be expressed as follows:

[0069]

[0070] in, They are classification, regression, IoU loss, λ 1 , 2 , 3 The embodiments of the present application mainly but not limited to use Focal loss, L1 loss and GIoU loss.

[0071] Furthermore, when the present application uses 2D annotated data to perform loss calculation of the 3D target detection model to implement 2D supervised training, the fine-tuned performance indicators can be obtained at the same time, and the performance indicators can be used to evaluate the fine-tuning effect of the 3D detection model and confirm the reliability of the 2D detection model.

[0072] It should be noted that when the 3D object detection fine-tuning model obtained here is fine-tuned again, it is still necessary to use the multi-view images of the target containing labels as input to fine-tune the 3D object detection fine-tuning model again.

[0073] Optionally, in one embodiment of the present application, after fine-tuning the target loss function using the matching results and preset 2D annotation data, it also includes: fine-tuning the pre-trained 2D target detection model according to the 2D annotation data, and determining whether the fine-tuned pre-trained 2D target detection model meets the preset detection performance requirements; when the fine-tuned pre-trained 2D target detection model meets the preset detection performance requirements, inputting the preset unlabeled multi-view image into the pre-trained 3D target detection model and the pre-trained 2D target detection model respectively to output a second 3D detection box prediction output result and a second 2D detection box prediction output result; obtaining a second 2D projection detection box corresponding to the second 3D detection box prediction output result, and matching the second 2D projection detection box and the second 2D detection box prediction output result to obtain an unlabeled matching result; based on the unlabeled matching result, optimizing the target loss function, and generating a 3D target detection fine-tuning model corresponding to the unlabeled multi-view image.

[0074] Based on the relevant descriptions of other embodiments, it can be understood that the 3D object detection model can be fine-tuned by using certain 2D annotated data to perform loss calculation of the pre-trained 3D object detection model, thereby obtaining a 3D object detection fine-tuning model under 2D supervised training. When fine-tuning again, the multi-view images of the target containing labels can be used as input to achieve fine-tuning of the 3D object detection fine-tuning model.

[0075] Furthermore, in order to completely avoid manual labeling and reduce costs, the embodiments of the present application can fine-tune the pre-trained 2D target detection model while fine-tuning the pre-trained 3D target detection model using certain 2D labeling data, and detect whether the fine-tuned pre-trained 2D target detection model, i.e., the 2D target detection fine-tuning model, meets certain performance requirements, such as whether the stability is sufficient.

[0076] When it is detected that the 2D object detection fine-tuning model is stable enough, the accurate 2D detection frame generated by the pre-trained 2D detection model can be used as a label (pseudo-label), thereby realizing that the unlabeled multi-view image can be used as input and input into the pre-trained 3D object detection model and the pre-trained 2D object detection model, and the second 3D detection frame prediction output result and the second 2D detection frame prediction output result are output similarly; and then through the same projection process, the second 2D projection detection frame corresponding to the second 3D detection frame prediction output result is obtained, and the second 2D projection detection frame and the second 2D detection frame prediction output result are matched to obtain the matching result with the unlabeled image as input, which is used to optimize the target loss function, and when the target loss function meets the preset optimization end condition, a 3D object detection fine-tuning model that can realize automatic fine-tuning corresponding to the unlabeled multi-view image is generated.

[0077] It should be noted that in the embodiments of the present application, both the labeled target image as input and the unlabeled target image as input belong to 2D supervised fine-tuning. When the labeled target image is used as input, the 2D label, i.e., the 2D annotation data, is mainly used as the 2D true value to supervise the training for fine-tuning. When the unlabeled target image is used as input, there is no need to use the 2D true value, but the fine-tuned 2D target detection fine-tuning model is used to generate 2D pseudo-labels to supervise the training. Both implementation methods use 2D labels for supervision, but one is the true value marked manually, and the other is the pseudo-label supervision predicted by the 2D target detector (no manual marking is required, so it is called unlabeled here), and the 2D target detection fine-tuning model needs to be trained with the 2D true value before it can be actually used.

[0078] The embodiments of the present application can fine-tune the pre-trained 2D target detection model while fine-tuning the pre-trained 3D target detection model based on 2D supervised training, and use the 2D pseudo-label supervised training generated by the fine-tuned 2D target detection fine-tuning model to train the pre-trained 3D target detection model, thereby obtaining a 3D target detection fine-tuning model that can use unlabeled target multi-view images as input, thereby avoiding manual labeling work and achieving automated fine-tuning.

[0079] According to the BEV 3D target detection model fine-tuning method based on 2D supervision proposed in the embodiment of the present application, the pre-trained 3D target detection model and the pre-trained 2D target detection model can be used to synchronously fine-tune the pre-trained 2D target detection model and the pre-trained 3D target detection model based on 2D annotation data to obtain a 2D target detection fine-tuning model and a 3D target detection fine-tuning model, and the 2D target detection fine-tuning model can be used to generate 2D pseudo labels to achieve automatic fine-tuning of the 3D target detection fine-tuning model. The fine-tuning framework based on 2D supervision solves the problems of difficulty and high cost in obtaining 3D annotation data during fine-tuning in actual vehicle deployment, reduces dependence on large-scale annotation data, reduces fine-tuning costs, and accelerates the deployment of the BEV 3D target detection model and subsequent iterative optimization. At the same time, the 3D detection frame projection is independent of the BEV 3D target detection model, and the 2D detection frame is quickly obtained by simple coordinate transformation. The pluggable feature enables the fine-tuning framework of the present application to be quickly applied to a variety of 3D detection models and other BEV perception models. This solves the problem that the autonomous driving target detection algorithms in related technologies have low robustness in complex scenarios, and that some target detection algorithms cannot take into account cross-domain generalization capabilities while improving accuracy. In addition, most target detection algorithms rely on a large amount of 3D annotated data to train the models. Data acquisition and annotation are time-consuming and costly, and are difficult to adapt to the updates and iterations of vehicle models or mission scenarios in real life.

[0080] Next, a BEV 3D target detection model fine-tuning device based on 2D supervision proposed in accordance with an embodiment of the present application is described with reference to the accompanying drawings.

[0081] Figure 4 It is a structural schematic diagram of a BEV 3D target detection model fine-tuning device based on 2D supervision according to an embodiment of the present application.

[0082] like Figure 4 As shown, the BEV 3D object detection model fine-tuning device 10 based on 2D supervision includes: a processing module 100, a projection module 200 and a fine-tuning module 300.

[0083] The processing module 100 is used to input the target multi-view image into a pre-trained 3D target detection model and a pre-trained 2D target detection model to obtain a 3D detection box prediction output result and a 2D detection box prediction output result respectively.

[0084] The projection module 200 is used to obtain the 2D projection detection frame corresponding to the 3D detection frame prediction output result based on the 3D detection frame prediction output result, the 2D detection frame prediction output result and the preset detection frame projection strategy, and match the 2D projection detection frame with the preset 2D detection frame true value to obtain a matching result.

[0085] The fine-tuning module 300 is used to optimize the target loss function of the pre-trained 3D target detection model using the matching results and preset 2D annotation data to generate a 3D target detection fine-tuning model supervised by 2D annotation data.

[0086] Optionally, in one embodiment of the present application, the BEV 3D object detection model fine-tuning device 10 based on 2D supervision further includes: a judgment module, an output module, a matching module and a generation module.

[0087] Among them, the judgment module is used to fine-tune the pre-trained 2D target detection model according to the 2D annotation data after fine-tuning the target loss function using the matching results and the preset 2D annotation data, and to determine whether the fine-tuned pre-trained 2D target detection model meets the preset detection performance requirements.

[0088] The output module is used to input the preset unlabeled multi-view images into the pre-trained 3D target detection model and the pre-trained 2D target detection model respectively when the fine-tuned pre-trained 2D target detection model meets the preset detection performance requirements, so as to output the second 3D detection box prediction output result and the second 2D detection box prediction output result.

[0089] The matching module is used to obtain the second 2D projection detection frame corresponding to the second 3D detection frame prediction output result, and match the second 2D projection detection frame with the second 2D detection frame prediction output result to obtain an unlabeled matching result.

[0090] The generation module is used to optimize the target loss function based on the unlabeled matching results and generate a 3D target detection fine-tuning model corresponding to the unlabeled multi-view images.

[0091] Optionally, in one embodiment of the present application, the projection module 200 includes:

[0092] The first conversion unit is used to convert the 3D detection frame prediction output result in the BEV space coordinate system into the image coordinate system of each camera to obtain the 3D detection frame in the image coordinate system.

[0093] The second conversion unit is used to convert the 3D detection frame in the image coordinate system into a 2D detection frame to obtain a 2D projected detection frame corresponding to the 3D detection frame prediction output result.

[0094] The matching unit is used to match the 2D projection detection frame with the preset 2D detection frame true value based on the image coordinate system to obtain a matching result.

[0095] Optionally, in one embodiment of the present application, the 3D detection frame prediction output result in the BEV space coordinate system is converted to the image coordinate system of each camera, and the expression formula of the 3D detection frame in the image coordinate system can be:

[0096] P img =K·T·P BEV

[0097] Among them, K is the camera internal parameter matrix, T is the camera external parameter matrix, P img and P BEV They are the coordinates of the 3D detection box corner points in the image coordinate system and the BEV space coordinate system respectively.

[0098] It should be noted that the aforementioned explanation of the embodiment of the BEV 3D target detection model fine-tuning method based on 2D supervision is also applicable to the BEV 3D target detection model fine-tuning device based on 2D supervision in this embodiment, and will not be repeated here.

[0099] According to the BEV 3D target detection model fine-tuning device based on 2D supervision proposed in the embodiment of the present application, the pre-trained 3D target detection model and the pre-trained 2D target detection model can be used to synchronously fine-tune the pre-trained 2D target detection model and the pre-trained 3D target detection model based on 2D annotation data to obtain a 2D target detection fine-tuning model and a 3D target detection fine-tuning model, and the 2D target detection fine-tuning model can be used to generate 2D pseudo labels to achieve automatic fine-tuning of the 3D target detection fine-tuning model. The fine-tuning framework based on 2D supervision solves the problems of difficulty and high cost in obtaining 3D annotation data during fine-tuning of actual vehicle deployment, reduces dependence on large-scale annotation data, reduces fine-tuning costs, and accelerates the deployment of the BEV 3D target detection model and subsequent iterative optimization. At the same time, the 3D detection frame projection is independent of the BEV 3D target detection model, and the 2D detection frame is quickly obtained by simple coordinate transformation. The pluggable feature enables the fine-tuning framework of the present application to be quickly applied to a variety of 3D detection models and other BEV perception models. This solves the problem that the autonomous driving target detection algorithms in related technologies have low robustness in complex scenarios, and that some target detection algorithms cannot take into account cross-domain generalization capabilities while improving accuracy. In addition, most target detection algorithms rely on a large amount of 3D annotated data to train the models. Data acquisition and annotation are time-consuming and costly, and are difficult to adapt to the updates and iterations of vehicle models or mission scenarios in real life.

[0100] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0101] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0102] When the processor 502 executes the program, the BEV 3D object detection model fine-tuning method based on 2D supervision provided in the above embodiment is implemented.

[0103] Furthermore, the electronic device further comprises:

[0104] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0105] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0106] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0107] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0108] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0109] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0110] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned BEV 3D target detection model fine-tuning method based on 2D supervision.

[0111] An embodiment of the present application also provides a computer program product, including a computer program, which can run computer instructions. When the computer instructions are executed by a processor, the BEV3D target detection model fine-tuning method based on 2D supervision provided in the embodiment of the present application is implemented.

[0112] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0113] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0114] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0115] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or N wirings (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary and then storing it in a computer memory.

[0116] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0117] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0118] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0119] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A BEV 3D object detection model fine-tuning method based on 2D supervision, characterized in that: The following steps are involved: Input the target multi-view image into the pre-trained 3D target detection model and the pre-trained 2D target detection model to obtain the 3D detection box prediction output result and the 2D detection box prediction output result respectively; Based on the 3D detection box prediction output result, the 2D detection box prediction output result and a preset detection box projection strategy, obtaining a 2D projection detection box corresponding to the 3D detection box prediction output result, and matching the 2D projection detection box with a preset 2D detection box true value to obtain a matching result; The target loss function of the pre-trained 3D object detection model is optimized using the matching result and the preset 2D annotation data to generate a 3D object detection fine-tuning model supervised by the 2D annotation data.

2. The method according to claim 1, characterized in that After fine-tuning the target loss function using the matching result and the preset 2D annotation data, the method further includes: Fine-tune the pre-trained 2D object detection model according to the 2D annotated data, and determine whether the fine-tuned pre-trained 2D object detection model meets the preset detection performance requirements; When the fine-tuned pre-trained 2D object detection model meets the preset detection performance requirement, inputting the preset unlabeled multi-view image into the pre-trained 3D object detection model and the pre-trained 2D object detection model respectively to output a second 3D detection box prediction output result and a second 2D detection box prediction output result; Obtaining a second 2D projection detection frame corresponding to the second 3D detection frame prediction output result, and matching the second 2D projection detection frame with the second 2D detection frame prediction output result to obtain an unlabeled matching result; Based on the unlabeled matching result, the target loss function is optimized to generate a 3D target detection fine-tuning model corresponding to the unlabeled multi-view image.

3. The method according to claim 1, characterized in that The step of obtaining a 2D projection detection frame corresponding to the 3D detection frame prediction output result based on the 3D detection frame prediction output result, the 2D detection frame prediction output result and a preset detection frame projection strategy, and matching the 2D projection detection frame with a preset 2D detection frame true value to obtain a matching result includes: The 3D detection frame prediction output result in the BEV space coordinate system is converted into the image coordinate system of each camera to obtain the 3D detection frame in the image coordinate system; Convert the 3D detection frame in the image coordinate system into a 2D detection frame to obtain a 2D projection detection frame corresponding to the 3D detection frame prediction output result; Based on the image coordinate system, the 2D projection detection frame is matched with a preset 2D detection frame true value to obtain the matching result.

4. The method according to claim 3, characterized in that The 3D detection frame prediction output result in the BEV space coordinate system is converted to the image coordinate system of each camera, and the expression formula of the 3D detection frame in the image coordinate system is obtained as follows: P img =K·T·P BEV Among them, K is the camera internal parameter matrix, T is the camera external parameter matrix, P img and P BEV They are the coordinates of the 3D detection box corner points in the image coordinate system and the BEV space coordinate system respectively.

5. A BEV 3D object detection model fine-tuning device based on 2D supervision, characterized in that: include: A processing module, used to input the target multi-view image into a pre-trained 3D target detection model and a pre-trained 2D target detection model, and obtain a 3D detection box prediction output result and a 2D detection box prediction output result respectively; a projection module, configured to obtain a 2D projection detection frame corresponding to the 3D detection frame prediction output result based on the 3D detection frame prediction output result, the 2D detection frame prediction output result and a preset detection frame projection strategy, and match the 2D projection detection frame with a preset 2D detection frame true value to obtain a matching result; A fine-tuning module is used to optimize the target loss function of the pre-trained 3D object detection model using the matching results and preset 2D annotation data to generate a 3D object detection fine-tuning model supervised by the 2D annotation data.

6. The device according to claim 5, characterized in that Also includes: A judgment module, configured to, after fine-tuning the target loss function using the matching result and the preset 2D annotation data, fine-tune the pre-trained 2D target detection model according to the 2D annotation data, and judge whether the fine-tuned pre-trained 2D target detection model meets the preset detection performance requirement; An output module, configured to input the preset unlabeled multi-view image into the pre-trained 3D object detection model and the pre-trained 2D object detection model respectively, to output a second 3D detection box prediction output result and a second 2D detection box prediction output result, when the fine-tuned pre-trained 2D object detection model meets the preset detection performance requirement; A matching module, used to obtain a second 2D projection detection frame corresponding to the second 3D detection frame prediction output result, and match the second 2D projection detection frame with the second 2D detection frame prediction output result to obtain an unlabeled matching result; A generation module is used to optimize the target loss function based on the unlabeled matching result and generate a 3D target detection fine-tuning model corresponding to the unlabeled multi-view image.

7. The device according to claim 6, characterized in that The projection module comprises: A first conversion unit, configured to perform coordinate conversion on the 3D detection frame prediction output result in the BEV space coordinate system to the image coordinate system of each camera to obtain the 3D detection frame in the image coordinate system; A second conversion unit, configured to convert the 3D detection frame in the image coordinate system into a 2D detection frame to obtain a 2D projection detection frame corresponding to the 3D detection frame prediction output result; A matching unit is used to match the 2D projection detection frame with a preset 2D detection frame true value based on the image coordinate system to obtain the matching result.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the BEV 3D target detection model fine-tuning method based on 2D supervision as described in any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the BEV 3D target detection model fine-tuning method based on 2D supervision as described in any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed, it is used to implement the BEV 3D target detection model fine-tuning method based on 2D supervision as described in any one of claims 1-4.

Images