Method and device for training three-dimensional object detection model based on cross-modal knowledge distillation

Abstract

An embodiment of the present invention relates to a method and apparatus for training a three-dimensional object detection model based on cross-modal knowledge distillation. The method includes: obtaining a teacher model, a student model, and a student model loss function; obtaining training data from a training data set; determining feature loss, attention loss, and similarity loss functions for knowledge distillation from the teacher model to the student model; and forming an overall loss function from the student model loss function, feature loss function, attention loss function, and similarity loss function; performing model self-training on the student model; performing teacher-student feature imitation training; performing teacher-student attention imitation training; performing teacher-student similarity training; and performing overall training. Upon completion of training, a new set of training data is selected from the training data set to train the student model for the next round until a specified number of times is reached. The present invention can improve the detection accuracy of an image 3D object detection model.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for training a three-dimensional object detection model based on cross-modal knowledge distillation. Background Art

[0002] The perception module of an autonomous driving system is used to detect objects based on sensor data. Currently, mainstream sensors include cameras, lidar, and radar. Common 3D object detection models fall into two main categories: neural network models that use point clouds as input and neural network models that use images as input. Both models have their own advantages and disadvantages: 1) Point clouds inherently contain quasi-range (depth) information, allowing point cloud-based 3D object detection models to achieve high detection accuracy for relatively close objects. However, due to the sparsity of point clouds, point cloud-based 3D object detection models exhibit poor detection accuracy for distant objects. 2) Image-based 3D object detection models achieve high recognition accuracy for both near and distant objects due to the uniform distribution and high information density of field of view information on images. However, images lack depth information, requiring depth estimation by the model. This depth estimation can result in significant detection errors, resulting in insufficient localization accuracy of the bounding box (bbox) output by image-based 3D object detection models. Summary of the Invention

[0003] The purpose of the present invention is to address the defects of the existing technology and provide a method, device, electronic device and computer-readable storage medium for training a three-dimensional target detection model based on cross-modal knowledge distillation; a maturely trained point cloud-based 3D target detection model is used as a teacher model (Teacher Model), and an image-based 3D target detection model is used as a student model (Student Model), and the point cloud BEV features of the teacher model are distilled to the student model using the knowledge distillation mechanism, and the student model is trained to help the student model learn depth features similar to the point cloud data from the image data; through the present invention, the image 3D target detection model can be helped to reduce the depth estimation error, thereby achieving the purpose of improving the detection accuracy of the image 3D target detection model.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present invention provides a method for training a three-dimensional object detection model based on cross-modal knowledge distillation, the method comprising:

[0005] Obtain a mature point cloud-based 3D object detection model as the corresponding teacher model, obtain a to-be-trained image-based 3D object detection model as the corresponding student model; and obtain the model loss function of the student model as the corresponding student model loss function L det ;

[0006] Obtaining an original point cloud and an original image of the same scene from a training data set as corresponding first point clouds and first images; and obtaining recognition frame annotation information and false positive region annotation information of the original point clouds as corresponding first recognition frame sets and first false positive region sets;

[0007] The feature loss function L for performing knowledge distillation from the teacher model to the student model fea , attention loss function L att And the similarity loss function L aff Determine; and by the student model loss function L det , the feature loss function L fea , the attention loss function L att And the similarity loss function L aff Add together the corresponding overall loss function L all ,L all =L det +L fea +L att +L aff ;

[0008] According to the first image, the first recognition box set and the student model loss function L det The student model is self-trained; after the model self-training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the feature loss function L are used to calculate the model self-training. fea The student model is trained to imitate the teacher-student features; after the teacher-student feature imitation training is completed, the first point cloud, the first image and the attention loss function L are used to imitate the teacher-student features. att The student model is trained to imitate the teacher-student attention; after the teacher-student attention imitation training is completed, the first point cloud, the first image and the similarity loss function L are used to simulate the teacher-student attention. aff The student model is trained for teacher-student similarity; after the teacher-student similarity training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the overall loss function L are used. all Performing overall training on the student model;

[0009] After the overall training is completed, a new set of original point clouds, original images, recognition box annotation information of the original point clouds, and false positive area annotation information are selected from the training data set to perform the next round of training on the student model until the total number of training rounds reaches the specified number.

[0010] Preferably, the teacher model includes a point cloud column feature network, a BEV pooling network, a first BEV encoder and a first target detection head network; the output of the point cloud column feature network is connected to the input of the BEV pooling network; the output of the BEV pooling network is connected to the input of the first BEV encoder; the input of the first BEV encoder is connected to the input of the first target detection head network;

[0011] The student model includes an image encoder, an LSS view converter, a second BEV encoder, and a second object detection head network; the output of the image encoder is connected to the input of the LSS view converter; the output of the LSS view converter is connected to the input of the second BEV encoder; the input of the second BEV encoder is connected to the input of the second object detection head network;

[0012] The first identification box set includes a plurality of three-dimensional first identification boxes bbox1; the shape of the first identification box bbox1 is H bbox1 *W bbox1 *Z bbox1 , H bbox1 、W bbox1 , Z bbox1 The depth, width and height of the first identification box bbox1;

[0013] The first false positive region set includes a plurality of first false positive regions FP.

[0014] Preferably, the feature loss function L for performing knowledge distillation from the teacher model to the student model is fea , attention loss function L att And the similarity loss function L aff To determine, including:

[0015] Step 31, determine the feature loss function L fea for:

[0016] ;

[0017] in,

[0018] α, β, and γ are the preset loss coefficients;

[0019] F Tis the teacher feature map output by the first BEV encoder of the teacher model, H, W, C are the teacher feature map F T The height, width and channel dimensions of the teacher feature map F T It can be decomposed into H*W one-dimensional teacher feature channel vectors of shape 1×C The teacher characteristic graph F T It can also be decomposed into C two-dimensional teacher sub-feature maps of shape H×W The teacher characteristic graph F T It can also be decomposed into C*H*W teacher characteristic data 1≤k≤C, 1≤i≤H, 1≤j≤W;

[0020] F S f is the student feature map output by the second BEV encoder of the student model; proj () is the projection function from the student model BEV space to the teacher model BEV space; The student characteristic graph F S The corresponding projection feature map, The projected feature map With the teacher characteristic graph F T The height, width and channel dimensions of the projection feature map are consistent; It can be decomposed into H*W one-dimensional projection feature channel vectors of shape 1×C The projected feature map It can also be decomposed into C two-dimensional projection sub-feature maps of shape H×W The projected feature map It can also be decomposed into C*H*W projection feature data

[0021] M() is the foreground-background binary mask function;

[0022] N() is the false positive-background binary mask function;

[0023] S() is the size mask function;

[0024] A s () is the channel vector attention function;

[0025] A c () is the sub-feature map attention function;

[0026] The teacher feature map F T With the student characteristic graph F S The foreground feature loss branch, The teacher feature map F TWith the student characteristic graph F S The false positive feature loss branch, The teacher feature map F T With the student characteristic graph F S Background feature loss branch;

[0027] Step 32: Determine the attention loss function L att for:

[0028]

[0029] in,

[0030] η is the preset attention loss hyperparameter;

[0031] L1 is the L1_Loss loss function;

[0032] Step 33, from the teacher feature map F T and the projected feature map Optional preset quantity Q 2 Pixel points form corresponding matching point pairs 1≤i′≤Q, 1≤j′≤Q;

[0033] in,

[0034] Each pixel In the teacher characteristic graph F T The above corresponds to a row feature tensor with a shape of 1×W×C It also corresponds to a column feature tensor of shape H×1×C Each pixel In the projected feature map The above corresponds to a row feature tensor with a shape of 1×W×C It also corresponds to a column feature tensor of shape H×1×C

[0035] Step 34: according to the preset number Q 2 The matching point pair Determine the similarity loss function L aff for:

[0036]

[0037] in,

[0038] ζ is the preset similarity loss hyperparameter;

[0039] ‖‖ smoothl1 is the Smooth_L1_loss loss function;

[0040] Aff () is the similarity function.

[0041] Furthermore, the foreground-background binary mask function M() is:

[0042]

[0043] The false positive-background binary mask function N() is:

[0044]

[0045] The size mask function S() is:

[0046]

[0047] in,

[0048] H bbox1 、W bbox1 are the depth and width of the corresponding first identification box bbox1;

[0049] N gb is the teacher sub-feature graph The number of background points on

[0050] The channel vector attention function A s ()for:

[0051]

[0052] in,

[0053] T is the preset distillation hyperparameter;

[0054] softmax s [] is the activation function of the channel vector attention function;

[0055] F 1 is the input feature, the input feature F 1 Includes C feature components In the input feature F 1 is the teacher feature channel vector When the teacher feature channel vector The C components are the corresponding characteristic components In the input feature F 1 is the projected feature channel vector When the projected feature channel vector The C components are the corresponding characteristic components

[0056] The sub-feature map attention function A c ()for:

[0057]

[0058] in,

[0059] softmax c [] is the activation function of the sub-feature map attention function;

[0060] F 2 is the input feature, the input feature F 2 Includes H*W feature components In the input feature F 2 The teacher sub-feature graph When the teacher sub-feature map The H*W components are the corresponding characteristic components In the input feature F 2 is the projection sub-feature map When the projection sub-feature map The H*W components are the corresponding characteristic components

[0061] Furthermore, the similarity function A ff ()for:

[0062]

[0063] Among them, D i′ 、D j′ is the input feature vector;

[0064] In the input feature vector D i′ 、D j′ The teacher feature map F T The previous pixel The row feature tensor of and the column feature tensor When , the similarity function Specifically:

[0065] In the input feature vector D i′ 、D j′ is the projection feature map The previous pixel The row feature tensor of and the column feature tensor Time similarity function Specifically:

[0066] Preferably, the first image, the first recognition frame set and the student model loss function L det Performing model self-training on the student model, specifically including:

[0067] Step 61: Input the first image into the student model for step-by-step calculation, and in the calculation process, use the target recognition frame set output by the second target detection head network of the student model as the corresponding second recognition frame set; the second recognition frame set includes multiple three-dimensional second recognition frames bbox2; the shape of the second recognition frame bbox2 is H bbox2 *W bbox2 *Z bbox2 , H bbox2 、W bbox2 , Z bbox2 The depth, width and height of the second identification box bbox2;

[0068] Step 62: Substitute the first and second recognition frame sets into the student model loss function L det Performing loss value estimation to generate a corresponding first loss value;

[0069] Step 63: Identify whether the first loss value satisfies a preset first loss convergence range; if so, go to step 65; if not, go to step 64;

[0070] Step 64: Substitute the model parameters of the student model into the student model loss function L det Constructing a corresponding first objective function; solving the model parameters that minimize the first objective function; updating the model parameters of the student model according to the solution results; and returning to step 61 after the update;

[0071] Step 65: Confirm that the model self-training is completed.

[0072] Preferably, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the feature loss function L fea The student model is trained to imitate the teacher-student characteristics, specifically including:

[0073] Step 71: input the first point cloud into the teacher model for step-by-step calculation, and extract the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F during the calculation process. T; and input the first image into the student model for step-by-step operation, and in the operation process, extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F S ; and based on the projection function f proj () for the student characteristic graph F S Projection from the student model BEV space to the teacher model BEV space generates the corresponding projection feature map

[0074] Step 72: the teacher feature map F T , the projection feature map The first recognition frame set and the first false positive area set are substituted into the feature loss function L fea Performing a loss value estimation to generate a corresponding second loss value;

[0075] Step 73: Identify whether the second loss value satisfies a preset second loss convergence range; if so, go to step 75; if not, go to step 74;

[0076] Step 74: Substitute the model parameters of the student model into the feature loss function L fea Constructing a corresponding second objective function; solving the model parameters that minimize the second objective function; updating the model parameters of the student model according to the solution results; and returning to step 71 after the update;

[0077] Step 75: confirm that the teacher-student feature imitation training is completed.

[0078] Preferably, the first point cloud, the first image and the attention loss function L att The student model is trained to imitate teacher-student attention, specifically including:

[0079] Step 81: input the first point cloud into the teacher model for step-by-step calculation, and extract the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F during the calculation process. T ; and input the first image into the student model for step-by-step operation, and in the operation process, extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F S ; and based on the projection function f proj () for the student characteristic graph F S Projection from the student model BEV space to the teacher model BEV space generates the corresponding projection feature map

[0080] Step 82: the teacher feature graph F T and the projected feature map Substitute the attention loss function L att Performing loss value estimation to generate a corresponding third loss value;

[0081] Step 83, identifying whether the third loss value satisfies a preset third loss convergence range; if so, proceeding to step 85; if not, proceeding to step 84;

[0082] Step 84: Substitute the model parameters of the student model into the attention loss function L att Constructing a corresponding third objective function; solving the model parameters that minimize the third objective function; and updating the model parameters of the student model according to the solution results; and returning to step 81 after the update;

[0083] Step 85: confirm that the teacher-student attention imitation training is completed.

[0084] Preferably, the first point cloud, the first image and the similarity loss function L aff The student model is trained for teacher-student similarity, specifically including:

[0085] Step 91: input the first point cloud into the teacher model for step-by-step calculation, and extract the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F during the calculation process. T ; and input the first image into the student model for step-by-step operation, and in the operation process, extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F S ; and based on the projection function f proj () for the student characteristic graph F S Projection from the student model BEV space to the teacher model BEV space generates the corresponding projection feature map

[0086] Step 92: the teacher feature graph F T and the projected feature map Substitute the similarity loss function L aff Performing a loss value estimation to generate a corresponding fourth loss value;

[0087] Step 93: Identify whether the fourth loss value satisfies a preset fourth loss convergence range; if so, go to step 95; if not, go to step 94;

[0088] Step 94: Substitute the model parameters of the student model into the similarity loss function L aff Constructing a corresponding fourth objective function; solving the model parameters that minimize the fourth objective function; and updating the model parameters of the student model according to the solution results; and returning to step 91 after the update;

[0089] Step 95: confirm that the teacher-student similarity training is completed.

[0090] Preferably, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the overall loss function L all The student model is trained as a whole, specifically including:

[0091] Step 101: input the first point cloud into the teacher model for step-by-step calculation, and extract the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F during the calculation process. T ; and input the first image into the student model for step-by-step operation, and in the operation process, extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F S , and the target recognition frame set output by the second target detection head network of the student model is used as the corresponding third recognition frame set; and based on the projection function f proj () for the student characteristic graph F S Projection from the student model BEV space to the teacher model BEV space generates the corresponding projection feature map The third identification frame set includes a plurality of three-dimensional third identification frames bbox3; the shape of the third identification frame bbox3 is H bbox3 *W bbox3 *Z bbox3 , H bbox3 、W bbox3 , Z bbox3 The depth, width and height of the third identification box bbox3;

[0092] Step 102: combine the first and third recognition frame sets and the teacher feature map F T , the projection feature map and the first false positive region set are substituted into the overall loss function L all Perform loss value estimation to generate the corresponding overall loss value;

[0093] Step 103, identifying whether the overall loss value satisfies a preset overall loss convergence range; if so, proceeding to step 105; if not, proceeding to step 104;

[0094] Step 104: Substitute the model parameters of the student model into the overall loss function L all Constructing a corresponding overall objective function; solving the model parameters that minimize the overall objective function; updating the model parameters of the student model according to the solution results; and returning to step 101 after the update;

[0095] Step 105: confirm that the overall training is completed.

[0096] A second aspect of an embodiment of the present invention provides a device for implementing the method for training a three-dimensional object detection model based on cross-modal knowledge distillation described in the first aspect, the device comprising: an acquisition module, a training data processing module, a loss function processing module, and a training processing module;

[0097] The acquisition module is used to acquire a mature point cloud-based 3D target detection model as the corresponding teacher model, and acquire a to-be-trained image-based 3D target detection model as the corresponding student model; and acquire the model loss function of the student model as the corresponding student model loss function L det ;

[0098] The training data processing module is used to obtain the original point cloud and original image of the same scene from the training data set as the corresponding first point cloud and first image; and obtain the recognition frame annotation information and false positive area annotation information of the original point cloud as the corresponding first recognition frame set and first false positive area set;

[0099] The loss function processing module is used to perform knowledge distillation from the teacher model to the student model using the feature loss function L fea , attention loss function L att And the similarity loss function L aff Determine; and by the student model loss function L det , the feature loss function L fea , the attention loss function L att And the similarity loss function L aff Add together the corresponding overall loss function L all ,L all =L det +L fea +L att +L aff ;

[0100] The training processing module is used to calculate the loss function L of the student model based on the first image, the first recognition frame set and the student model loss function L detThe student model is self-trained; after the model self-training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the feature loss function L are used to calculate the model self-training. fea The student model is trained to imitate the teacher-student features; after the teacher-student feature imitation training is completed, the first point cloud, the first image and the attention loss function L are used to imitate the teacher-student features. att The student model is trained to imitate the teacher-student attention; after the teacher-student attention imitation training is completed, the first point cloud, the first image and the similarity loss function L are used to simulate the teacher-student attention. aff The student model is trained for teacher-student similarity; after the teacher-student similarity training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the overall loss function L are used. all The student model is trained as a whole; after the overall training is completed, a new set of original point clouds, original images, recognition box annotation information of the original point clouds, and false positive area annotation information are selected from the training data set to perform the next round of training on the student model until the total number of training rounds reaches a specified number.

[0101] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0102] The processor is configured to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0103] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0104] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0105] An embodiment of the present invention provides a method, device, electronic device and computer-readable storage medium for training a three-dimensional target detection model based on cross-modal knowledge distillation; a maturely trained point cloud-based 3D target detection model is used as a teacher model, and an image-based 3D target detection model is used as a student model. The knowledge distillation mechanism is used to distill the point cloud BEV features of the teacher model to the student model and train the student model, so as to help the student model learn depth features similar to the point cloud data from the image data; through the present invention, the image 3D target detection model can be helped to reduce the depth estimation error and improve the detection accuracy of the image 3D target detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0106] Figure 1 A schematic diagram of a method for training a three-dimensional object detection model based on cross-modal knowledge distillation provided in Example 1 of the present invention;

[0107] Figure 2 A module structure diagram of a device for training a three-dimensional object detection model based on cross-modal knowledge distillation, provided in Example 2 of the present invention;

[0108] Figure 3 This is a structural diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0109] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described herein are merely some, rather than all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0110] The first embodiment of the present invention provides a method for training a three-dimensional object detection model based on cross-modal knowledge distillation, such as Figure 1 As shown in the schematic diagram of a method for training a three-dimensional object detection model based on cross-modal knowledge distillation provided in the first embodiment of the present invention, the method mainly includes the following steps:

[0111] Step 1: Obtain a mature point cloud-based 3D object detection model as the corresponding teacher model, obtain the image-based 3D object detection model to be trained as the corresponding student model; and obtain the model loss function of the student model as the corresponding student model loss function L det ;

[0112] The teacher model includes a point cloud column feature network, a BEV pooling network, a first BEV encoder, and a first object detection head network; the output of the point cloud column feature network is connected to the input of the BEV pooling network; the output of the BEV pooling network is connected to the input of the first BEV encoder; the input of the first BEV encoder is connected to the input of the first object detection head network;

[0113] The student model includes an image encoder, an LSS view converter, a second BEV encoder, and a second object detection head network; the output of the image encoder is connected to the input of the LSS view converter; the output of the LSS view converter is connected to the input of the second BEV encoder; and the input of the second BEV encoder is connected to the input of the second object detection head network.

[0114] Here, the teacher model of the embodiment of the present invention is a mature point cloud-based 3D target detection model; the point cloud column feature network of the teacher model is similar to the Pillar Feature Net of the PointPillars model; the BEV pooling network of the teacher model performs height pooling processing on the output of the point cloud column feature network to obtain a BEV feature map under a bird's eye view (BEV); the first BEV encoder of the teacher model further encodes information on the BEV feature map and outputs a corresponding BEV thermal feature map; the first target detection head network of the teacher model performs BEV target detection based on the BEV thermal feature map to obtain multiple two-dimensional target recognition frames, and performs 3D shape regression calculation on each two-dimensional target recognition frame through the internal fully connected network to output multiple three-dimensional target recognition frames;

[0115] The student model of the embodiment of the present invention is an image-based 3D target detection model to be trained; the image encoder of the student model extracts features from the input image; the LSS (Lift Splat Shoot, LSS) view transformer of the student model is similar to the view transformer in the technical paper "Lift, splat, shoot: Encoding images from arbitrary camerarigs by implicitly unprojecting to 3d", which can extract BEV features from image data and output corresponding BEV feature maps; the second BEV encoder of the student model further encodes information on the BEV feature map and outputs the corresponding BEV thermal feature map; the second target detection head network of the student model performs BEV target detection based on the BEV thermal feature map to obtain multiple two-dimensional target recognition frames, and performs 3D shape regression calculation on each two-dimensional target recognition frame through the internal fully connected network to output multiple three-dimensional target recognition frames. It should be noted that the second target detection head network of the student model of the embodiment of the present invention can also be other neural network structures that can output BEV feature maps and BEV thermal feature maps; the model loss function of the student model of the embodiment of the present invention is a known loss function denoted as the student model loss function L det .

[0116] Step 2: Obtain the original point cloud and original image of the same scene from the training data set as the corresponding first point cloud and first image; and obtain the recognition box annotation information and false positive area annotation information of the original point cloud as the corresponding first recognition box set and first false positive area set;

[0117] The first identification frame set includes multiple three-dimensional first identification frames bbox1; the shape of the first identification frame bbox1 is H bbox1 *W bbox1 *Z bbox1 , H bbox1 、W bbox1 , Z bbox1 The depth, width and height of the first identification box bbox1;

[0118] The first false positive region set includes a plurality of first false positive regions FP.

[0119] Here, the training data set of the present invention is used to store multiple training data records; each training data record corresponds to a set of original point clouds, original images, recognition frame annotation information and false positive area annotation information of the original point clouds under the same scene; the original point clouds and the original images have substantially the same field of view and are both generated at the same time; the recognition frame annotation information and false positive area annotation information of the original point clouds are manually annotated information, and the original point clouds and the recognition frame annotation information and false positive area annotation information of the original point clouds are also one of the training data of the teacher model. The false positive (FP) area mentioned here is an area in the original point cloud space that is not occupied by the target recognition frame but has a point cloud of a physical object. Correspondingly, the area in the original point cloud space that is occupied by the recognition frame is the foreground area, and the area in the original point cloud space that is neither a foreground area nor a false positive area is called the background area.

[0120] Step 3: Feature loss function L for knowledge distillation from teacher model to student model fea , attention loss function L att And the similarity loss function L aff Determine; and by the student model loss function L det , feature loss function L fea , attention loss function L att And the similarity loss function L aff Add together the corresponding overall loss function L all ;

[0121] Among them, L all =L det +L fea +L att +L aff ;

[0122] Specifically including: Step 31, the feature loss function L of the knowledge distillation from the teacher model to the student model fea , attention loss function L att And the similarity loss function L aff to confirm;

[0123] Specifically including: Step 311, determining the feature loss function L fea for:

[0124]

[0125] in,

[0126] 1) α, β, and γ are the preset loss coefficients;

[0127] 2)F T is the teacher feature map output by the first BEV encoder of the teacher model, H, W, C are the teacher feature map F T The height, width and channel dimensions of the teacher feature map F T It can be decomposed into H*W one-dimensional teacher feature channel vectors of shape 1×C Teacher characteristic graph F T It can also be decomposed into C two-dimensional teacher sub-feature maps of shape H×W Teacher characteristic graph F T It can also be decomposed into C*H*W teacher characteristic data 1≤k≤C, 1≤i≤H, 1≤j≤W;

[0128] 3)F S is the student feature map output by the second BEV encoder of the student model; f proj () is the projection function from the student model BEV space to the teacher model BEV space; The student characteristic graph F S The corresponding projection feature map, Projected feature map and teacher characteristic graph F T The height, width and channel dimensions of the projection feature map are consistent; It can be decomposed into H*W one-dimensional projection feature channel vectors of shape 1×C Projected feature map It can also be decomposed into C two-dimensional projection sub-feature maps of shape H×W Projected feature map It can also be decomposed into C*H*W projection feature data

[0129] 4) M() is the foreground-background binary mask function:

[0130]

[0131] Here, (i, j) is the teacher sub-feature graph The foreground-background binary mask function specifies that as long as the coordinate is in the foreground area occupied by any first identification box bbox1, the corresponding mask output is 1. Conversely, if the coordinate is not in any foreground area, the corresponding mask output is 0.

[0132] 5) N() is the false positive-background binary mask function:

[0133]

[0134] Here, (i, j) is the teacher sub-feature graph The false positive-background binary mask function specifies that as long as the coordinate is in any first false positive region FP, the corresponding mask output is 1; otherwise, if the coordinate is not in any false positive region, the corresponding mask output is 0.

[0135] 6) S() is the size mask function:

[0136]

[0137] in,

[0138] H bbox1 、W bbox1 is the depth and width of the corresponding first identification box bbox1;

[0139] N gb Teacher sub-characteristic map The number of background points on

[0140]

[0141] Here, (i, j) is the teacher sub-feature graph The pixel coordinates on the image are given by the size mask function. As long as the coordinates are in the foreground area occupied by any first identification box bbox1, the corresponding size mask output is the depth H of the corresponding first identification box bbox1. bbox1 and width W bbox1 If the coordinate is not in any foreground area, the corresponding size mask output is the number of background points N. gb The reciprocal of

[0142] 7)A s () is the channel vector attention function:

[0143]

[0144] in,

[0145] T is the preset distillation hyperparameter;

[0146] softmaxs [] is the activation function of the channel vector attention function;

[0147] F 1 is the input feature, input feature F 1 Includes C feature components

[0148] In the input feature F 1 is the teacher feature channel vector When the teacher feature channel vector The C components of The channel vector attention function at this time is:

[0149]

[0150] In the input feature F 1 is the projected feature channel vector When the feature channel vector is projected The C components of The channel vector attention function at this time is:

[0151]

[0152] 8)A c () is the sub-feature map attention function:

[0153]

[0154] in,

[0155] softmax c [] is the activation function of the sub-feature map attention function;

[0156] F 2 is the input feature, input feature F 2 Includes H*W feature components

[0157] In the input feature F 2 Teacher sub-characteristic map When the teacher sub-feature map The H*W components of The sub-feature graph attention function at this time is:

[0158]

[0159] In the input feature F 2 is the projection sub-feature map When the sub-feature map is projected The H*W components of The sub-feature graph attention function at this time is:

[0160]

[0161] Here, it is not difficult to see from the above description that the feature loss function L fea in is the teacher feature map F T and student characteristic graph F S The foreground feature loss branch, is the teacher feature map F T and student characteristic graph F S The false positive feature loss branch, is the teacher feature map F T and student characteristic graph F S Background feature loss branch; through the feature loss function L fea It can help students improve their models’ positioning accuracy;

[0162] Step 312: Determine the attention loss function L att for:

[0163]

[0164] Among them, η is the preset attention loss hyperparameter; L1 is the L1_Loss loss function;

[0165] Here, the attention loss function L att In fact, the channel vector attention function and the sub-feature map attention function are used to calculate the teacher feature map F. T and student characteristic graph F S Compare the attention features of att It can help student models improve target recognition accuracy;

[0166] Step 313: From the teacher feature graph F T and projected feature maps Optional preset quantity Q 2 Pixel points form corresponding matching point pairs 1≤i′≤Q, 1≤j′≤Q;

[0167] Among them, each pixel In the teacher characteristic graph F T The above corresponds to a row feature tensor with a shape of 1×W×C It also corresponds to a column feature tensor of shape H×1×C Each pixel In the projected feature map The above corresponds to a row feature tensor with a shape of 1×W×C It also corresponds to a column feature tensor of shape H×1×C

[0168] Here, from the teacher feature map F T and projected feature maps Select the preset quantity Q 2 Pixel points form a matching point pair And each matching point pair The two pixels in the teacher feature map F T and projected feature maps The pixel coordinates on the same;

[0169] For example, Q = 2, then from the teacher feature map F T Select four points with coordinates (1,2), (1,3), (1,4), and (1,5) as the corresponding Then we should also start from the projected feature map Select four points with coordinates (1,2), (1,3), (1,4), and (1,5) as the corresponding Then, according to the corresponding relationship of (i′, j′), we get 4 matching point pairs:

[0170] Step 314: According to the preset number Q 2 Matching point pairs Determine the similarity loss function L aff for:

[0171]

[0172] in,

[0173] 1)ζ is the preset similarity loss hyperparameter;

[0174] 2)‖‖ smoothl1 is the Smooth_L1_loss loss function;

[0175] 3)A ff () is the similarity function:

[0176]

[0177] in,

[0178] D i′ 、D j′ is the input feature vector;

[0179] In the input feature vector D i′ 、D j′is the teacher feature map F T Previous pixel The row feature tensor of and column feature tensors When , the similarity function Specifically:

[0180] In the input feature vector D i′ 、D j′ is the projected feature map Previous pixel The row feature tensor of and column feature tensors Time similarity function Specifically:

[0181] Here, the similarity function used in the embodiment of the present invention is the cosine similarity function; through the similarity loss function L aff Can help student models improve model performance;

[0182] Step 32, and the student model loss function L det , feature loss function L fea , attention loss function L att And the similarity loss function L aff Add together the corresponding overall loss function L all ;

[0183] Among them, L all =L det +L fea +L att +L aff ;

[0184] Step 4: Based on the first image, the first recognition frame set and the student model loss function L det The student model is self-trained; after the model self-training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the feature loss function L are used to calculate the student model. fea The student model is trained to imitate the teacher-student features. After the teacher-student feature imitation training is completed, the first point cloud, the first image and the attention loss function L are used. att The student model is trained to imitate the teacher-student attention. After the teacher-student attention imitation training is completed, the first point cloud, the first image and the similarity loss function L are used. aff The student model is trained for teacher-student similarity; after the teacher-student similarity training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the overall loss function L are used. all Conduct overall training on the student model;

[0185] The embodiment of the present invention utilizes the knowledge distillation mechanism to distill the point cloud BEV features of the teacher model to the student model and train the student model. During the training process, the student model is firstly trained by itself; then, based on the three loss functions for knowledge distillation (feature loss function L fea , attention loss function L att And the similarity loss function L aff ) to train the student model step by step; finally, based on the overall loss function L all Conduct overall training on the student model;

[0186] Specifically comprising: step 41, according to the first image, the first recognition frame set and the student model loss function L det Perform model self-training on the student model;

[0187] Specifically, the steps include: step 411, inputting the first image into the student model for step-by-step operation, and using the target recognition frame set output by the second target detection head network of the student model as the corresponding second recognition frame set during the operation;

[0188] The second identification frame set includes multiple three-dimensional second identification frames bbox2; the shape of the second identification frame bbox2 is H bbox2 *W bbox2 *Z bbox2 , H bbox2 、W bbox2 , Z bbox2 The depth, width and height of the second identification box bbox2;

[0189] Step 412: Substitute the first and second recognition box sets into the student model loss function L det Performing loss value estimation to generate a corresponding first loss value;

[0190] Step 413: Identify whether the first loss value satisfies a preset first loss convergence range; if so, go to step 415; if not, go to step 414;

[0191] Step 414: Substitute the model parameters of the student model into the student model loss function L det Constructing a corresponding first objective function; solving the model parameters that minimize the first objective function; and updating the model parameters of the student model according to the solution results; and returning to step 411 after the update;

[0192] Step 415, confirming that the model self-training is complete;

[0193] Step 42: After the model self-training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the feature loss function L are used to calculate the model self-training.fea Train the student model to imitate the characteristics of the teacher and the student;

[0194] Specifically, step 421 includes: inputting the first point cloud into the teacher model for step-by-step calculation, and extracting the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F T ; Input the first image into the student model for step-by-step operation, and extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F during the operation S ; and based on the projection function f proj () for student characteristic graph F S Project the student model BEV space to the teacher model BEV space to generate the corresponding projection feature map

[0195] Step 422: The teacher feature graph F T , projection feature map The first recognition box set and the first false positive area set are substituted into the feature loss function L fea Performing a loss value estimation to generate a corresponding second loss value;

[0196] Step 423, identifying whether the second loss value satisfies a preset second loss convergence range; if so, proceeding to step 425; if not, proceeding to step 424;

[0197] Step 424: Substitute the model parameters of the student model into the feature loss function L fea Constructing a corresponding second objective function; solving the model parameters that minimize the second objective function; and updating the model parameters of the student model according to the solution results; and returning to step 421 after the update;

[0198] Step 425, confirming that the teacher-student feature imitation training is completed;

[0199] Step 43: After the teacher-student feature imitation training is completed, the first point cloud, the first image and the attention loss function L att Conduct teacher-student attention imitation training on student models;

[0200] Specifically, the method includes: step 431, inputting the first point cloud into the teacher model for step-by-step operation, and extracting the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map FT during the operation; and inputting the first image into the student model for step-by-step operation, and extracting the output feature map of the second BEV encoder of the student model as the corresponding student feature map F during the operation. S ; and based on the projection function f proj () for student characteristic graph F SProject the student model BEV space to the teacher model BEV space to generate the corresponding projection feature map

[0201] Step 432: The teacher feature graph F T and projected feature maps Substitute the attention loss function L att Performing loss value estimation to generate a corresponding third loss value;

[0202] Step 433: Identify whether the third loss value satisfies a preset third loss convergence range; if so, go to step 435; if not, go to step 434;

[0203] Step 434: Substitute the model parameters of the student model into the attention loss function L att Constructing a corresponding third objective function; solving the model parameters that minimize the third objective function; and updating the model parameters of the student model according to the solution results; and returning to step 431 after the update;

[0204] Step 435, confirming that the teacher-student attention imitation training is completed;

[0205] Step 44: After the teacher-student attention imitation training is completed, the first point cloud, the first image and the similarity loss function L aff Train the student model on teacher-student similarity;

[0206] Specifically, step 441 includes: inputting the first point cloud into the teacher model for step-by-step calculation, and extracting the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F T ; Input the first image into the student model for step-by-step operation, and extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F during the operation S ; and based on the projection function f proj () for student characteristic graph F S Project the student model BEV space to the teacher model BEV space to generate the corresponding projection feature map

[0207] Step 442: The teacher feature graph F T and projected feature maps Substitute the similarity loss function L aff Performing a loss value estimation to generate a corresponding fourth loss value;

[0208] Step 443: Identify whether the fourth loss value satisfies a preset fourth loss convergence range; if so, go to step 445; if not, go to step 444;

[0209] Step 444: Substitute the model parameters of the student model into the similarity loss function L aff Constructing a corresponding fourth objective function; solving the model parameters that minimize the fourth objective function; and updating the model parameters of the student model according to the solution results; and returning to step 441 after the update;

[0210] Step 445, confirming that the teacher-student similarity training is completed;

[0211] Step 45: After the teacher-student similarity training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the overall loss function L are used to calculate the similarity between the teacher and the student. all Conduct overall training on the student model;

[0212] Specifically, step 451 includes: inputting the first point cloud into the teacher model for step-by-step calculation, and extracting the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F T ; Input the first image into the student model for step-by-step operation, and extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F during the operation S , and the target recognition frame set output by the second target detection head network of the student model is used as the corresponding third recognition frame set; and based on the projection function f proj () for student characteristic graph F S Project the student model BEV space to the teacher model BEV space to generate the corresponding projection feature map The third identification frame set includes multiple three-dimensional third identification frames bbox3; the shape of the third identification frame bbox3 is H bbox3 *W bbox3 *Z bbox3 , H bbox3 、W bbox3 , Z bbox3 The depth, width and height of the third identification box bbox3;

[0213] Step 452: combine the first and third recognition frame sets and the teacher feature map F T , projection feature map and the first false positive region set into the overall loss function L all Perform loss value estimation to generate the corresponding overall loss value;

[0214] Step 453: Identify whether the overall loss value meets the preset overall loss convergence range; if so, go to step 455; if not, go to step 454;

[0215] Step 454: Substitute the model parameters of the student model into the overall loss function Lall Constructing a corresponding overall objective function; solving the model parameters that minimize the overall objective function; and updating the model parameters of the student model according to the solution results; and returning to step 451 after the update;

[0216] Step 455: Confirm that the overall training is completed.

[0217] In step 5, after the overall training is completed, a new set of original point clouds, original images, recognition box annotation information of the original point clouds, and false positive area annotation information are selected from the training data set to perform the next round of training on the student model until the total number of training rounds reaches the specified number.

[0218] Figure 2 The module structure diagram of a device for training a three-dimensional object detection model based on cross-modal knowledge distillation provided in the second embodiment of the present invention is a terminal device or server that implements the aforementioned method embodiment, and can also be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: an acquisition module 201, a training data processing module 202, a loss function processing module 203 and a training processing module 204.

[0219] The acquisition module 201 is used to obtain a mature point cloud-based 3D object detection model as the corresponding teacher model, obtain a to-be-trained image-based 3D object detection model as the corresponding student model; and obtain the model loss function of the student model as the corresponding student model loss function L det .

[0220] The training data processing module 202 is used to obtain the original point cloud and original image of the same scene from the training data set as the corresponding first point cloud and first image; and obtain the recognition box annotation information and false positive area annotation information of the original point cloud as the corresponding first recognition box set and first false positive area set.

[0221] The loss function processing module 203 is used to perform the feature loss function L on the knowledge distillation from the teacher model to the student model. fea , attention loss function L att And the similarity loss function L aff Determine; and by the student model loss function L det , feature loss function L fea , attention loss function L att And the similarity loss function L aff Add together the corresponding overall loss function L all ,L all =L det +Lfea +L att +L aff .

[0222] The training processing module 204 is used to train the first image, the first recognition frame set and the student model loss function L det The student model is self-trained; after the model self-training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the feature loss function L are used to calculate the student model. fea The student model is trained to imitate the teacher-student features. After the teacher-student feature imitation training is completed, the first point cloud, the first image and the attention loss function L are used. att The student model is trained to imitate the teacher-student attention. After the teacher-student attention imitation training is completed, the first point cloud, the first image and the similarity loss function L are used. aff The student model is trained for teacher-student similarity; after the teacher-student similarity training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the overall loss function L are used. all The student model is trained as a whole. After the overall training is completed, a new set of original point clouds, original images, recognition box annotation information of the original point clouds, and false positive area annotation information are selected from the training data set to train the student model for the next round, until the total number of training rounds reaches the specified number.

[0223] An embodiment of the present invention provides a device for training a three-dimensional target detection model based on cross-modal knowledge distillation, which can execute the method steps in the above method embodiment. Its implementation principles and technical effects are similar and will not be repeated here.

[0224] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by a processing element; or they can all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the acquisition module can be a separately established processing element, or it can be integrated into a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called by a processing element of the above device to perform the functions of the above determination module. The implementation of other modules is similar. In addition, these modules can all or partly be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0225] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0226] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the above method embodiments are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) means. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, hard disk, tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0227] Figure 3 This is a schematic diagram of the structure of an electronic device provided in the third embodiment of the present invention. The electronic device may be the aforementioned terminal device or server, or may be a terminal device or server connected to the aforementioned terminal device or server to implement the method of the embodiment of the present invention. Figure 3As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303's transceiver actions. Various instructions may be stored in the memory 302 for completing various processing functions and implementing the processing steps described in the aforementioned method embodiment. Preferably, the electronic device involved in the embodiment of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The above-mentioned communication port 306 is used for connection and communication between the electronic device and other peripherals.

[0228] exist Figure 3 The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used to realize communication between the database access device and other devices (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory (Non-Volatile Memory), such as at least one disk storage.

[0229] The above-mentioned processors can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0230] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the methods and processing procedures provided in the above embodiments.

[0231] An embodiment of the present invention further provides a chip for executing instructions, which is used to execute the processing steps described in the above method embodiment.

[0232] An embodiment of the present invention provides a method, device, electronic device and computer-readable storage medium for training a three-dimensional target detection model based on cross-modal knowledge distillation; a maturely trained point cloud-based 3D target detection model is used as a teacher model, and an image-based 3D target detection model is used as a student model. The knowledge distillation mechanism is used to distill the point cloud BEV features of the teacher model to the student model and train the student model, so as to help the student model learn depth features similar to the point cloud data from the image data; through the present invention, the image 3D target detection model can be helped to reduce the depth estimation error and improve the detection accuracy of the image 3D target detection model.

[0233] Professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0234] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0235] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training a 3D object detection model based on cross-modal knowledge distillation, characterized in that: The method comprises: Obtain a mature point cloud-based 3D object detection model as the corresponding teacher model, obtain a to-be-trained image-based 3D object detection model as the corresponding student model; and obtain the model loss function of the student model as the corresponding student model loss function L det ; Obtaining an original point cloud and an original image of the same scene from a training data set as corresponding first point clouds and first images; and obtaining recognition frame annotation information and false positive region annotation information of the original point clouds as corresponding first recognition frame sets and first false positive region sets; The feature loss function L for performing knowledge distillation from the teacher model to the student model fea , attention loss function L att And the similarity loss function L aff Determine; and by the student model loss function L det , the feature loss function L fea , the attention loss function L att And the similarity loss function L aff Add together the corresponding overall loss function L all ,L all =L det +L fea +L att +L aff ; According to the first image, the first recognition box set and the student model loss function L det The student model is self-trained; after the model self-training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the feature loss function L are used to calculate the model self-training. fea The student model is trained to imitate the teacher-student features; after the teacher-student feature imitation training is completed, the first point cloud, the first image and the attention loss function L are used to imitate the teacher-student features. att The student model is trained to imitate the teacher-student attention; after the teacher-student attention imitation training is completed, the first point cloud, the first image and the similarity loss function L are used to simulate the teacher-student attention. aff The student model is trained for teacher-student similarity; after the teacher-student similarity training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the overall loss function L are used. all Performing overall training on the student model; After the overall training is completed, a new set of original point clouds, original images, recognition box annotation information of the original point clouds, and false positive area annotation information are selected from the training data set to perform the next round of training on the student model until the total number of training rounds reaches the specified number.

2. The method for training a three-dimensional object detection model based on cross-modal knowledge distillation according to claim 1, characterized in that: The teacher model includes a point cloud column feature network, a BEV pooling network, a first BEV encoder and a first object detection head network; the output of the point cloud column feature network is connected to the input of the BEV pooling network; the output of the BEV pooling network is connected to the input of the first BEV encoder; the input of the first BEV encoder is connected to the input of the first object detection head network; The student model includes an image encoder, an LSS view converter, a second BEV encoder, and a second object detection head network; the output of the image encoder is connected to the input of the LSS view converter; the output of the LSS view converter is connected to the input of the second BEV encoder; the input of the second BEV encoder is connected to the input of the second object detection head network; The first identification box set includes a plurality of three-dimensional first identification boxes bbox1; the shape of the first identification box bbox1 is H bbox1 *W bbox1 *Z bbox1 , H bbox1 、W bbox1 , Z bbox1 The depth, width and height of the first identification box bbox1; The first false positive region set includes a plurality of first false positive regions FP.

3. The method for training a three-dimensional object detection model based on cross-modal knowledge distillation according to claim 2, characterized in that: The feature loss function L for performing knowledge distillation from the teacher model to the student model fea , attention loss function L att And the similarity loss function L aff To determine, including: Step 31, determine the feature loss function L fea for: ； in, α, β, and γ are the preset loss coefficients; F T is the teacher feature map output by the first BEV encoder of the teacher model, H, W, C are the teacher feature map F T The height, width and channel dimensions of the teacher feature map F T It can be decomposed into H*W one-dimensional teacher feature channel vectors of shape 1×C The teacher characteristic graph F T It can also be decomposed into C two-dimensional teacher sub-feature maps of shape H×W The teacher characteristic graph F T It can also be decomposed into C*H*W teacher characteristic data F S f is the student feature map output by the second BEV encoder of the student model; proj () is the projection function from the student model BEV space to the teacher model BEV space; The student characteristic graph F S The corresponding projection feature map, The projected feature map With the teacher characteristic graph F T The height, width and channel dimensions of the projection feature map are consistent; It can be decomposed into H*W one-dimensional projection feature channel vectors of shape 1×C The projected feature map It can also be decomposed into C two-dimensional projection sub-feature maps of shape H×W The projected feature map It can also be decomposed into C*H*W projection feature data M() is the foreground-background binary mask function; N() is the false positive-background binary mask function; S() is the size mask function; A s () is the channel vector attention function; A c () is the sub-feature map attention function; The teacher feature map F T With the student characteristic graph F S The foreground feature loss branch, The teacher feature map F T With the student characteristic graph F S The false positive feature loss branch, The teacher feature map F T With the student characteristic graph F S Background feature loss branch; Step 32: Determine the attention loss function L att for: in, η is the preset attention loss hyperparameter; L1 is the L1_Loss loss function; Step 33, from the teacher feature map F T and the projected feature map Optional preset quantity Q 2 Pixel points form corresponding matching point pairs in, Each pixel In the teacher characteristic graph F T The above corresponds to a row feature tensor with a shape of 1×W×C It also corresponds to a column feature tensor of shape H×1×C Each pixel In the projected feature map The above corresponds to a row feature tensor with a shape of 1×W×C It also corresponds to a column feature tensor of shape H×1×C Step 34: according to the preset number Q 2 The matching point pair Determine the similarity loss function L aff for: in, ζ is the preset similarity loss hyperparameter; ‖ ‖ smoothl1 is the Smooth_L1_loss loss function; A ff () is the similarity function.

4. The method for training a three-dimensional object detection model based on cross-modal knowledge distillation according to claim 3, characterized in that: The foreground-background binary mask function M() is: The false positive-background binary mask function N() is: The size mask function S() is: in, H bbox1 、W bbox1 are the depth and width of the corresponding first identification box bbox1; N gb is the teacher sub-feature graph The number of background points on The channel vector attention function A s ()for: in, T is the preset distillation hyperparameter; softmax s [] is the activation function of the channel vector attention function; F 1 is the input feature, the input feature F 1 Includes C feature components In the input feature F 1 is the teacher feature channel vector When the teacher feature channel vector The C components are the corresponding characteristic components In the input feature F 1 is the projected feature channel vector When the projected feature channel vector The C components are the corresponding characteristic components The sub-feature map attention function A c ()for: in, softmax c [] is the activation function of the sub-feature map attention function; F 2 is the input feature, the input feature F 2 Includes H*W feature components In the input feature F 2 is the teacher sub-feature graph When the teacher sub-feature map The H*W components are the corresponding characteristic components In the input feature F 2 is the projection sub-feature map When the projection sub-feature map The H*W components are the corresponding characteristic components 5. The method for training a three-dimensional object detection model based on cross-modal knowledge distillation according to claim 3, characterized in that: The similarity function A ff ()for: Among them, D i′ 、D j′ is the input feature vector; In the input feature vector D i′ 、D j′ The teacher feature map F T The previous pixel The row feature tensor of and the column feature tensor When , the similarity function Specifically: In the input feature vector D i′ 、D j′ is the projection feature map The previous pixel The row feature tensor of and the column feature tensor Time similarity function Specifically:

6. The method for training a three-dimensional object detection model based on cross-modal knowledge distillation according to claim 3, characterized in that: The first image, the first recognition frame set and the student model loss function L det Performing model self-training on the student model, specifically including: Step 61: Input the first image into the student model for step-by-step calculation, and in the calculation process, use the target recognition frame set output by the second target detection head network of the student model as the corresponding second recognition frame set; the second recognition frame set includes multiple three-dimensional second recognition frames bbox2; the shape of the second recognition frame bbox2 is H bbox2 *W bbox2 *Z bbox2 , H bbox2 、W bbox2 , Z bbox2 The depth, width and height of the second identification box bbox2; Step 62: Substitute the first and second recognition frame sets into the student model loss function L det Performing loss value estimation to generate a corresponding first loss value; Step 63: Identify whether the first loss value satisfies a preset first loss convergence range; if so, go to step 65; if not, go to step 64; Step 64: Substitute the model parameters of the student model into the student model loss function L det Constructing a corresponding first objective function; solving the model parameters that minimize the first objective function; updating the model parameters of the student model according to the solution results; and returning to step 61 after the update; Step 65: Confirm that the model self-training is completed.

7. The method for training a three-dimensional object detection model based on cross-modal knowledge distillation according to claim 3, characterized in that: The method according to the first point cloud, the first image, the first recognition frame set, the first false positive area set and the feature loss function L fea The student model is trained to imitate the teacher-student characteristics, specifically including: Step 71: input the first point cloud into the teacher model for step-by-step calculation, and extract the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F during the calculation process. T ; and input the first image into the student model for step-by-step operation, and in the operation process, extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F S ; and based on the projection function f proj () for the student characteristic graph F S Projection from the student model BEV space to the teacher model BEV space generates the corresponding projection feature map Step 72: the teacher feature map F T , the projection feature map The first recognition frame set and the first false positive area set are substituted into the feature loss function L fea Performing a loss value estimation to generate a corresponding second loss value; Step 73: Identify whether the second loss value satisfies a preset second loss convergence range; if so, go to step 75; if not, go to step 74; Step 74: Substitute the model parameters of the student model into the feature loss function L fea Constructing a corresponding second objective function; solving the model parameters that minimize the second objective function; updating the model parameters of the student model according to the solution results; and returning to step 71 after the update; Step 75: confirm that the teacher-student feature imitation training is completed.

8. The method for training a three-dimensional object detection model based on cross-modal knowledge distillation according to claim 3, characterized in that: The first point cloud, the first image and the attention loss function L att The student model is trained to imitate teacher-student attention, specifically including: Step 81: input the first point cloud into the teacher model for step-by-step calculation, and extract the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F during the calculation process. T ; and input the first image into the student model for step-by-step operation, and in the operation process, extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F S ; and based on the projection function f proj () for the student characteristic graph F S Projection from the student model BEV space to the teacher model BEV space generates the corresponding projection feature map Step 82: the teacher feature graph F T and the projected feature map Substitute the attention loss function L att Performing loss value estimation to generate a corresponding third loss value; Step 83, identifying whether the third loss value satisfies a preset third loss convergence range; if so, proceeding to step 85; if not, proceeding to step 84; Step 84: Substitute the model parameters of the student model into the attention loss function L att Constructing a corresponding third objective function; solving the model parameters that minimize the third objective function; and updating the model parameters of the student model according to the solution results; and returning to step 81 after the update; Step 85: confirm that the teacher-student attention imitation training is completed.

9. The method for training a three-dimensional object detection model based on cross-modal knowledge distillation according to claim 3, characterized in that: The first point cloud, the first image and the similarity loss function L aff The student model is trained for teacher-student similarity, specifically including: Step 91: input the first point cloud into the teacher model for step-by-step calculation, and extract the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F during the calculation process. T ; and input the first image into the student model for step-by-step operation, and in the operation process, extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F S ; and based on the projection function f proj () for the student characteristic graph F S Projection from the student model BEV space to the teacher model BEV space generates the corresponding projection feature map Step 92: the teacher feature graph F T and the projected feature map Substitute the similarity loss function L aff Performing a loss value estimation to generate a corresponding fourth loss value; Step 93: Identify whether the fourth loss value satisfies a preset fourth loss convergence range; if so, go to step 95; if not, go to step 94; Step 94: Substitute the model parameters of the student model into the similarity loss function L aff Constructing a corresponding fourth objective function; solving the model parameters that minimize the fourth objective function; and updating the model parameters of the student model according to the solution results; and returning to step 91 after the update; Step 95: confirm that the teacher-student similarity training is completed.

10. The method for training a three-dimensional object detection model based on cross-modal knowledge distillation according to claim 3, characterized in that: The method according to the first point cloud, the first image, the first recognition frame set, the first false positive area set and the overall loss function L all The student model is trained as a whole, specifically including: Step 101: input the first point cloud into the teacher model for step-by-step calculation, and extract the output feature map of the first BEV encoder of the teacher model as the corresponding teacher feature map F during the calculation process. T ; and input the first image into the student model for step-by-step operation, and in the operation process, extract the output feature map of the second BEV encoder of the student model as the corresponding student feature map F S , and the target recognition frame set output by the second target detection head network of the student model is used as the corresponding third recognition frame set; and based on the projection function f proj () for the student characteristic graph F S Projection from the student model BEV space to the teacher model BEV space generates the corresponding projection feature map The third identification frame set includes a plurality of three-dimensional third identification frames bbox3; the shape of the third identification frame bbox3 is H bbox3 *W bbox3 *Z bbox3 , H bbox3 、W bbox3 , Z bbox3 The depth, width and height of the third identification box bbox3; Step 102: combine the first and third recognition frame sets and the teacher feature map F T , the projection feature map and the first false positive region set are substituted into the overall loss function L all Perform loss value estimation to generate the corresponding overall loss value; Step 103, identifying whether the overall loss value satisfies a preset overall loss convergence range; if so, proceeding to step 105; if not, proceeding to step 104; Step 104: Substitute the model parameters of the student model into the overall loss function L all Constructing a corresponding overall objective function; solving the model parameters that minimize the overall objective function; updating the model parameters of the student model according to the solution results; and returning to step 101 after the update; Step 105: confirm that the overall training is completed.

11. A device for executing the method for training a three-dimensional object detection model based on cross-modal knowledge distillation according to any one of claims 1 to 10, characterized in that: The device includes: an acquisition module, a training data processing module, a loss function processing module and a training processing module; The acquisition module is used to acquire a mature point cloud-based 3D target detection model as the corresponding teacher model, and acquire a to-be-trained image-based 3D target detection model as the corresponding student model; and acquire the model loss function of the student model as the corresponding student model loss function L det ; The training data processing module is used to obtain the original point cloud and original image of the same scene from the training data set as the corresponding first point cloud and first image; and obtain the recognition frame annotation information and false positive area annotation information of the original point cloud as the corresponding first recognition frame set and first false positive area set; The loss function processing module is used to perform knowledge distillation from the teacher model to the student model using the feature loss function L fea , attention loss function L att And the similarity loss function L aff Determine; and by the student model loss function L det , the feature loss function L fea , the attention loss function L att And the similarity loss function L aff Add together the corresponding overall loss function L all ,L all =L det +L fea +L att +L aff ; The training processing module is used to calculate the loss function L of the student model based on the first image, the first recognition frame set and the student model loss function L det The student model is self-trained; after the model self-training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the feature loss function L are used to calculate the model self-training. fea The student model is trained to imitate the teacher-student features; after the teacher-student feature imitation training is completed, the first point cloud, the first image and the attention loss function L are used to imitate the teacher-student features. att The student model is trained to imitate the teacher-student attention; after the teacher-student attention imitation training is completed, the first point cloud, the first image and the similarity loss function L are used to simulate the teacher-student attention. aff The student model is trained for teacher-student similarity; after the teacher-student similarity training is completed, the first point cloud, the first image, the first recognition frame set, the first false positive area set and the overall loss function L are used. all The student model is trained as a whole; after the overall training is completed, a new set of original point clouds, original images, recognition box annotation information of the original point clouds, and false positive area annotation information are selected from the training data set to perform the next round of training on the student model until the total number of training rounds reaches a specified number.

12. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is configured to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in any one of claims 1 to 10; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed by a computer, enable the computer to execute the method according to any one of claims 1 to 10.

Images