A vehicle-road collaborative 3D target detection method and device based on knowledge distillation

Abstract

The present invention discloses a vehicle-road collaborative 3D target detection method and device based on knowledge distillation, which belongs to the field of autonomous driving perception. The method includes: obtaining a pre-trained teacher model; the input of the teacher model is multimodal data of the vehicle itself and multimodal data of other vehicles; the multimodal data includes multi-view two-dimensional image data and lidar point cloud data; the relevant modules in the teacher model that process the lidar point cloud data are removed, and the remaining modules are used to obtain a student model with a similar architecture to the teacher model; the input of the student model is multi-view two-dimensional image data; the knowledge distillation method is used to supervise the student model using the teacher model, and the trained student model is used to perform 3D target detection on the vehicle itself. The present invention can reduce computational complexity while maintaining high detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception technology, and in particular to a vehicle-road collaborative 3D target detection method and device based on knowledge distillation. Background Art

[0002] With the rapid development of artificial intelligence, autonomous driving has become a key development direction for the automotive industry, significantly impacting traffic safety, efficiency, and comfort. In the field of autonomous driving perception, 3D object detection technology is key to enabling vehicles to accurately perceive their surroundings. However, in complex traffic scenarios, the sensor's limited field of view can lead to occlusion of objects, compromising the accuracy and reliability of 3D object detection.

[0003] Related technologies use vehicle-road collaborative detection methods to address the problem of obscured targets. By exchanging information between vehicles and between vehicles and the road, they effectively address the limited perception of a single vehicle. However, implementing vehicle-road collaboration requires processing and integrating large amounts of data from multiple sensors and other vehicles, which not only places higher demands on computing power but also increases system complexity.

[0004] Therefore, there is an urgent need to provide a new vehicle-road collaborative 3D target detection method that can reduce the computational complexity while maintaining high detection accuracy. Summary of the Invention

[0005] This invention provides a method and device for vehicle-road collaborative 3D object detection based on knowledge distillation. The technical solution is as follows:

[0006] On the one hand, a vehicle-road collaborative 3D object detection method based on knowledge distillation is provided, the method comprising:

[0007] Obtain a pre-trained teacher model; the teacher model is used to perform multimodal fusion processing on the input self-vehicle multimodal data and the input other-vehicle multimodal data, and then collaboratively process the self-vehicle teacher bird's-eye view features obtained by the fusion processing with the bird's-eye view features of each other-vehicle teacher, so as to output a 3D target detection result for the self-vehicle based on the collaboratively processed teacher bird's-eye view collaborative features; the other-vehicle is another vehicle located in the same scene as the self-vehicle; the multimodal data includes multi-view two-dimensional image data and lidar point cloud data;

[0008] Relevant modules for processing lidar point cloud data in the teacher model are removed, and the remaining modules are used to obtain a student model with a similar architecture to the teacher model; the student model is used to perform single-modal feature extraction on the input multi-view two-dimensional image data, and then collaboratively process the extracted student bird's-eye view features of the own vehicle and the student bird's-eye view features of each other vehicle, so as to output a 3D target detection result for the own vehicle based on the collaboratively processed student bird's-eye view collaborative features;

[0009] The knowledge distillation method is used to supervise the student model using the teacher model, so as to perform 3D target detection on the vehicle using the trained student model.

[0010] On the other hand, a vehicle-road collaborative 3D object detection device based on knowledge distillation is provided, the device comprising:

[0011] an acquisition unit for acquiring a pre-trained teacher model; the teacher model is used to perform multimodal fusion processing on the input self-vehicle multimodal data and the input other-vehicle multimodal data, and then collaboratively process the self-vehicle teacher bird's-eye view perspective features obtained by the fusion processing and the bird's-eye view perspective features of each other-vehicle teacher, so as to output a 3D target detection result for the self-vehicle based on the collaboratively processed teacher bird's-eye view collaborative features; the other-vehicle is another vehicle located in the same scene as the self-vehicle; the multimodal data includes multi-view two-dimensional image data and lidar point cloud data;

[0012] a model construction unit, configured to remove modules related to processing the lidar point cloud data from the teacher model, and utilize the remaining modules to obtain a student model having a similar architecture to the teacher model; the student model is configured to perform single-modal feature extraction on the input multi-view two-dimensional image data, and then collaboratively process the extracted student bird's-eye view features of the own vehicle and the student bird's-eye view features of each other vehicle, so as to output a 3D target detection result for the own vehicle based on the collaboratively processed student bird's-eye view collaborative features;

[0013] The model training unit is used to utilize the knowledge distillation method to supervise the student model using the teacher model, so as to perform 3D target detection on the vehicle using the trained student model.

[0014] On the other hand, a computer device is provided, which includes a memory and a processor, the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory to implement the steps of the above-mentioned vehicle-road collaborative 3D target detection method based on knowledge distillation.

[0015] On the other hand, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned vehicle-road collaborative 3D target detection method based on knowledge distillation are implemented.

[0016] On the other hand, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned vehicle-road collaborative 3D target detection method based on knowledge distillation.

[0017] The technical solution provided by the present invention can at least bring the following beneficial effects:

[0018] The vehicle-road collaboration technology is combined with the knowledge distillation method. The vehicle-road collaboration technology is used to collect data from other vehicles in the same scene as the vehicle and the vehicle's data to jointly detect targets, thereby solving the problem of target occlusion. In addition, the knowledge distillation method is used to construct a teacher model and a student model with similar architectures. The teacher model is trained using multimodal data, so the teacher model has higher accuracy. Compared with the teacher model, the student model removes the relevant modules for processing point cloud data. Although only two-dimensional image data is used for training, the student model is supervised and trained by the teacher model with higher accuracy through the knowledge distillation method, so that the student model can be close to the teacher model in detection accuracy. Furthermore, the teacher model also adopts the vehicle-road collaboration method, which can have higher environmental perception ability when solving the problem of target occlusion. It can be seen that the student model supervised and trained by this scheme has a detection accuracy similar to that of the teacher model, and has a simple architecture, less data to be processed, lower computational complexity, and faster detection speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 This is a flow chart of a vehicle-road collaborative 3D object detection method based on knowledge distillation provided by one embodiment of the present invention;

[0021] Figure 2 This is a diagram of a teacher model architecture provided by one embodiment of the present invention;

[0022] Figure 3 This is an architecture diagram of a fusion processing module in a teacher model provided by one embodiment of the present invention;

[0023] Figure 4 This is a student model architecture diagram provided by one embodiment of the present invention;

[0024] Figure 5 This is an architecture diagram of a feature extraction module in a student model provided by one embodiment of the present invention;

[0025] Figure 6 This is a structural diagram of a vehicle-road collaborative 3D object detection device based on knowledge distillation provided by one embodiment of the present invention;

[0026] Figure 7 This is a hardware architecture diagram of a computer device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0028] Please refer to Figure 1 , an embodiment of the present invention provides a vehicle-road collaborative 3D target detection method based on knowledge distillation, the method comprising:

[0029] Step 100: Obtain a pre-trained teacher model; the teacher model is used to perform multimodal fusion processing on the input multimodal data of the self-vehicle and the multimodal data of other vehicles, and then collaboratively process the bird's-eye view perspective features of the self-vehicle teacher obtained by the fusion processing with the bird's-eye view perspective features of each other vehicle teacher, so as to output a 3D target detection result for the self-vehicle based on the collaboratively processed teacher bird's-eye view collaborative features; the other vehicles are other vehicles located in the same scene as the self-vehicle; the multimodal data includes multi-view 2D image data and LiDAR point cloud data;

[0030] Step 102: Removing modules related to processing the lidar point cloud data from the teacher model, and using the remaining modules to obtain a student model with a similar architecture to the teacher model; the student model is used to perform single-modal feature extraction on the input multi-view two-dimensional image data, and then collaboratively process the extracted student bird's-eye view features of the vehicle and the student bird's-eye view features of each other vehicle, so as to output a 3D object detection result for the vehicle based on the collaboratively processed student bird's-eye view collaborative features;

[0031] Step 104 , using a knowledge distillation method to supervise the student model using the teacher model, and then using the trained student model to perform 3D target detection on the vehicle.

[0032] In an embodiment of the present invention, the vehicle-road collaboration technology is combined with the knowledge distillation method. The vehicle-road collaboration technology is used to collect data from other vehicles in the same scene as the vehicle and the vehicle's data to jointly perform target detection, thereby solving the problem of target occlusion. In addition, the knowledge distillation method is used to construct a teacher model and a student model with similar architectures. The teacher model is trained using multimodal data, so the teacher model has higher accuracy. The student model removes the relevant modules for processing point cloud data compared to the teacher model. Although only two-dimensional image data is used for training, the student model is supervised and trained by the teacher model with higher accuracy through the knowledge distillation method, so that the student model can be close to the teacher model in detection accuracy. Furthermore, the teacher model also adopts the vehicle-road collaboration method, which can have higher environmental perception ability when solving the problem of target occlusion. It can be seen that the student model supervised and trained by this scheme has a detection accuracy similar to that of the teacher model, and has a simple architecture, less data to be processed, lower computational complexity, and faster detection speed.

[0033] Described below Figure 1 How to perform the steps shown.

[0034] Step 100 , step 102 , and step 104 are described simultaneously.

[0035] Knowledge distillation is a model compression technology that can transfer knowledge from large and complex models to small models, thereby significantly reducing the model's computational complexity and inference time while maintaining detection accuracy. Therefore, in an embodiment of the present invention, the constructed teacher model can be a model with a complex structure and more accurate knowledge to be learned, and the student model can be a model with a lighter structure. In addition, in order to solve the problem of limited perception ability of a single vehicle and obstruction of the detection target during 3D target detection, both the teacher model and the student model adopt vehicle-road collaborative technology.

[0036] In order to achieve vehicle-road collaboration, the teacher model in the embodiment of the present invention is used to perform multimodal fusion processing on the input self-vehicle multimodal data and the other-vehicle multimodal data respectively, and then collaboratively process the self-vehicle teacher bird's-eye view perspective features and each other-vehicle teacher bird's-eye view perspective features obtained by the fusion processing, so as to output a 3D target detection result for the self-vehicle based on the collaboratively processed teacher bird's-eye view collaborative features; the other vehicles are other vehicles located in the same scene as the self-vehicle; the multimodal data includes multi-perspective two-dimensional image data and lidar point cloud data.

[0037] In one implementation, please refer to the teacher model architecture diagram Figure 2 and Figure 3 The teacher model includes: a fusion processing module, a first collaborative processing module, and a first target detection module; the fusion processing module includes: a first backbone network, a second backbone network, a third backbone network, a feature fuser, and a first bird's-eye view feature extractor.

[0038] The multimodal fusion processing method includes: using the first backbone network to extract features from the lidar point cloud data to obtain three-dimensional point cloud features; using the second backbone network to extract features from the multi-view two-dimensional image data to obtain multi-view two-dimensional semantic features of the teacher; using the third backbone network to perform depth estimation on the multi-view two-dimensional semantic features of the teacher to obtain three-dimensional semantic features of the teacher; using the feature fusion module to fuse the three-dimensional point cloud features and the three-dimensional semantic features of the teacher to obtain three-dimensional mixed features; using the first bird's-eye view feature extraction module to project the three-dimensional mixed features into the bird's-eye view features of the teacher;

[0039] The first collaborative processing module collaboratively processes the bird's-eye view features of the self-driving teacher obtained by fusion processing and the bird's-eye view features of each other-driving teacher;

[0040] The first target detection module is used to output the 3D target detection results for the ego vehicle based on the collaboratively processed teacher bird's-eye view collaborative features. Specifically, the decoder network is used to upsample and refine the teacher bird's-eye view collaborative features to obtain preliminary target detection features, which are then refined and non-maximum suppressed by the target detection head to obtain the 3D target detection results.

[0041] The vehicle can be equipped with a laser radar and an onboard camera to acquire point cloud data and two-dimensional image data. There are multiple onboard cameras, and the multiple onboard cameras are spaced around the vehicle to obtain multi-view two-dimensional image data.

[0042] The first backbone network is used to extract features from point cloud data acquired by a laser radar installed on a vehicle. Therefore, in an embodiment of the present invention, the first backbone network may adopt a PointPillars network.

[0043] The second backbone network is used to extract semantic features from two-dimensional image data. Therefore, the first backbone network can use a ResNet network. The more layers a ResNet network has, the higher the accuracy and the greater the computational complexity. To ensure the detection accuracy of the teacher model, the first backbone network can preferably use a ResNet101 network.

[0044] The third backbone network is used to perform depth estimation on the two-dimensional semantic features of the multi-view teacher. In one embodiment of the present invention, the third backbone network may adopt a VIT network (‌Vision Transformer, an image classification network based on the Transformer architecture).

[0045] Both the feature fusion module and the first bird's-eye view feature extractor can be implemented based on convolutional networks. After fusing the 3D point cloud features with the teacher's 3D semantic features to obtain a 3D hybrid feature, it is projected into the teacher's bird's-eye view feature. Since the 3D hybrid feature contains both point cloud features extracted from point cloud data and features extracted from multi-view 2D image data, the teacher's bird's-eye view feature obtained from this feature-rich 3D hybrid feature can improve the detection accuracy of 3D targets when performing subsequent target detection.

[0046] Since 3D object detection results are based on the ego vehicle's perspective, when training the teacher model, the input data in the sample pairs must distinguish between ego vehicle data and other vehicle data. The output 3D object detection results are also based on the ego vehicle's perspective. The input data in the sample pairs includes multi-view 2D image data and LiDAR point cloud data. When obtaining the sample pairs used to train the teacher model, each vehicle uses its onboard camera and LiDAR to acquire ego vehicle data and sends this data to a server. After receiving the data from each vehicle, the server determines which vehicles are in the same scene and then labels the data for all vehicles in the same scene. The data for one vehicle in the same scene is labeled as ego vehicle data, while the data for all other vehicles is labeled as other vehicle data. This generates the sample input data. The 3D object detection results for the perspective corresponding to the vehicle serving as ego vehicle data are then obtained for the sample input data. This generates the sample output data, and ultimately, the sample pairs.

[0047] By pre-training the teacher model using a large number of samples, a pre-trained teacher model can be obtained for subsequent supervised training of the student model. This teacher model has very high detection accuracy and can accurately solve the problem of occlusion of the detection target.

[0048] It should be noted that how to determine which vehicles are located in the same scene based on the data of each vehicle belongs to the existing technology and will not be described in detail in this embodiment.

[0049] Since lidar point cloud data is three-dimensional data, the amount of computation required to process point cloud data is very large. Therefore, after the teacher model is trained, in order to obtain a lighter student model, the relevant modules in the teacher model that process lidar point cloud data can be removed, and the remaining modules can be used to obtain a student model with a similar architecture to the teacher model.

[0050] The teacher model's modules for processing LiDAR point cloud data include the first backbone network and the feature fusion unit. After removing these two modules, the remaining student model is used to extract single-modal features from the input multi-view 2D image data. The extracted student bird's-eye view features of the vehicle and the student bird's-eye view features of each other vehicle are then collaboratively processed to output 3D object detection results for the vehicle.

[0051] In one implementation, please refer to the student model architecture diagram Figure 4 and Figure 5 The student model includes: a feature extraction module, a second collaborative processing module and a second target detection module; the feature extraction module includes: a fourth backbone network, a fifth backbone network and a second bird's-eye view feature extractor.

[0052] The single-modal feature extraction method includes: using the fourth backbone network to extract features from multi-view two-dimensional image data to obtain multi-view student two-dimensional semantic features; using the fifth backbone network to perform depth estimation on the multi-view student two-dimensional semantic features to obtain student three-dimensional semantic features; using the second bird's-eye view feature extractor to project the student three-dimensional semantic features into student bird's-eye view features;

[0053] The second collaborative processing module is used to extract the feature of the student's own car and the bird's-eye view feature of each other car student;

[0054] The second target detection module is used to output the 3D target detection result for the vehicle based on the collaboratively processed student bird's-eye view collaborative features.

[0055] Similar to the second backbone network, the fourth backbone network can use a ResNet network. Since the teacher model will be used to supervise the training of the student model in the subsequent process, the ResNet network used in the fourth backbone network can have fewer layers than the ResNet network used in the second backbone network. Preferably, when the second backbone network uses the ResNet101 network, the fourth backbone network uses the ResNet18 network.

[0056] Similar to the third backbone network, the fifth backbone network can adopt the VIT network. Similar to the first bird's-eye view feature extractor, the second bird's-eye view feature extractor can be implemented based on a convolutional network.

[0057] After the student model is supervised and trained using the teacher model, the student model has the high detection accuracy of the teacher model and is faster.

[0058] In one embodiment of the present invention, when performing 3D target detection on the vehicle using the trained student model, the following steps may be specifically included:

[0059] Step 1040: Acquire the multi-view two-dimensional image data of the vehicle at the current moment; use the feature extraction module to extract features from the multi-view two-dimensional image data of the vehicle to obtain the bird's-eye view features of the vehicle at the current moment;

[0060] Step 1042: Obtain the bird's-eye view features of the other student in the car at the current moment sent by the external server;

[0061] Step 1044: using the second collaborative processing module to collaboratively process the bird's-eye view perspective features of the student in the own vehicle and the bird's-eye view perspective features of the student in the other vehicle at the current moment, to obtain the student bird's-eye view collaborative features at the current moment;

[0062] Step 1046: Use the second target detection module to perform target detection on the student bird's-eye view collaborative features at the current moment to obtain the 3D target detection result of the vehicle at the current moment.

[0063] In practical applications, the student model is applied to each vehicle. When performing 3D target detection, the vehicle itself can obtain multi-perspective 2D image data based on its own onboard camera. The multi-perspective 2D image data of the other vehicle needs to be forwarded by the server. The feature extraction module then performs feature extraction on the other vehicle's multi-perspective 2D image data to obtain the student bird's-eye view features of the other vehicle. Because the transmission of large amounts of data is time-consuming, in an embodiment of the present invention, the other vehicle data received by the vehicle from the external server can be the student bird's-eye view features of the other vehicle obtained after feature extraction using the feature extraction module in the student model. In this way, only the student bird's-eye view features of the other vehicle are transmitted, without the need to transmit a large amount of 2D image data. This improves data transmission speed. Furthermore, when performing 3D target detection in the vehicle itself, there is no need to process the other vehicle data using the feature extraction module, further improving the speed of 3D target detection in the vehicle itself.

[0064] In order to further improve the speed of 3D object detection in the self-driving car, the bird's-eye view features of other vehicles that are collaboratively processed in the teacher model and the student model are the effective bird's-eye view features of other vehicles after effective feature screening; the bird's-eye view features of other vehicles include the bird's-eye view features of the teacher of other vehicles and the bird's-eye view features of other vehicles; the effective bird's-eye view features of other vehicles include the effective bird's-eye view features of the teacher of other vehicles and the effective bird's-eye view features of other vehicles. Please refer to Figure 2 and Figure 4 ,The teacher model also includes a first feature screening module, and the student model also includes a second feature screening module.

[0065] In one implementation, the feature screening module may perform effective feature screening on the bird's-eye view feature by:

[0066] S1: Use the classification network to classify the bird's-eye view features of the other car to obtain the spatial confidence matrix of the other car;

[0067] S2: For each element in the spatial confidence matrix, assign a value of 1 to the element whose value is greater than a set value, and assign a value of 0 to the element whose value is not greater than the set value, to obtain a binary selection matrix;

[0068] S3 multiplies the binary selection matrix by the corresponding position of the other vehicle's bird's-eye view feature to obtain the other vehicle's effective bird's-eye view feature.

[0069] This method can be used to filter out effective features from the bird's-eye view features and eliminate invalid features. The effective bird's-eye view features of other vehicles obtained in this way not only carry effective information, but are also more conducive to the collaborative processing of subsequent features. The amount of data required to be transmitted is less than that of the bird's-eye view features of other vehicles, thereby improving data transmission efficiency.

[0070] When fusing the bird's-eye view features of the vehicle itself with the effective bird's-eye view features of other vehicles, feature fusion can be performed by taking the maximum value point by point, thereby obtaining the bird's-eye view collaborative features.

[0071] After the supervised training of the student model by the teacher model is completed by using the knowledge distillation method, the second feature screening module can also be set on each vehicle or on the server, thereby completing the screening process of the effective bird's-eye view features of the other vehicle student on the other vehicle or server, and the effective feature screening process of the other vehicle data is no longer performed on the own vehicle, thereby further improving the 3D target detection speed on the own vehicle.

[0072] In an embodiment of the present invention, in order to ensure that the trained student model has a high detection accuracy similar to that of the teacher model, the total loss in the supervised training process includes at least: the target detection loss between the 3D target detection result output by the student model and the true value, the distillation loss between the student model and the teacher model at the feature level, and the distillation loss between the student model and the teacher model at the prediction result level.

[0073] The three types of losses mentioned above are explained below.

[0074] 1. Object Detection Loss

[0075] The target detection loss is used to supervise the student model to produce more accurate 3D target detection results, which can include: classification loss and bounding box regression loss.

[0076] Classification loss The calculation method can be:

[0077]

[0078] in, is the classification result set, is the one-hot encoding of the true label, is the probability that the model predicts that the sample belongs to the classification result c.

[0079] The bounding box regression loss can be calculated as:

[0080]

[0081] in, are the predicted bounding box parameters, is the true bounding box parameter, is the intersection-over-union (IoU) of the predicted bounding box and the true bounding box.

[0082] Then, the target detection loss It can be:

[0083]

[0084] in, 、 is a hyperparameter.

[0085] 2. Feature-level distillation loss

[0086] The feature-level distillation loss can include: the mean squared error distillation loss between the student bird's-eye view features in the student model and the teacher bird's-eye view features in the teacher model, and the mean squared error distillation loss between the student bird's-eye view collaborative features in the student model and the teacher bird's-eye view collaborative features in the teacher model.

[0087] The mean squared error distillation loss of the bird's-eye view feature is used to supervise the alignment of the student bird's-eye view features extracted by the student model to the teacher bird's-eye view features extracted by the teacher model, and the mean squared error distillation loss of the bird's-eye view feature is used to The calculation method can be:

[0088]

[0089] in, Provide students with a bird's-eye view of the characteristics of perspective, Provides a bird's-eye view of features for teachers.

[0090] The mean squared error distillation loss of the bird's-eye view collaborative features is used to supervise the alignment of the student bird's-eye view collaborative features in the student model to the teacher bird's-eye view collaborative features in the teacher model. ... The calculation method can be:

[0091]

[0092] in, Provide students with a bird's-eye view of collaborative features, Provide teachers with a bird's-eye view of collaborative features.

[0093] Then, the distillation loss at the feature level It can be:

[0094]

[0095] in, 、 is a hyperparameter.

[0096] 3. Distillation loss at the prediction result level

[0097] The distillation loss at the prediction result level may include: the normalized distillation loss between the semantic category probability prediction results output by the student model and the semantic category probability prediction results output by the teacher model, and the intersection-over-union distillation loss between the target detection anchor box position prediction results output by the student model and the target detection anchor box position prediction results output by the teacher model; wherein the semantic category probability prediction results and the target detection anchor box position prediction results are both located in the 3D target detection results.

[0098] Normalized distillation loss of semantic category probability prediction results, used to supervise the alignment of student model semantic category probability prediction results to teacher model semantic category probability prediction results, normalized distillation loss of semantic category probability prediction results The calculation method can be:

[0099]

[0100] in, is the semantic category probability prediction result of the student model, This is the semantic category probability prediction result of the teacher model.

[0101] The intersection-over-union distillation loss of the target detection anchor box position prediction result is used to supervise the alignment of the target detection anchor box position prediction result of the student model to the target detection anchor box position prediction result of the teacher model. The calculation method can be:

[0102]

[0103] in, The target detection anchor box position prediction result of the student model, The target detection anchor box position prediction results of the teacher model.

[0104] Then, the distillation loss at the prediction result level It can be:

[0105]

[0106] in, 、 is a hyperparameter.

[0107] By using the total loss function obtained by the above loss to supervise the training of the student model, a student model with a detection accuracy close to that of the teacher model can be obtained.

[0108] To further reduce the gap in detection accuracy between the student model and the teacher model, bringing the student model's detection accuracy closer to that of the teacher model, the total loss function includes at least a distillation loss for semantic category relevance and / or a distillation loss for volume rendering. These two types of losses are described below.

[0109] 4. Distillation loss of semantic category correlation

[0110] The distillation loss for semantic category relevance is calculated as follows:

[0111] First, the multi-view original images corresponding to the multi-view two-dimensional image data in the training samples are classified and labeled pixel by pixel, and the mapping relationship between each pixel in the multi-view original image and each tensor in the bird's-eye view feature in the model is determined. The semantic feature tensor corresponding to each category is calculated based on the mapping relationship; among them, the mapping relationship can be determined by the internal and external parameters of the camera.

[0112] In the embodiment of the present invention, the category q Corresponding semantic feature tensor Calculated by the following formula:

[0113]

[0114] in, Belong to category q The total number of pixels, N is the total number of pixels in the multi-view original image, Pixel The category label, For pixels Tensor in the bird's-eye view feature with mapping relationship, Indicates that only q Pixels Participation semantic feature tensor Calculation.

[0115] Then, for each pixel point, the normalized dot product is used to calculate the semantic category similarity between the tensor in the bird's-eye view feature with a mapping relationship for the pixel point and the semantic feature tensor of each category.

[0116] In the embodiment of the present invention, the pixel point Tensors in bird's-eye view features with mapping relationships With category q Semantic feature tensor of Similarity It can be calculated by the following formula:

[0117]

[0118] According to the above formula, pixel points can be obtained for both the teacher model and the student model Tensors in bird's-eye view features with mapping relationships With category q Semantic feature tensor of Semantic category similarity 、 .

[0119] Finally, based on the semantic category similarity of the teacher model and the semantic category similarity of the student model, the distillation loss of the semantic category relevance between the student model and the teacher model is calculated.

[0120] In the embodiment of the present invention, the distillation loss of semantic category relevance It can be calculated by the following formula:

[0121]

[0122] Among them, K is the category set, is a hyperparameter.

[0123] Since the bird's-eye view feature is a relatively sparse graph, and the purpose of the model provided by the embodiment of the present invention is to achieve 3D target detection, if only the bird's-eye view features of the student model and the teacher model are aligned, the effect on the categories in the 3D target detection results is not very good. Therefore, when the student model is supervised by the distillation loss of semantic category relevance to map the pixels into the tensor of the bird's-eye view feature, it can be aligned with the teacher model, so that the student model can more keenly perceive the differences in the categories of the pixels.

[0124] 5. Distillation Loss at Volume Rendering Level

[0125] The distillation loss at the volume rendering level can include volume rendering depth loss and volume rendering semantic loss.

[0126] Considering that the student three-dimensional semantic features in the student model include not only semantic features but also depth features, and semantic features can be accurately obtained from multi-view two-dimensional image data, while depth features need to be obtained through depth estimation. In order to improve the accuracy of the student model at the depth estimation level, in an embodiment of the present invention, volume rendering technology is used to project three-dimensional features onto two-dimensional features, so as to divide the three-dimensional features into two angles of semantic features and depth features, and then supervised training is performed on the two angles separately. Compared with directly using three-dimensional features for supervised training, the student model trained in this way has better training effects in both semantic features and depth features.

[0127] The depth loss for volume rendering is calculated as:

[0128] First, a classification query is performed on each voxel in the 3D space according to the 3D hybrid features to obtain the teacher density matrix of the teacher model. Then, based on the teacher density matrix and the 3D hybrid features, the voxel size and density are accumulated using volume rendering technology to obtain the teacher depth features from multiple perspectives in the teacher model.

[0129] Then, each voxel in the three-dimensional space is classified and queried according to the student's three-dimensional semantic features to obtain the student density matrix of the student model. Based on the student density matrix and the student's three-dimensional semantic features, the voxel size and density are accumulated using volume rendering technology to obtain the student depth features of the student model from multiple perspectives.

[0130] Finally, the KL divergence is calculated for the teacher depth features and the student depth features to obtain the volume rendering depth loss between the student model and the teacher model.

[0131] Volume rendering depth loss requires the use of depth features to calculate. For the teacher model, the 3D hybrid feature is a fusion of depth information from LiDAR point cloud data and depth estimation from multi-view 2D image data. This provides more accurate depth information. Therefore, when obtaining the teacher depth features for the teacher model, a 2D projection of the 3D hybrid feature is performed on the depth features in multiple viewing directions. For the student model, however, its depth information comes solely from depth estimation from multi-view 2D image data. Therefore, when obtaining the student depth features for the student model, a 2D projection of the student 3D semantic features is performed on the depth features in multiple viewing directions.

[0132] Among them, the density matrix is ​​obtained by using a classifier to classify each voxel in the three-dimensional space to obtain a voxel classification matrix, and then the category of each voxel is selected based on the voxel classification matrix. Then, by querying known parameters, the density information of each voxel is obtained, and finally the density matrix is ​​obtained.

[0133] After obtaining the density matrix, the voxel size and density are accumulated using volume rendering technology. Specifically, the center of each pixel on the camera imaging plane is used as the starting point of the light path, and the direction from the camera optical center to the pixel center is used as the light path direction. The size and density of all visible voxels along the light path are accumulated. The density accumulation result is combined with the volume rendering formula to make a judgment. If the density accumulation result of accumulating the density of each voxel on the light path reaches a critical value, it is considered invisible; otherwise, it is considered visible. In this way, the depth characteristics of the teacher model and the student model can be obtained.

[0134] After obtaining the depth features, the volume rendering depth loss can be calculated using the following formula: :

[0135]

[0136] in, n is the total number of viewing angles, is the height of the two-dimensional semantic feature, is the width of the two-dimensional semantic feature, is the first matrix corresponding to the student depth feature of the i-th perspective j Rank k Column data, is the first matrix corresponding to the teacher's depth feature of the i-th perspective j Rank k Column data.

[0137] The semantic loss of volume rendering is calculated as:

[0138] First, a classification query is performed on each voxel in the three-dimensional space according to the student's three-dimensional semantic features to obtain the student density matrix of the student model. Based on the student density matrix and the student's three-dimensional semantic features, the semantic information and density are accumulated using volume rendering technology to obtain the student semantic features from multiple perspectives in the student model. The student affinity matrix is ​​then calculated based on the student semantic features.

[0139] Then, the teacher affinity matrix is ​​calculated based on the two-dimensional semantic features of teachers from multiple perspectives;

[0140] Finally, the student affinity matrix and the teacher affinity matrix are used to calculate the volume rendering semantic loss of the student model relative to the teacher model.

[0141] In an embodiment of the present invention, the volume rendering semantic loss needs to be calculated using semantic features. For the teacher model, the lidar point cloud data is used to provide geometric features, that is, depth features, while the semantic features are mainly derived from multi-view two-dimensional image data. Moreover, after passing through the third backbone network and the feature fusion device, the teacher's two-dimensional semantic features will increase errors and cause losses in the semantic features, while the semantic feature information in the teacher's two-dimensional semantic features is more accurate. Therefore, in order to improve the understanding of semantic features when supervising the training of the student model, when calculating the volume rendering semantic loss, the teacher model directly uses the multi-view teacher's two-dimensional semantic features for calculation.

[0142] For the student model, what is supervised during the supervised training process is the entire neural network of the student model. It is necessary to ensure that the semantic feature understanding after passing through the fifth backbone network is still very accurate. Therefore, when calculating the volume rendering semantic loss, the student model uses the student 3D semantic features obtained after passing through the fifth backbone network, and uses volume rendering technology to perform two-dimensional projection of the student 3D semantic features to obtain student semantic features under multiple perspectives, ensuring that the student semantic features are aligned with the teacher 2D semantic features in the teacher model.

[0143] Therefore, for the student model, it is also necessary to use volume rendering technology to project the student's three-dimensional semantic features into the student semantic features in two dimensions. First, it is necessary to obtain the student density matrix. For the method of obtaining the student density matrix, please refer to the description of the calculation method of the volume rendering depth loss of the student model and the teacher model in the above embodiment, which will not be repeated here.

[0144] When projecting into two dimensions, the projection onto semantic features is different from the projection onto depth features. Volume rendering technology needs to accumulate semantic information and also use the density accumulation results to determine whether it is visible, and finally obtain the student semantic features from multiple perspectives in the student model.

[0145] In order to calculate the volume rendering semantic loss of the student model relative to the teacher model, the teacher semantic features and the student two-dimensional semantic features need to be semantically embedded first to adjust the feature dimensions for subsequent calculations, and obtain the teacher semantic embedding features and the student semantic embedding features respectively; then, the teacher affinity matrix can be calculated for the teacher semantic embedding features, and the student affinity matrix can be calculated for the student semantic embedding features to obtain the affinity between pixels in each model. The affinity matrix is ​​calculated as follows:

[0146]

[0147] in, Affinity Matrix Location data, is the data of the xth row and zth column of the semantic embedding feature, The data of the yth row and zth column of the semantic embedding feature, 、 It is the vector composed of the xth and yth row data of the semantic embedding feature.

[0148] According to the above formula, the teacher affinity matrix and the student affinity matrix can be calculated respectively. Then, the volume rendering semantic loss is calculated using the following formula: :

[0149]

[0150] in, refers to the semantic affinity loss, is the volume rendering semantic loss, is a hyperparameter, For the teacher affinity matrix, Affinity Matrix for Students, is the number of channels of the two-dimensional semantic feature, M is the size of the semantic embedding feature after pooling dimensionality reduction, is the feature tensor of the multi-view teacher 2D semantic features of the teacher model; It is the feature tensor of the student semantic features under multiple perspectives of the student model.

[0151] After obtaining the above losses, we can use hyperparameters to add up the above losses to get the total loss function.

[0152] Please refer to Figure 6 , an embodiment of the present invention provides a vehicle-road collaborative 3D object detection device based on knowledge distillation, the device comprising:

[0153] An acquisition unit 600 is configured to acquire a pre-trained teacher model; the teacher model is configured to perform multimodal fusion processing on the input ego-vehicle multimodal data and the input other-vehicle multimodal data, and then collaboratively process the fused ego-vehicle teacher bird's-eye view features and each other-vehicle teacher bird's-eye view features to output a 3D target detection result for the ego-vehicle based on the collaboratively processed teacher bird's-eye view collaborative features; the other-vehicle is another vehicle located in the same scene as the ego-vehicle; the multimodal data includes multi-view 2D image data and LiDAR point cloud data;

[0154] A model construction unit 602 is configured to remove modules related to processing the lidar point cloud data from the teacher model, and utilize the remaining modules to obtain a student model having a similar architecture to the teacher model; the student model is configured to perform single-modal feature extraction on the input multi-view two-dimensional image data, and then collaboratively process the extracted student bird's-eye view features of the own vehicle and the student bird's-eye view features of each other vehicle, thereby outputting a 3D object detection result for the own vehicle based on the collaboratively processed student bird's-eye view collaborative features;

[0155] The model training unit 604 is used to utilize the knowledge distillation method to utilize the teacher model to perform supervised training on the student model, and to utilize the trained student model to perform 3D target detection on the vehicle.

[0156] In one embodiment of the present invention, the teacher model includes: a fusion processing module, a first collaborative processing module, and a first target detection module; the fusion processing module includes: a first backbone network, a second backbone network, a third backbone network, a feature fuser, and a first bird's-eye view feature extractor;

[0157] The multimodal fusion processing method includes: using the first backbone network to extract features from the lidar point cloud data to obtain three-dimensional point cloud features; using the second backbone network to extract features from the multi-view two-dimensional image data to obtain multi-view two-dimensional semantic features of the teacher; using the third backbone network to perform depth estimation on the multi-view two-dimensional semantic features of the teacher to obtain three-dimensional semantic features of the teacher; using the feature fusion module to fuse the three-dimensional point cloud features and the three-dimensional semantic features of the teacher to obtain three-dimensional mixed features; using the first bird's-eye view feature extraction module to project the three-dimensional mixed features into the bird's-eye view features of the teacher;

[0158] The modules related to processing the lidar point cloud data in the teacher model include: a first backbone network and a feature fusion device;

[0159] The student model includes: a feature extraction module, a second collaborative processing module and a second target detection module; the feature extraction module includes: a fourth backbone network, a fifth backbone network and a second bird's-eye view feature extractor;

[0160] The method of single-modal feature extraction includes: using the fourth backbone network to extract features from multi-view two-dimensional image data to obtain multi-view student two-dimensional semantic features; using the fifth backbone network to perform depth estimation on the multi-view student two-dimensional semantic features to obtain student three-dimensional semantic features; using the second bird's-eye view feature extractor to project the student three-dimensional semantic features into student bird's-eye view features.

[0161] In one embodiment of the present invention, when performing 3D target detection on the vehicle using the trained student model, the following steps are specifically included:

[0162] Acquire the multi-view two-dimensional image data of the vehicle at the current moment; use the feature extraction module to extract features from the multi-view two-dimensional image data of the vehicle to obtain the bird's-eye view features of the vehicle at the current moment;

[0163] Obtain the bird's-eye view features of other students in the car at the current moment sent by the external server;

[0164] The second collaborative processing module is used to collaboratively process the bird's-eye view perspective features of the student in the own car and the bird's-eye view perspective features of the student in the other car at the current moment to obtain the bird's-eye view collaborative features of the student in the current moment;

[0165] The second target detection module is used to perform target detection on the student bird's-eye view collaborative features at the current moment to obtain the 3D target detection result of the vehicle at the current moment.

[0166] In one embodiment of the present invention, the bird's-eye view perspective features of the other car that are collaboratively processed are the effective bird's-eye view perspective features of the other car after effective feature screening; the bird's-eye view perspective features of the other car include the bird's-eye view perspective features of the teacher in the other car and the bird's-eye view perspective features of the student in the other car; the effective bird's-eye view perspective features of the other car include the effective bird's-eye view perspective features of the teacher in the other car and the effective bird's-eye view perspective features of the student in the other car.

[0167] In one embodiment of the present invention, the effective feature screening method includes:

[0168] Use the classification network to classify the bird's-eye view features of the other car to obtain the spatial confidence matrix of the other car;

[0169] For each element in the spatial confidence matrix, assign a value of 1 to an element whose value is greater than a set value, and assign a value of 0 to an element whose value is not greater than the set value, to obtain a binary selection matrix;

[0170] The binary selection matrix is ​​multiplied by the corresponding position of the other vehicle's bird's-eye view feature to obtain the other vehicle's effective bird's-eye view feature.

[0171] In one embodiment of the present invention, the total loss during the supervised training process includes at least: a distillation loss of semantic category relevance;

[0172] The calculation method of the distillation loss of the semantic category relevance includes:

[0173] The multi-view original image corresponding to the multi-view 2D image data is classified and labeled pixel by pixel, and the mapping relationship between each pixel in the multi-view original image and each tensor in the bird's-eye view feature in the model is determined. The semantic feature tensor corresponding to each category is calculated based on the mapping relationship;

[0174] For each pixel, the normalized dot product is used to calculate the semantic category similarity between the tensor in the bird's-eye view feature with which the pixel has a mapping relationship and the semantic feature tensor of each category;

[0175] Based on the semantic category similarity of the teacher model and the semantic category similarity of the student model, the distillation loss of the semantic category relevance between the student model and the teacher model is calculated.

[0176] It should be noted that the vehicle-road collaborative 3D target detection device based on knowledge distillation provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the vehicle-road collaborative 3D target detection device based on knowledge distillation provided in the above embodiment and the vehicle-road collaborative 3D target detection method embodiment based on knowledge distillation belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0177] The embodiment of the present application also provides a computer device, please refer to Figure 7 The computer device includes a processor and a memory, in which at least one instruction, at least one program, code set or instruction set is stored. The at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the vehicle-road collaborative 3D target detection method based on knowledge distillation provided in the above-mentioned method embodiments.

[0178] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the vehicle-road collaborative 3D target detection method based on knowledge distillation provided in the above-mentioned method embodiments.

[0179] An embodiment of the present application also provides a computer program product, which includes a computer program. The processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the vehicle-road collaborative 3D target detection method based on knowledge distillation as described in any of the above embodiments.

[0180] For the convenience of description, the above systems or devices are described as being divided into various modules or units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0181] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0182] Finally, it should be noted that, in this document, relational terms such as first, second, third, and fourth are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0183] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A vehicle-road collaborative 3D object detection method based on knowledge distillation, characterized in that: The method comprises: Obtain a pre-trained teacher model; the teacher model is used to perform multimodal fusion processing on the input self-vehicle multimodal data and the input other-vehicle multimodal data, and then collaboratively process the self-vehicle teacher bird's-eye view features obtained by the fusion processing with the bird's-eye view features of each other-vehicle teacher, so as to output a 3D target detection result for the self-vehicle based on the collaboratively processed teacher bird's-eye view collaborative features; the other-vehicle is another vehicle located in the same scene as the self-vehicle; the multimodal data includes multi-view two-dimensional image data and lidar point cloud data; Relevant modules for processing lidar point cloud data in the teacher model are removed, and the remaining modules are used to obtain a student model with a similar architecture to the teacher model; the student model is used to perform single-modal feature extraction on the input multi-view two-dimensional image data, and then collaboratively process the extracted student bird's-eye view features of the own vehicle and the student bird's-eye view features of each other vehicle, so as to output a 3D target detection result for the own vehicle based on the collaboratively processed student bird's-eye view collaborative features; Using a knowledge distillation method to supervise the student model using the teacher model, and then using the trained student model to perform 3D object detection on the vehicle; The total loss during the supervised training process includes at least: a distillation loss of semantic category relevance; The calculation method of the distillation loss of the semantic category relevance includes: The multi-view original image corresponding to the multi-view 2D image data is classified and labeled pixel by pixel, and the mapping relationship between each pixel in the multi-view original image and each tensor in the bird's-eye view feature in the model is determined. The semantic feature tensor corresponding to each category is calculated based on the mapping relationship; For each pixel, the normalized dot product is used to calculate the semantic category similarity between the tensor in the bird's-eye view feature with which the pixel has a mapping relationship and the semantic feature tensor of each category; Based on the semantic category similarity of the teacher model and the semantic category similarity of the student model, the distillation loss of the semantic category correlation between the student model and the teacher model is calculated by the following formula : Among them, Q is the category set, is a hyperparameter, N is the total number of pixels in the multi-view original image, 、 Get pixel points for teacher model and student model respectively l Tensors and categories in bird's-eye view features with mapping relationships q The semantic category similarity of the semantic feature tensor.

2. The method according to claim 1, characterized in that The teacher model includes: a fusion processing module, a first collaborative processing module, and a first target detection module; the fusion processing module includes: a first backbone network, a second backbone network, a third backbone network, a feature fuser, and a first bird's-eye view feature extractor; The multimodal fusion processing method includes: using the first backbone network to extract features from the lidar point cloud data to obtain three-dimensional point cloud features; using the second backbone network to extract features from the multi-view two-dimensional image data to obtain multi-view teacher two-dimensional semantic features; using the third backbone network to perform depth estimation on the multi-view teacher two-dimensional semantic features to obtain teacher three-dimensional semantic features; using the feature fusion module to fuse the three-dimensional point cloud features and the teacher three-dimensional semantic features to obtain three-dimensional mixed features; using the first bird's-eye view feature extraction module to project the three-dimensional mixed features into teacher bird's-eye view features; The modules related to processing the lidar point cloud data in the teacher model include: a first backbone network and a feature fusion device; The student model includes: a feature extraction module, a second collaborative processing module and a second target detection module; the feature extraction module includes: a fourth backbone network, a fifth backbone network and a second bird's-eye view feature extractor; The method of single-modal feature extraction includes: using the fourth backbone network to extract features from multi-view two-dimensional image data to obtain multi-view student two-dimensional semantic features; using the fifth backbone network to perform depth estimation on the multi-view student two-dimensional semantic features to obtain student three-dimensional semantic features; using the second bird's-eye view feature extractor to project the student three-dimensional semantic features into student bird's-eye view features.

3. The method according to claim 2, characterized in that The 3D target detection of the vehicle using the trained student model includes: Acquire the multi-view two-dimensional image data of the vehicle at the current moment; use the feature extraction module to extract features from the multi-view two-dimensional image data of the vehicle to obtain the bird's-eye view features of the vehicle at the current moment; Obtain the bird's-eye view features of other students in the car at the current moment sent by the external server; The second collaborative processing module is used to collaboratively process the bird's-eye view perspective features of the student in the own car and the bird's-eye view perspective features of the student in the other car at the current moment to obtain the bird's-eye view collaborative features of the student in the current moment; The second target detection module is used to perform target detection on the student bird's-eye view collaborative features at the current moment to obtain the 3D target detection result of the vehicle at the current moment.

4. The method according to claim 2 or 3, characterized in that The other car's bird's-eye view features that are collaboratively processed are the other car's effective bird's-eye view features after effective feature screening; the other car's bird's-eye view features include the other car's teacher's bird's-eye view features and the other car's student's bird's-eye view features; the other car's effective bird's-eye view features include the other car's teacher's effective bird's-eye view features and the other car's student's effective bird's-eye view features.

5. The method according to claim 4, characterized in that Effective feature screening methods include: Use the classification network to classify the bird's-eye view features of the other car to obtain the spatial confidence matrix of the other car; For each element in the spatial confidence matrix, assign a value of 1 to an element whose value is greater than a set value, and assign a value of 0 to an element whose value is not greater than the set value, to obtain a binary selection matrix; The binary selection matrix is ​​multiplied by the corresponding position of the other vehicle's bird's-eye view feature to obtain the other vehicle's effective bird's-eye view feature.

6. A vehicle-road collaborative 3D object detection device based on knowledge distillation, characterized in that: The device comprises: an acquisition unit for acquiring a pre-trained teacher model; the teacher model is used to perform multimodal fusion processing on the input self-vehicle multimodal data and the input other-vehicle multimodal data, and then collaboratively process the self-vehicle teacher bird's-eye view perspective features obtained by the fusion processing and the bird's-eye view perspective features of each other-vehicle teacher, so as to output a 3D target detection result for the self-vehicle based on the collaboratively processed teacher bird's-eye view collaborative features; the other-vehicle is another vehicle located in the same scene as the self-vehicle; the multimodal data includes multi-view two-dimensional image data and lidar point cloud data; a model construction unit, configured to remove modules related to processing the lidar point cloud data from the teacher model, and utilize the remaining modules to obtain a student model having a similar architecture to the teacher model; the student model is configured to perform single-modal feature extraction on the input multi-view two-dimensional image data, and then collaboratively process the extracted student bird's-eye view features of the own vehicle and the student bird's-eye view features of each other vehicle, so as to output a 3D target detection result for the own vehicle based on the collaboratively processed student bird's-eye view collaborative features; A model training unit is configured to utilize a knowledge distillation method to supervise the student model using the teacher model, and to perform 3D object detection on the ego-vehicle using the trained student model; The total loss during the supervised training process includes at least: a distillation loss of semantic category relevance; The calculation method of the distillation loss of the semantic category relevance includes: The multi-view original image corresponding to the multi-view 2D image data is classified and labeled pixel by pixel, and the mapping relationship between each pixel in the multi-view original image and each tensor in the bird's-eye view feature in the model is determined. The semantic feature tensor corresponding to each category is calculated based on the mapping relationship; For each pixel, the normalized dot product is used to calculate the semantic category similarity between the tensor in the bird's-eye view feature with which the pixel has a mapping relationship and the semantic feature tensor of each category; Based on the semantic category similarity of the teacher model and the semantic category similarity of the student model, the distillation loss of the semantic category correlation between the student model and the teacher model is calculated by the following formula : Among them, Q is the category set, is a hyperparameter, N is the total number of pixels in the multi-view original image, 、 Get pixel points for teacher model and student model respectively l Tensors and categories in bird's-eye view features with mapping relationships q The semantic category similarity of the semantic feature tensor.

7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the steps of any one of the methods described in claims 1-5.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, characterized in that The method comprises a computer program, which implements the steps of the method according to any one of claims 1 to 5 when the computer program is executed by a processor.

Images