Camera 3D object detection method and system based on multimodal distillation

Abstract

The present invention provides a camera 3D target detection method and system based on multimodal distillation, which relate to the technical field of target detection. The method includes: determining a trained multimodal teacher model with point cloud and multi-view camera images as input, using the point cloud features in the teacher model to perform cross-modal comparative distillation on a student model constructed with multi-view camera images as input, so that the student model has spatial geometric reasoning capabilities to construct pseudo point cloud features, and using a multimodal adaptive smoothing distillation method to align the student model features obtained by heterogeneous fusion of pseudo point cloud features and image features with the teacher model features obtained by channel fusion of point cloud features and image features. When the regression output of the teacher model is better than the regression output of the student model, its regression box knowledge is selected to be distilled into the student model. Through the above training process, the structural consistency of the student model and the teacher model can be ensured, thereby improving the 3D target detection accuracy of the student model.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a camera three-dimensional target detection method and system based on multimodal distillation. Background Art

[0002] Existing camera-based 3D object detection relies on monocular or binocular cameras to infer 3D information from image data. Methods such as geometric constraints and deep learning can achieve 3D detection. However, the camera's inability to directly provide depth information limits its accuracy. While direct projection mechanisms can address the lack of depth information, these methods are hindered by their heavy reliance on idealized geometric constraints and unlearnable priors, leading to a shift towards data-driven paradigms that utilize neural depth estimation. For example, the feature-based Bird's-Eye View (BEV) paradigm has become a leading detection method, achieving unified 3D perception through end-to-end learnable multi-camera feature fusion. This approach demonstrates the ability to effectively mitigate geometric feature loss in depth estimation. Furthermore, the unified BEV representation across multiple views mitigates feature degradation caused by partially occluded scenes. BEV-based methods can currently be divided into two categories: depth estimation methods and query-based methods. However, depth estimation methods still lack sufficient spatial reasoning capabilities due to the lack of accurate LiDAR point cloud guidance. Query-based methods are also limited by considerable computational overhead and inaccurate depth information.

[0003] Although the knowledge distillation method can solve the problems of the BEV-based method, there is a structural inconsistency between the student model and the multimodal teacher model, which makes it difficult to effectively align features and the low-quality output of the teacher model may interfere with the student model. In other words, the existing technology fails to fully consider the inherent differences between the simulated modality and the real modality and the negative impact of forced alignment during the fusion and distillation process of the student model. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a camera three-dimensional target detection method and system based on multimodal distillation in response to the above-mentioned defects of the prior art, aiming to solve the structural inconsistency between the student model and the multimodal teacher model in the prior art, which makes it difficult to perform effective feature alignment and the low-quality output of the teacher model may interfere with the student model.

[0005] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0006] A camera three-dimensional object detection method based on multimodal distillation, wherein the method comprises:

[0007] Identify a trained multimodal teacher model that takes point clouds and multi-view camera images as input for 3D object detection, and build a student model that takes multi-view camera images as input;

[0008] During the training of the student model, the point cloud features in the multimodal teacher model are used to perform cross-modal comparative distillation on the image features in the student model, thereby obtaining a student model with spatial geometric reasoning capabilities, so that the student model can construct corresponding pseudo point cloud features based on the image features.

[0009] Aligning student model features and teacher model features using a multimodal adaptive smoothing distillation method; wherein the student model features are fused features obtained by heterogeneously fusing the pseudo point cloud features and the image features of the student model, and the teacher model features are fused features obtained by channel-wise fusing the multimodal teacher model based on the point cloud features and the image features;

[0010] Obtain the regression output of the student model and the regression output of the multimodal teacher model during training, and determine whether the regression output of the multimodal teacher model is better than the regression output of the student model;

[0011] When the regression output of the multimodal teacher model is better than the regression output of the student model, the regression box knowledge of the multimodal teacher model is distilled and transferred to the student model to obtain a trained student model for 3D object detection.

[0012] In one implementation, performing cross-modal contrast distillation on image features in a student model using point cloud features in a multimodal teacher model includes:

[0013] Calculate the similarity strength of the positive sample feature pair and the similarity strength of the negative sample feature pair;

[0014] Using a preset contrast loss function, and based on the similarity strength of the positive sample feature pair and the similarity strength of the negative sample feature pair, a corresponding contrast loss result is calculated;

[0015] Controlling the student model to implicitly learn a feature space shared with the multimodal teacher model based on the contrastive loss result;

[0016] Among them, the positive sample feature pairs are the point cloud features extracted by the multimodal teacher model and the image features extracted at the corresponding spatial positions of the student model in the same scene, and the negative sample feature pairs are the point cloud features extracted by the multimodal teacher model and the image features extracted by the student model in other scenes, or the point cloud features extracted by the multimodal teacher model and the image features extracted by the student model at non-corresponding spatial positions in the same scene.

[0017] In one implementation, the preset contrast loss function is:

[0018] ;

[0019] in, represents the contrast loss result, represents the cosine similarity function, represents the point cloud features in the multimodal teacher model, represents the positive sample feature, represents the negative sample feature, represents the scaling factor, Represents the similarity strength of the scaled positive sample feature pair, Represents the similarity strength of the scaled negative sample feature pair, Represents the sum of the similarity strengths of all scaled negative sample feature pairs.

[0020] In one implementation, constructing corresponding pseudo point cloud features based on the image features includes:

[0021] Global geometric compensation is performed on the image features to construct corresponding pseudo point cloud features.

[0022] In one implementation, after constructing corresponding pseudo point cloud features based on the image features, the method further includes:

[0023] The pseudo point cloud features in the student model and the point cloud features in the multimodal teacher model are aligned using a cross-modal adaptive smoothing distillation method.

[0024] In one implementation, aligning student model features with teacher model features using a multimodal adaptive smoothing distillation method includes:

[0025] Normalize the student model features and the teacher model features to obtain normalized student model features and normalized teacher model features;

[0026] Calculating the cosine similarity between the normalized student model features and the normalized teacher model features;

[0027] Calculating a corresponding error based on the cosine similarity, and determining whether the error is less than a preset error threshold;

[0028] When the error is less than the preset error threshold, aligning the student model features and the teacher model features using L2 loss;

[0029] When the error is not less than the preset error threshold, the student model features and the teacher model features are aligned using L1 loss.

[0030] In one implementation, heterogeneous fusion of the pseudo point cloud features and the image features is performed to obtain student model features, including:

[0031] Perform channel splicing on the pseudo point cloud BEV features and the image BEV features to obtain the spliced ​​input features;

[0032] Performing global average pooling on the concatenated input features to obtain channel-level global features;

[0033] Generating a channel attention weight based on the channel-level global feature, and correcting the channel attention weight using a preset bias suppression coefficient of the pseudo point cloud feature to obtain a corrected channel attention weight;

[0034] The spliced ​​input features are weighted according to the corrected channel attention weights to obtain corresponding weighted features, and student model features are generated based on the weighted features.

[0035] In one implementation, determining whether the regression output of the multimodal teacher model is better than the regression output of the student model includes:

[0036] Calculate the first three-dimensional intersection-over-union between the regression output of the multimodal teacher model and the true label, and calculate the second three-dimensional intersection-over-union between the regression output of the student model and the true label;

[0037] Adding the second three-dimensional intersection-union ratio to a preset difference threshold to obtain a corresponding addition result;

[0038] Comparing whether the first three-dimensional intersection-union ratio is greater than the addition result to obtain a corresponding comparison result;

[0039] Determining whether the regression output of the multimodal teacher model is better than the regression output of the student model based on the comparison result;

[0040] Wherein, judging whether the regression output of the multimodal teacher model is better than the regression output of the student model based on the comparison result includes:

[0041] When the comparison result indicates that the first three-dimensional intersection-to-union ratio is greater than the addition result, it is determined that the regression output of the multimodal teacher model is better than the regression output of the student model;

[0042] When the comparison result indicates that the first three-dimensional intersection-to-union ratio is not greater than the addition result, it is determined that the regression output of the multimodal teacher model is not better than the regression output of the student model.

[0043] In one implementation, after determining whether the regression output of the multimodal teacher model is better than the regression output of the student model, the method further includes:

[0044] When the regression output of the multimodal teacher model is not better than the regression output of the student model, there is no need to transfer the regression box knowledge distillation of the multimodal teacher model to the student model, and the student model after feature alignment is used as the trained student model for 3D object detection.

[0045] The present invention also discloses a camera three-dimensional object detection system based on multimodal distillation, wherein the system comprises:

[0046] The student model construction module is used to determine the multimodal teacher model trained for 3D object detection using point clouds and multi-view camera images as input, and to construct a student model using multi-view camera images as input;

[0047] The first distillation module is used to perform cross-modal comparative distillation on the image features in the student model using the point cloud features in the multimodal teacher model during the training of the student model, thereby obtaining a student model with spatial geometric reasoning capabilities, so that the student model can construct corresponding pseudo point cloud features based on the image features;

[0048] A second distillation module is configured to align student model features and teacher model features using a multimodal adaptive smoothing distillation method; wherein the student model features are fused features obtained by heterogeneously fusing the pseudo point cloud features and the image features of the student model, and the teacher model features are fused features obtained by channel-wise fusing the point cloud features and the image features of the multimodal teacher model;

[0049] A judgment module is used to obtain the regression output of the student model and the regression output of the multimodal teacher model during the training process, and to judge whether the regression output of the multimodal teacher model is better than the regression output of the student model;

[0050] The third distillation module is used to distill the regression box knowledge of the multimodal teacher model to the student model when the regression output of the multimodal teacher model is better than the regression output of the student model, thereby obtaining a trained student model for 3D target detection.

[0051] The present invention also discloses a terminal, which includes: a memory, a processor, and a camera three-dimensional target detection program based on multimodal distillation stored in the memory and runnable on the processor. When the camera three-dimensional target detection program based on multimodal distillation is executed by the processor, the steps of the camera three-dimensional target detection method based on multimodal distillation as described above are implemented.

[0052] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the camera three-dimensional target detection method based on multimodal distillation as described above.

[0053] The present invention provides a camera 3D target detection method and system based on multimodal distillation. The camera 3D target detection method based on multimodal distillation includes: determining a trained multimodal teacher model that uses point clouds and multi-view camera images as input for 3D target detection, and constructing a student model that uses multi-view camera images as input; in the process of training the student model, using the point cloud features in the multimodal teacher model to perform cross-modal comparative distillation on the image features in the student model to obtain a student model with spatial geometric reasoning capabilities, so that the student model can construct corresponding pseudo point cloud features based on the image features; using a multimodal adaptive smoothing distillation method to align the student model features with the teacher model features Characteristic; wherein, the student model feature is a fused feature obtained by heterogeneously fusing the pseudo point cloud feature and the image feature of the student model, and the teacher model feature is a fused feature obtained by channel fusion of the multimodal teacher model based on the point cloud feature and the image feature; the regression output of the student model and the regression output of the multimodal teacher model are obtained during the training process, and it is determined whether the regression output of the multimodal teacher model is better than the regression output of the student model; when the regression output of the multimodal teacher model is better than the regression output of the student model, the regression box knowledge possessed by the multimodal teacher model is distilled and transferred to the student model to obtain a trained student model for three-dimensional target detection. It can be seen that the present invention enables the student model to have spatial geometric reasoning capabilities through cross-modal contrast distillation, and can construct a pseudo point cloud, ensuring the structural consistency of the student model and the multimodal teacher model, and then uses the multimodal adaptive smooth distillation method to align the student model features obtained based on the heterogeneous fusion of pseudo point cloud features and image features and the teacher model features obtained based on the channel fusion of point cloud features and image features. Among them, the reliability of the student model fusion can be improved by heterogeneous fusion of pseudo point cloud features and image features. Compared with equal-weight fusion, it reduces the influence of unreliable modalities on the fused multimodal features, improves the effectiveness of subsequent multimodal distillation, and adopts multimodal Adaptive smooth distillation allows a certain amount of retention space between the student model and the multimodal teacher model during distillation, effectively reducing the impact of misalignment and enhancing model robustness. When the regression output of the multimodal teacher model is better than that of the student model, the regression box knowledge of the multimodal teacher model is distilled into the student model. That is, the regression box knowledge of the multimodal teacher model is distilled and transferred to the student model through selective regression distillation, so that the outputs of the student model and the multimodal teacher model are further aligned, which improves the geometric reasoning ability of the student model, avoids the introduction of regression noise, reduces unnecessary regression distillation, and thus improves the accuracy of the student model's three-dimensional target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flowchart of a preferred embodiment of the camera 3D object detection method based on multimodal distillation in the present invention;

[0055] Figure 2 This is a specific schematic block diagram of pseudo point cloud generation logic disclosed in the present invention;

[0056] Figure 3 It is a logical schematic block diagram based on multimodal distillation in the present invention;

[0057] Figure 4 This is a flowchart of a specific camera 3D object detection method based on multimodal distillation disclosed in the present invention;

[0058] Figure 5 It is another logic schematic block diagram based on multimodal distillation in the present invention;

[0059] Figure 6 This is a functional principle block diagram of a preferred embodiment of a camera-based three-dimensional object detection system based on multimodal distillation in the present invention;

[0060] Figure 7 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0062] In existing technologies, camera-based three-dimensional target detection can rely on monocular or binocular cameras to infer the three-dimensional information of objects through image data. For example, three-dimensional detection can be achieved through methods such as geometric constraints and deep learning. However, the camera cannot directly provide depth information, which limits its detection accuracy.

[0063] Early methods attempted to address the problem of insufficient depth information through direct projection mechanisms, but were hindered by heavy reliance on idealized geometric constraints and non-learnable priors, turning to a data-driven paradigm that utilizes neural depth estimation.

[0064] Building on these advances, the feature-based bird's-eye view paradigm has emerged as a leading approach to achieve unified 3D perception through end-to-end learnable multi-camera feature fusion. This approach demonstrates the ability to effectively mitigate the loss of geometric features in depth estimation, while also alleviating feature degradation caused by partially occluded scenes through a unified BEV representation across multiple views.

[0065] Among them, BEV-based methods can currently be divided into two categories: depth estimation methods and query-based methods. First, multi-view 3D object detection methods based on depth estimation, such as LSS and CaDDN, first use a depth estimation network to predict the depth of each pixel and then promote it to 3D space. On this basis, BEVDet integrates the input of multiple cameras to create a unified bird's-eye view space, thereby eliminating complex 2D to 3D post-processing and improving detection efficiency. BEVDet4D adds temporal modeling to BEVDet, integrating historical BEV frames to capture the motion trajectory of dynamic objects. BEVDepth enhances the features of BEV through explicit depth estimation and uses sensors such as lidar to provide depth supervision signals to achieve more accurate 3D geometric modeling. Although this method effectively solves the depth ambiguity problem of the camera and improves positioning accuracy, it still lacks sufficient spatial reasoning capabilities without the guidance of accurate lidar point clouds. Second, query-based multi-view 3D object detection methods typically learn a set of fixed or learnable queries, which can be in 3D or 2D representations, and use them to sample and aggregate features from multiple views. The advantages of these methods lie in their good scalability, high flexibility, and efficient feature fusion. For example, DETR3D utilizes learnable 3D object queries and an attention mechanism to transform 2D image features from multiple views into a 3D object detection problem. PETR encodes 3D coordinate position information into image features, enabling object queries to perceive position-aware 3D features, thereby achieving efficient end-to-end multi-view 3D object detection. PETRv2 proposes a unified framework for 3D perception from multi-view images. By combining temporal modeling and task-specific queries, it significantly improves the performance of 3D object detection, BEV segmentation, and 3D lane detection. SparseBEV utilizes scale-adaptive attention mechanisms, adaptive spatiotemporal sampling, and adaptive hybrid sampling to efficiently aggregate and decode features in the BEV space, outperforming dense detectors while maintaining real-time performance. However, query-based multi-view 3D object detection methods are limited by considerable computational overhead and inaccurate depth information.

[0066] While knowledge distillation can address the issues with BEV-based methods, it can suffer from issues such as insufficient multimodal teacher model performance, structural inconsistencies between the student model and the multimodal teacher model, and inherent differences between simulated and real modalities. This can lead to a failure to fully consider inherent differences between simulated and real modalities during the fusion and distillation process, as well as negative impacts from forced alignment. Knowledge distillation improves student model performance by transferring knowledge from a large, high-performance multimodal teacher model to a low-performance, lightweight student model without changing the network structure or significantly increasing the number of parameters or computational cost.

[0067] For example, as a groundbreaking framework, BEVDistill utilizes dense feature distillation to emphasize foreground areas, enabling the student model to focus on sparse but critical instances and key areas. UniDistill also employs triple distillation (feature / correlation / response) to suppress the influence of background while maintaining semantic consistency in the alignment of sparse foreground features. Next, by dividing the regions, DistillBEV designs a regional distillation method to reduce the interference between background and imbalanced samples. Distill-based adversarial training is also used to reduce modal differences in feature alignment, and label distillation is used to minimize the transfer of erroneous information from the multimodal teacher model. However, these methods cannot address the inherent limitations of using only a lidar model as a multimodal teacher model, nor the negative impact of the structural inconsistency between the student model and the multimodal teacher model. These deficiencies hinder the effectiveness of knowledge transfer and limit the overall performance of the framework.

[0068] For example, SimDistill uses a lidar-camera multimodal model as a multimodal teacher model, and achieves structural alignment between the student model and the multimodal teacher model through a geometric compensation module, ensuring the effective transfer of knowledge and improved performance. It adopts a comprehensive framework that combines multimodal alignment, intra-modal alignment, cross-modal alignment and response distribution. However, in the fusion and distillation process, it fails to fully consider the inherent differences between the simulated modality and the real modality and the negative impact of forced alignment.

[0069] To this end, the present application provides a camera three-dimensional target detection solution based on multimodal distillation, which can ensure the structural consistency of the student model and the multimodal teacher model, allow a certain retention space between the student model and the multimodal teacher model during distillation, effectively reduce the impact of misalignment, enhance the model robustness, and further align the outputs of the student model and the multimodal teacher model, thereby improving the geometric reasoning ability of the student model, avoiding the introduction of regression noise, reducing unnecessary regression distillation, and thus improving the regression accuracy of the student model.

[0070] See Figure 1 , Figure 1 This is a flow chart of the camera 3D target detection method based on multimodal distillation in the present invention. Figure 1 As shown, the camera 3D target detection method based on multimodal distillation according to the embodiment of the present invention includes:

[0071] Step S11: determine a trained multimodal teacher model that uses point clouds and multi-view camera images as input for three-dimensional object detection, and construct a student model that uses multi-view camera images as input.

[0072] In this embodiment, a multimodal teacher model is first determined, and its inputs are point clouds and multi-view camera images. Three-dimensional object detection is achieved based on these points and multi-view camera images. This means that the multimodal teacher model has two input branches, one for point cloud data and one for camera image data. Simultaneously, a student model is constructed, taking the multi-view camera image as input. This means that the student model has only one input, the camera image data. The point cloud can be point cloud data collected by a lidar. Because the unimodal teacher model and the student model have significant modal and structural differences, which can affect the distillation effect, and the performance of a unimodal teacher model is far inferior to that of a multimodal teacher model that integrates lidar and camera, a multimodal teacher model that integrates lidar and camera can be used to improve the teacher model's performance. Furthermore, to ensure structural similarity between the multimodal teacher model and the student model, a pseudo point cloud can be constructed on the student side. For example, SimDistill's multimodal architecture can be used as the multimodal teacher model. Cross-modal contrastive distillation is then used to equip the student model with spatial geometric reasoning capabilities to construct the pseudo point cloud.

[0073] It should be noted that the multimodal teacher model refers to a multimodal artificial intelligence model that plays the role of a teacher in the knowledge distillation scenario. Its core function is to use its powerful multimodal understanding and reasoning capabilities to guide the training of student models. The student model is usually a single modal or lightweight model, and the multimodal teacher model can simultaneously process and integrate multiple types of data (modalities), and the multimodal teacher model can understand the relationship between different modalities.

[0074] It's also worth noting that 3D object detection aims to identify and locate objects in three-dimensional space from sensor data, outputting information such as the object's category, position, size, and orientation. Unlike 2D object detection, 3D object detection provides depth information for objects, which is crucial for applications such as autonomous driving and robotic navigation. 3D object detection is achieved through multi-view cameras. Multi-view camera-based 3D object detection utilizes image data from multiple cameras and fuses information from different perspectives to improve the accuracy and robustness of 3D object detection. Compared to monocular or binocular cameras, multi-view cameras can cover a wider field of view, reduce occlusion issues, and provide richer geometric information.

[0075] Step S12: During the training of the student model, the point cloud features in the multimodal teacher model are used to perform cross-modal comparative distillation on the image features in the student model to obtain a student model with spatial geometric reasoning capabilities, so that the student model can construct corresponding pseudo point cloud features based on the image features.

[0076] It's important to note that the multimodal teacher model leverages its deep understanding of multimodal data to generate richer supervisory signals, encompassing not only labels but also implicit logic and feature associations. Guided by the multimodal teacher model, the trained student model can achieve performance close to that of the teacher model, with lower computational cost and easier deployment, empowering single-modal models with the potential for cross-modal understanding.

[0077] In this embodiment, the student model is endowed with spatial geometric reasoning capabilities through cross-modal contrast distillation, that is, the image features in the student model are subjected to cross-modal contrast distillation using the point cloud features corresponding to the point cloud input into the multimodal teacher model, to obtain a student model with spatial geometric reasoning capabilities, so that the student model with spatial geometric reasoning capabilities can construct corresponding pseudo point cloud features based on the image features corresponding to the input multi-view camera images, thereby improving the reliability of the pseudo point cloud features.

[0078] Understandably, the pseudo point cloud generated solely by the camera-side student model is unreliable. Therefore, during training, the point cloud features from the multimodal teacher model are first introduced to perform cross-modal contrastive distillation on the camera-side student model. This constructs more reliable pseudo point cloud features with the help of the point cloud features. This means that the point cloud is used as the input for the multimodal teacher model, and the spatial reasoning capabilities of the multimodal teacher model are distilled and transferred to the student model, thereby enhancing the student model's spatial geometric reasoning capabilities. Cross-modal contrastive distillation effectively alleviates problems such as depth ambiguity and sparse spatial feature representation in the student model.

[0079] In this embodiment, the point cloud features in the multimodal teacher model are used to perform cross-modal contrast distillation on the image features in the student model, which may specifically include: calculating the matching strength of the positive sample feature pair and the matching strength of the negative sample feature pair; using a preset contrast loss function, and calculating the corresponding contrast loss result based on the matching strength of the positive sample feature pair and the matching strength of the negative sample feature pair; based on the contrast loss result, controlling the student model to implicitly learn the feature space shared with the multimodal teacher model; wherein, the positive sample feature pair is the point cloud features extracted by the multimodal teacher model and the image features extracted at the corresponding spatial position of the student model in the same scene, and the negative sample feature pair is the point cloud features extracted by the multimodal teacher model and the image features extracted by the student model in other scenes, or the point cloud features extracted by the multimodal teacher model and the image features extracted by the student model at non-corresponding spatial positions in the same scene.

[0080] Among them, the preset contrast loss function is:

[0081] ;

[0082] in, Represents the contrast loss result, which is used to measure the degree of feature space alignment. Represents the cosine similarity function, which is used to calculate the similarity of two feature vectors to measure The degree of matching with the positive sample feature pair or the negative sample feature pair, represents the point cloud features in the multimodal teacher model, represents the positive sample features, that is, the image features that match the point cloud features of the multimodal teacher model in the student model feature space, represents negative sample features, that is, image features that do not match the point cloud features of the multimodal teacher model in the student model feature space, Represents a scaling factor, which is used to adjust the similarity score distribution, that is, to adjust the sharpness of the probability distribution. represents the similarity strength of the scaled positive sample feature pair, that is, the similarity between the point cloud features of the multimodal teacher model and the positive sample features, represents the positive sample feature pair, Represents the similarity strength of the scaled negative sample feature pair, that is, the similarity between the point cloud features of the multimodal teacher model and the negative sample features, represents the negative sample feature pair, Represents the sum of the similarity strengths of all scaled negative sample feature pairs. In addition, the denominator of the above preset contrast loss function is the overall similarity strength of the positive and negative sample feature pairs. The optimization direction is to maximize the similarity of the positive sample feature pairs and minimize the similarity of the negative sample feature pairs.

[0083] It is understandable that the raw data distribution of sparse 3D point clouds and dense 2D pixels is usually very different. Directly using mean square error to force alignment will cause the model to overfit to noise, such as projection error and sensor noise. Therefore, this application adopts contrast loss to model the similarity relationship between sample pairs, such as the similarity of positive sample feature pairs and the dissimilarity of negative sample feature pairs, and implicitly learn the shared feature space across modalities, so that the student model does not need to directly align values, but learns high-level semantic associations between modalities.

[0084] Furthermore, in this embodiment, a student model with spatial geometric reasoning capabilities constructs a pseudo point cloud based on image features. Specifically, this may include performing global geometric compensation on the image features to construct the corresponding pseudo point cloud. It should be noted that in detection methods based on multi-view cameras, a view projection operation is required to convert perspective view (PV) features into BEV space. However, inaccurate geometric information inference during this process can lead to geometric misalignment between image features and point cloud features, exacerbating inter-modal differences. Therefore, SimDistill's geometric compensation method can be used to perform global geometric compensation before and after the view projection operation to learn more accurate geometric features in perspective view space and bird's-eye view space. For example, a transformer architecture is introduced based on the deformable self-attention layer of the student model to construct a geometric compensation module, and global geometric compensation is achieved through the geometric compensation module in the student model.

[0085] For example, see Figure 2 As shown in the figure, the point cloud features corresponding to the point cloud collected by the lidar are introduced to perform cross-modal comparative distillation on the student model on the camera side, so that it has spatial geometric reasoning capabilities, and the image features corresponding to the multi-view camera images collected by the camera are geometrically compensated through the geometric compensation module to construct the corresponding pseudo point cloud features, thereby improving the accuracy of the generated pseudo point cloud features and further improving the reliability of the pseudo point cloud features.

[0086] For example, first generate a uniformly distributed grid point set as the query point for each image feature, that is, given a 2D camera feature map F∈R H×W×C , first generate a uniformly distributed query point set Q, namely:

[0087] ;

[0088] in, It is Feature query points The image coordinates of N=H×W are the total number of feature query points.

[0089] For each feature query point, the single head offset is first calculated to generate a set of points that are most relevant to it. The points learned in the point set will be used as reference points and keys to sample feature values ​​from image features, thereby providing preliminary position correction for geometric compensation for the single attention head.

[0090] For example, first, for each feature query point , learn its offset ,Right now;

[0091] ;

[0092] Among them, MLP (Multi-Layer Perceptron) represents a multi-layer perceptron, which is used to query points based on features. and 2D camera feature map F, predicting how to adjust the features location or characteristics;

[0093] Secondly, based on the position update mechanism, the dynamic adjustment of the feature query point is achieved through spatial position iteration. Specifically, for each reference point , the feature query point With offset Add, that is:

[0094] ;

[0095] in, is the learned reference point, that is, the adjusted feature query point.

[0096] Finally, at the position of the adjusted feature query point, the feature value of the reference point is calculated by neighborhood weighted aggregation, that is:

[0097] ;

[0098] in, for The surrounding neighborhood, is the weight calculated based on bilinear interpolation, Represents neighborhood points The eigenvalue at .

[0099] The geometric compensation module optimizes the attention weights by back-propagating gradients through the self-attention mechanism, dynamically adjusts the attention space position distribution, thereby compensating for the geometric transformation in the PV space (xy plane), and calculating the attention weights through QK correlation , which enables the network to autonomously learn how to adjust the weights of geometric transformations in PV space, such as translation, rotation, or scaling, thereby improving the spatial adaptability of features, namely:

[0100] ;

[0101] in, Indicates the query vector With the Key vector The degree of correlation, is the key vector Dimensions, is the natural exponential function, Indicates the Key vector The transpose of .

[0102] Map the dot product similarity to the positive range and amplify the weight differences of significantly related query-key pairs, that is:

[0103] ;

[0104] in, Is the value (Value), used for weighted summation to obtain the final output .

[0105] The above operation improves the robustness of the student model to geometric errors, enabling it to capture more accurate geometric features. At the same time, the standard multi-head attention mechanism is applied, and each attention head learns the offset separately. , learn the offset separately, that is:

[0106] ;

[0107] in, is the feature query point, which represents the grid point on the 2D image. F represents the input 2D camera feature map with a shape of H×W×C. Represents the first The attention head is The geometric offset of the feature query point is used to adjust the position of the query point. For the A multi-layer perceptron (MLP) dedicated to each attention head dynamically predicts the position correction for each feature query point in the current feature space through a learnable MLP network. Different attention heads independently learn offsets to achieve multi-scale geometric modeling.

[0108] Then, update the reference point:

[0109] ;

[0110] in, For the The reference point corresponding to each attention head, that is, the corrected feature query point, is constrained to be within the valid range of the feature map;

[0111] Then perform feature sampling, that is:

[0112] ;

[0113] in, For the The compensated eigenvector of the head, For the reference point area, The sampling location The learnable weights of , generated by bilinear interpolation or an independent weight prediction network, For the feature map The eigenvalue of the location.

[0114] Finally, the output of each attention head is weighted fusion into the final feature ,Right now:

[0115] ;

[0116] in, is the importance weight of the attention head, is a learnable parameter, and B is the total number of attention heads.

[0117] This approach can capture rich information, improve feature representation, and introduce a multi-head attention mechanism in the geometric compensation module. Through multi-subspace collaborative modeling, hardware-level computational optimization, and dynamic feature fusion, it achieves global modeling and a two-way breakthrough in accuracy and efficiency. The geometric compensation module significantly improves the geometric misalignment of PV and BEV spaces. It utilizes a two-stage deformable attention mechanism and a multi-head self-attention mechanism (PV space + BEV space) to dynamically predict the position offset of feature points and compensate before and after view projection. This corrects the geometric distortion caused by projecting the image onto a bird's-eye view, achieves spatial alignment of the student model features with the multimodal teacher model LiDAR features, effectively alleviating the modal difference between image features and point cloud features, thereby improving the accuracy of pseudo point cloud generation and the effectiveness of subsequent distillation.

[0118] Step S13: Use a multimodal adaptive smoothing distillation method to align the student model features and the teacher model features; wherein the student model features are fused features obtained by heterogeneously fusing the pseudo point cloud features and the image features of the student model, and the teacher model features are fused features obtained by channel fusion of the multimodal teacher model based on the point cloud features and the image features.

[0119] It should be pointed out that although pseudo point cloud features are constructed in the student model on the camera side, a multimodal fusion of pseudo point cloud features and image features is constructed, thereby ensuring the structural consistency of the student model and the multimodal teacher model, narrowing the feature differences between the student model and the teacher model, and improving the reliability of the pseudo point cloud features, there are still essential differences between the two from the perspective of input data. The student model only has multi-view camera images collected by the camera as input, while the multimodal teacher model has not only multi-view camera images collected by the camera as input but also point clouds collected by the lidar as input. Therefore, when performing multimodal distillation, using loss functions such as mean-variance to force the student model to learn all teacher model features and align them strictly in the same feature space may lose modality-specific information. At the same time, forced alignment may cause the student model to over-rely on the specific alignment pattern of the multimodal teacher model in the training data, rather than learning the essential cross-modal association, which may result in poor model generalization performance. Finally, forced alignment of different modalities may introduce complex constraints, such as strict loss functions, which makes optimization difficult, making the training process unstable, convergence slow, and requiring more computing resources. To this end, in this application, multimodal adaptive smooth distillation is used to address the shortcomings of forced alignment, allowing elastic correlation between modalities. When the student model and the multimodal teacher model are quite different, a certain amount of spatial misalignment is allowed. At the same time, high-frequency noise can be filtered through a threshold, and the alignment strength can be dynamically adjusted. That is, a multimodal adaptive smooth distillation method is used to align the student model features and the teacher model features, allowing a certain amount of retention space between the student model and the multimodal teacher model during distillation, effectively reducing the impact of misalignment and enhancing the robustness of the model. In other words, the multimodal adaptive smooth distillation method can alleviate the feature misalignment caused by forced alignment.

[0120] It should be noted that the student model uses heterogeneous fusion to fuse pseudo point cloud features with image features to generate student model features, while the multimodal teacher model uses channel fusion to fuse point cloud features with image features to generate teacher model features. While the aforementioned steps improve the accuracy of constructed pseudo point cloud features, the BEV feature reliability remains low compared to the camera's BEV features. To mitigate the impact of this low reliability on multimodal distillation, the student model does not fuse the pseudo point cloud and image features with equal weights. Instead, it adopts heterogeneous fusion, or asymmetric fusion, which prioritizes image features during fusion. This emphasizes the semantic information of the camera to dominate the fusion process while preserving the spatial structure of the pseudo point cloud as a supplement. For example, image features serve as the perceptual backbone, while pseudo point cloud features serve as auxiliary branches. Furthermore, during the feature fusion stage, image features are significantly weighted higher than pseudo point cloud features. This weighting is adaptively adjusted during training, but image features are always weighted higher than pseudo point cloud features. Heterogeneous fusion improves the reliability of the student model's fusion and facilitates subsequent feature alignment.

[0121] In this embodiment, student model features are obtained based on heterogeneous fusion of pseudo point cloud features and image features, which may specifically include: determining pseudo point cloud BEV features and image BEV features; performing channel splicing on the pseudo point cloud BEV features and image BEV features to obtain spliced ​​input features; performing global average pooling on the spliced ​​input features to obtain channel-level global features; generating channel attention weights based on the channel-level global features, and correcting the channel attention weights using the preset bias suppression coefficient of the pseudo point cloud features to obtain corrected channel attention weights; weighting the spliced ​​input features according to the corrected channel attention weights to obtain corresponding weighted features, and generating student model features based on the weighted features.

[0122] For example, using attention-guided fusion, we first perform channel concatenation, i.e.:

[0123] ;

[0124] in, is the point cloud BEV feature, the number of channels is , is the image BEV feature, the number of channels is , X is the concatenated input feature, H is the spatial height under the BEV perspective, and W is the spatial width under the BEV perspective.

[0125] Next, perform global average pooling (GAP) on X to obtain the channel-level global features, namely:

[0126] ;

[0127] in, It is the global feature on channel c, describing the overall activation strength of the channel. Indicates the spatial position of channel c The eigenvalue of .

[0128] After GAP, a channel description vector S is obtained, and the channel attention weight A is generated, that is:

[0129] ;

[0130] in, is the first layer fully connected weight matrix, is the second layer fully connected weight matrix, is the activation function, that is, the channel attention weight is generated by the fully connected layer and the activation function .

[0131] In order to give the image features a higher weight, define the bias mask, namely:

[0132] ;

[0133] Where C is the channel index, is the bias mask, is the channel number threshold corresponding to the image feature, is the bias suppression coefficient of the pseudo point cloud feature, and 0<λ<1, the mask acts on the channel attention weight , and get the corrected weight ,Right now:

[0134] ;

[0135] Then, according to the modified weight Weight the concatenated feature X, namely:

[0136] ;

[0137] in, Represents channel-wise dot product.

[0138] Finally, feature fusion is performed to obtain the fused features (i.e., student model features) ,Right now:

[0139] ;

[0140] in, It consists of convolution (Conv2D), batch normalization (BatchNorm), and ReLU activation function.

[0141] It should be pointed out that by calculating the fusion features through channel attention (SE mechanism), introducing pseudo point cloud bias, and artificially suppressing the pseudo point cloud weight, we can ensure that the camera contributes more during fusion, reduce the interference of unreliability on subsequent multimodal distillation, and improve the accuracy of distillation.

[0142] In this embodiment, a multimodal adaptive smoothing distillation method is used to align the student model features and the teacher model features, which can specifically include: normalizing the student model features and the teacher model features to obtain normalized student model features and normalized teacher model features; calculating the cosine similarity between the normalized student model features and the normalized teacher model features; calculating the corresponding error based on the cosine similarity, and judging whether the error is less than a preset error threshold; when the error is less than the preset error threshold, the student model features and the teacher model features are aligned using the L2 form loss; when the error is not less than the preset error threshold, the student model features and the teacher model features are aligned using the L1 form loss. It can be understood that the student model features and the teacher model features are aligned by a preset piecewise robust loss function to reduce the feature difference between the student model and the multimodal teacher model, wherein the L2 form loss, i.e., the L2 form loss, i.e., the preset mean square loss, can achieve refinement, and the L1 form loss, i.e., the L1 form loss, i.e., the preset linear loss, can effectively suppress the influence of outliers.

[0143] For example, first normalize the student model features and the teacher model features, that is:

[0144] ;

[0145] ;

[0146] in, is the normalized student model feature, is the normalized teacher model feature, is the student model feature, T is the teacher model feature, is the L2 norm.

[0147] Next, calculate the cosine similarity between the two features, that is:

[0148] ;

[0149] in, is the cosine similarity, which measures the degree of alignment of two features. It is the dimension of teacher characteristics.

[0150] Then, calculate the error ,Right now:

[0151] ;

[0152] Among them, the smaller the error, the closer the student model characteristics are to the teacher model characteristics.

[0153] When the error is less than the threshold, the L2 form loss is used, that is, , further reduce the feature difference between the student model and the multimodal teacher model, continue fine-grained optimization to achieve the best performance, and use L1 loss when the error is not less than the threshold, that is, , allowing certain differences between the characteristics of the student model and the teacher model, while effectively suppressing the influence of outliers and ensuring the stability of training, that is:

[0154] ;

[0155] Among them, margin is the set error threshold. Through multimodal adaptive smooth distillation, when the error is small, the student model features are more easily aligned with the teacher model features, improving the knowledge distillation effect. When the error is large, a certain degree of misalignment is allowed to ensure training stability and enhance model robustness.

[0156] Step S14: Obtain the regression output of the student model and the regression output of the multimodal teacher model during the training process, and determine whether the regression output of the multimodal teacher model is better than the regression output of the student model.

[0157] It should be noted that the output of the model includes two parts: classification and regression. The classification is judged by the texture and semantic information provided by the camera. Blindly passing the LiDAR information to the camera will only interfere with its originally correct classification score and confidence. Therefore, retaining the independent decision-making power of the student model can avoid the introduction of classification noise by sparse point clouds. At the same time, the precise geometric measurement characteristics of LiDAR, such as depth and size, are used to transfer the 3D positioning advantages of the multimodal teacher model to the student model through distillation. Therefore, in this embodiment, in order to reduce the interference of the classification knowledge of the multimodal teacher model, classification knowledge distillation is not performed. The output distillation is only performed on the regression part, and a judgment will be made, that is, whether the regression output of the multimodal teacher model is better than the regression output of the student model.

[0158] Specifically, the first three-dimensional intersection-and-union ratio between the regression output of the multimodal teacher model and the true label is calculated, and the second three-dimensional intersection-and-union ratio between the regression output of the student model and the true label is calculated; the second three-dimensional intersection-and-union ratio is added to a preset difference threshold to obtain a corresponding addition result; the first three-dimensional intersection-and-union ratio is compared to see whether it is greater than the addition result to obtain a corresponding comparison result; based on the comparison result, it is determined whether the regression output of the multimodal teacher model is better than the regression output of the student model; when the comparison result shows that the first three-dimensional intersection-and-union ratio is greater than the addition result, it is determined that the regression output of the multimodal teacher model is better than the regression output of the student model; when the comparison result shows that the first three-dimensional intersection-and-union ratio is not greater than the addition result, it is determined that the regression output of the multimodal teacher model is not better than the regression output of the student model.

[0159] Step S15: When the regression output of the multimodal teacher model is better than the regression output of the student model, the regression box knowledge of the multimodal teacher model is distilled and transferred to the student model to obtain a trained student model for three-dimensional target detection.

[0160] In this embodiment, distillation is performed only when the output of the multimodal teacher model is better than the output of the student model, so that the regression output of the student model imitates the regression output of the multimodal teacher model, narrowing the difference between the two. When the regression output of the multimodal teacher model is not better than the regression output of the student model, there is no need to distill the regression box knowledge possessed by the multimodal teacher model to the student model, and the student model after feature alignment is used as the trained student model for three-dimensional object detection. It can be understood that through selective regression distillation, the outputs of the student model and the multimodal teacher model are further aligned, thereby improving the geometric reasoning ability of the student model.

[0161] For example, first, the comparison result of the regression output of the student model and the regression output of the multimodal teacher model is used as a condition, that is:

[0162] ;

[0163] in, is the indicator function, when the condition is met =1, otherwise =0, IoU() calculates the 3D intersection-over-union ratio between the regression output and the true label, represents the regression output of the multimodal teacher model for the i-th sample, Indicates that the student model The regression output of samples is represents the true label of the i-th sample, is the IOU difference threshold that triggers distillation.

[0164] When the regression output quality of the multimodal teacher model is better than the regression output of the student model, distillation is triggered, and the distillation loss can be:

[0165] ;

[0166] ;

[0167] in, In order to smooth the L1 loss, a smooth L1 loss is used to enhance the robustness to outliers. d is the input variable, which represents the difference between the regression output and the true label. When the regression output of the multimodal teacher model is significantly better than the regression output of the student model, that is, |d|≥1, the student model is guided by a stable gradient to quickly approach the teacher output. When the student model is close to the teacher level, that is, |d|<1, the regression parameters are fine-tuned.

[0168] It can be seen that in the embodiment of the present invention, through cross-modal contrast distillation, the student model has spatial geometric reasoning ability and can construct a pseudo point cloud, ensuring the structural consistency of the student model and the multimodal teacher model. Then, the multimodal adaptive smooth distillation method is used to align the student model features obtained by heterogeneous fusion of pseudo point cloud features and image features and the teacher model features obtained by channel fusion of point cloud features and image features. Among them, the reliability of student model fusion can be improved by heterogeneous fusion of pseudo point cloud features and image features. Compared with equal-weight fusion, it reduces the influence of unreliable modalities on the fused multimodal features, improves the effectiveness of subsequent multimodal distillation, and adopts multi-modal adaptive smooth distillation. Modal adaptive smooth distillation allows a certain amount of retention space between the student model and the multimodal teacher model during distillation, effectively reducing the impact of misalignment and enhancing model robustness. When the regression output of the multimodal teacher model is better than that of the student model, the regression box knowledge of the multimodal teacher model is distilled into the student model. That is, the regression box knowledge of the multimodal teacher model is distilled and transferred to the student model through selective regression distillation, so that the outputs of the student model and the multimodal teacher model are further aligned, which improves the geometric reasoning ability of the student model, avoids the introduction of regression noise, reduces unnecessary regression distillation, and thus improves the accuracy of the student model's three-dimensional target detection.

[0169] It should be noted that the research on camera-based 3D object detection is of great significance, mainly in the following aspects:

[0170] (1) In autonomous driving systems, three-dimensional object detection is the core task of understanding the surrounding environment, such as vehicles, pedestrians, and obstacles. It provides the autonomous driving system with key information such as the precise position, size, and motion state of objects, such as speed and direction, and is a direct input for subsequent path planning, collision prediction, and decision control.

[0171] (2) Compared with LiDAR, cameras are cheaper and easier to deploy. Pure vision-based three-dimensional detection technology can promote the large-scale commercialization of autonomous driving. At the same time, pure vision solutions are also a prerequisite for building efficient multimodal systems.

[0172] See also Figure 3 As shown in the figure, the student model only has the multi-view camera images collected by the camera as input, while the multimodal teacher model not only has the multi-view camera images collected by the camera as input, but also has the point cloud collected by the lidar as input. Therefore, first, through cross-modal contrast distillation, the student model is given the ability of spatial geometric reasoning, the accuracy of the generated pseudo point cloud is improved, and the structural consistency of the student model and the multimodal teacher model is ensured. Then, the multimodal adaptive smooth distillation method is used to align the student model features obtained by the heterogeneous fusion of pseudo point cloud features and image features and the teacher model features obtained by the channel fusion of point cloud features and image features, that is, the multimodal adaptive smooth distillation method is used to align the student model features and the teacher model features. Among them, the heterogeneous fusion of pseudo point cloud features and image features can improve the reliability of the student model fusion. Compared with equal-weight fusion, it reduces the influence of unreliable modalities on the fused multimodal features, and improves the subsequent The effectiveness of multimodal distillation, and the multimodal adaptive smooth distillation allows a certain retention space for the student model and the multimodal teacher model during distillation, effectively reducing the impact of misalignment and enhancing the robustness of the model. When the regression output of the multimodal teacher model is better than the regression output of the student model, the regression box knowledge of the multimodal teacher model is distilled into the student model, that is, the regression box knowledge of the multimodal teacher model is distilled and transferred to the student model through selective regression distillation, so that the output of the student model and the multimodal teacher model are further aligned, which improves the geometric reasoning ability of the student model, avoids the introduction of regression noise, reduces unnecessary regression distillation, and thus improves the regression accuracy of the student model, that is, it can improve the performance of camera 3D target detection, greatly reduce the current camera 3D target detection in real-world scenes. The phenomenon of missed detection and inaccurate positioning, and keep the reasoning speed basically unchanged, achieving real-time detection effect.

[0173] See also Figure 4 As shown, an embodiment of the present invention discloses a specific camera three-dimensional target detection method based on multimodal distillation. Compared with the previous embodiment, this embodiment further explains and optimizes the technical solution.

[0174] Step S21: Determine a trained multimodal teacher model that uses point clouds and multi-view camera images as input for three-dimensional object detection, and construct a student model that uses multi-view camera images as input.

[0175] Step S22: During the training of the student model, the point cloud features in the multimodal teacher model are used to perform cross-modal comparative distillation on the image features in the student model to obtain a student model with spatial geometric reasoning capabilities, so that the student model can construct corresponding pseudo point cloud features based on the image features.

[0176] Step S23: align the pseudo point cloud features in the student model and the point cloud features in the multimodal teacher model using a cross-modal adaptive smoothing distillation method.

[0177] In this embodiment, after constructing the corresponding pseudo point cloud features based on the image features, the cross-modal adaptive smooth distillation method can be used to align the pseudo point cloud features in the student model and the point cloud features in the multimodal teacher model. It can be understood that in the process of training the student model, the point cloud features in the multimodal teacher model are used to perform cross-modal comparative distillation on the student model, so that it has a certain spatial geometric reasoning ability, and the corresponding pseudo point cloud features are constructed based on the image features. The cross-modal adaptive smooth distillation method can be further used to align the pseudo point cloud features in the student model and the point cloud features in the multimodal teacher model. On the one hand, it can improve the credibility of the pseudo point cloud features, enhance its geometric reasoning ability and the accuracy of depth estimation, and on the other hand, it can reduce modal differences, which is conducive to the subsequent heterogeneous fusion of multimodal features and multimodal adaptive smooth distillation. Then, in the subsequent steps, the multimodal adaptive smooth distillation method is used again to align the student model features obtained based on the heterogeneous fusion of pseudo point cloud features and image features and the teacher model features obtained based on the channel fusion of point cloud features and image features.

[0178] Step S24: align the student model features and the teacher model features using a multimodal adaptive smoothing distillation method; wherein the student model features are fused features obtained by heterogeneously fusing the pseudo point cloud features and the image features of the student model, and the teacher model features are fused features obtained by channel fusion of the multimodal teacher model based on the point cloud features and the image features.

[0179] Step S25: Obtain the regression output of the student model and the regression output of the multimodal teacher model during the training process, and determine whether the regression output of the multimodal teacher model is better than the regression output of the student model.

[0180] Step S26: When the regression output of the multimodal teacher model is better than the regression output of the student model, the regression box knowledge of the multimodal teacher model is distilled and transferred to the student model to obtain a trained student model for three-dimensional target detection.

[0181] For the specific contents of the above steps S21 to S22 and steps S24 to S26, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0182] It can be seen that in the embodiment of the present invention, through cross-modal contrast distillation, the student model has spatial geometric reasoning capabilities and can construct a pseudo point cloud, ensuring the structural consistency of the student model and the multimodal teacher model. Then, through the cross-modal adaptive smooth distillation method, the pseudo point cloud features in the student model and the point cloud features in the multimodal teacher model are aligned to further improve the reliability of the pseudo point cloud features. Then, the multimodal adaptive smooth distillation method is used to align the student model features obtained based on the heterogeneous fusion of pseudo point cloud features and image features and the teacher model features obtained based on the channel fusion of point cloud features and image features. Among them, the heterogeneous fusion of pseudo point cloud features and image features can improve the reliability of student model fusion, and reduce the influence of unreliable modalities on fusion compared with equal-weighted fusion. The influence of the combined multimodal features is improved, the effectiveness of subsequent multimodal distillation is improved, and the use of multimodal adaptive smooth distillation allows a certain retention space for the student model and the multimodal teacher model during distillation, effectively reducing the impact of misalignment and enhancing the robustness of the model. When the regression output of the multimodal teacher model is better than the regression output of the student model, the regression box knowledge of the multimodal teacher model is distilled into the student model, that is, the regression box knowledge possessed by the multimodal teacher model is distilled and transferred to the student model through selective regression distillation, so that the outputs of the student model and the multimodal teacher model are further aligned, which improves the geometric reasoning ability of the student model, avoids the introduction of regression noise, reduces unnecessary regression distillation, and thus improves the accuracy of the student model's three-dimensional target detection.

[0183] See also Figure 5As shown in the figure, the student model only has the multi-view camera images collected by the camera as input, while the multimodal teacher model not only has the multi-view camera images collected by the camera as input, but also the point cloud collected by the lidar as input. Therefore, firstly, through cross-modal contrast distillation, the student model is given the ability of spatial geometric reasoning, the accuracy of the generated pseudo point cloud is improved, and the structural consistency of the student model and the multimodal teacher model is ensured. Furthermore, through cross-modal adaptive smooth distillation, the pseudo point cloud features in the student model and the point cloud features in the multimodal teacher model are aligned to further improve the reliability of the pseudo point cloud features. Then, the multimodal adaptive smooth distillation method is used to align the student model features obtained by heterogeneous fusion of pseudo point cloud features and image features and the teacher model features obtained by channel fusion of point cloud features and image features. Among them, the heterogeneous fusion of pseudo point cloud features and image features can improve the reliability of student model fusion, and reduce the fusion of unreliable modalities compared with equal-weighted fusion. The influence of the multimodal features of the multimodal model is improved, the effectiveness of subsequent multimodal distillation is improved, and the multimodal adaptive smooth distillation allows a certain retention space for the student model and the multimodal teacher model during distillation, effectively reducing the impact of misalignment and enhancing the robustness of the model. When the regression output of the multimodal teacher model is better than the regression output of the student model, the regression box knowledge of the multimodal teacher model is distilled into the student model, that is, the regression box knowledge of the multimodal teacher model is distilled and transferred to the student model through selective regression distillation, so that the outputs of the student model and the multimodal teacher model are further aligned, which improves the geometric reasoning ability of the student model, avoids the introduction of regression noise, reduces unnecessary regression distillation, and thus improves the regression accuracy of the student model, that is, it can improve the performance of camera 3D target detection, greatly reduce the current camera 3D target detection in real-world scenes. The phenomenon of missed detection and inaccurate positioning, and keep the reasoning speed basically unchanged, achieving real-time detection effect.

[0184] In one embodiment, if Figure 6 As shown, based on the above-mentioned camera 3D object detection method based on multimodal distillation, the present invention also provides a camera 3D object detection system based on multimodal distillation, including:

[0185] a student model construction module 11 for determining a multimodal teacher model trained to perform 3D object detection using point clouds and multi-view camera images as input, and constructing a student model using multi-view camera images as input;

[0186] A first distillation module 12 is configured to perform cross-modal comparative distillation on image features in the student model using point cloud features in the multimodal teacher model during training of the student model, thereby obtaining a student model with spatial geometric reasoning capabilities, so that the student model can construct corresponding pseudo point cloud features based on the image features;

[0187] A second distillation module 13 is configured to align student model features and teacher model features using a multimodal adaptive smoothing distillation method; wherein the student model features are fused features obtained by heterogeneously fusing the pseudo point cloud features and the image features of the student model, and the teacher model features are fused features obtained by channel-wise fusing the point cloud features and the image features of the multimodal teacher model;

[0188] A judgment module 14 is used to obtain the regression output of the student model and the regression output of the multimodal teacher model during the training process, and to judge whether the regression output of the multimodal teacher model is better than the regression output of the student model;

[0189] The third distillation module 15 is used to distill the regression box knowledge of the multimodal teacher model to the student model when the regression output of the multimodal teacher model is better than the regression output of the student model, so as to obtain a trained student model for three-dimensional target detection.

[0190] In addition, it is worth noting that the working process of the camera three-dimensional target detection system based on multimodal distillation provided in this embodiment is the same as the working process of the above-mentioned camera three-dimensional target detection method based on multimodal distillation. It will not be repeated here. For details, please refer to the working process of the above-mentioned camera three-dimensional target detection method based on multimodal distillation.

[0191] Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0192] Memory 501 , processor 502 , and computer programs stored in the memory 501 and executable on the processor 502 .

[0193] When the processor 502 executes the program, the camera three-dimensional object detection method based on multimodal distillation provided in the above embodiment is implemented.

[0194] Furthermore, the terminal further includes:

[0195] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0196] The memory 501 is used to store computer programs that can be run on the processor 502 .

[0197] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0198] If memory 501, processor 502, and communication interface 503 are implemented independently, communication interface 503, memory 501, and processor 502 can be interconnected via a bus to facilitate communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the figure uses only one line, but this does not imply that there is only one bus or only one type of bus.

[0199] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0200] The processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0201] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned camera three-dimensional target detection method based on multimodal distillation.

[0202] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the claims.

[0203] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0204] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as a sequenced list of executable instructions for implementing the logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can read and execute instructions from an instruction execution system, apparatus, or device).

[0205] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logical functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0206] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A camera 3D object detection method based on multimodal distillation, characterized in that: The method comprises: Identify a trained multimodal teacher model that takes point clouds and multi-view camera images as input for 3D object detection, and build a student model that takes multi-view camera images as input; During the training of the student model, the point cloud features in the multimodal teacher model are used to perform cross-modal comparative distillation on the image features in the student model, thereby obtaining a student model with spatial geometric reasoning capabilities, so that the student model can construct corresponding pseudo point cloud features based on the image features. Aligning student model features and teacher model features using a multimodal adaptive smoothing distillation method; wherein the student model features are fused features obtained by heterogeneously fusing the pseudo point cloud features and the image features of the student model, and the teacher model features are fused features obtained by channel-wise fusing the multimodal teacher model based on the point cloud features and the image features; Obtain the regression output of the student model and the regression output of the multimodal teacher model during training, and determine whether the regression output of the multimodal teacher model is better than the regression output of the student model; When the regression output of the multimodal teacher model is better than that of the student model, the regression box knowledge of the multimodal teacher model is distilled and transferred to the student model to obtain a trained student model for 3D object detection. The method of performing cross-modal comparative distillation on image features in the student model using point cloud features in the multimodal teacher model includes: Calculate the similarity strength of the positive sample feature pair and the similarity strength of the negative sample feature pair; Using a preset contrast loss function, and based on the similarity strength of the positive sample feature pair and the similarity strength of the negative sample feature pair, a corresponding contrast loss result is calculated; Controlling the student model to implicitly learn a feature space shared with the multimodal teacher model based on the contrastive loss result; Among them, the positive sample feature pair is the point cloud feature extracted by the multimodal teacher model and the image feature extracted at the corresponding spatial position of the student model in the same scene, and the negative sample feature pair is the point cloud feature extracted by the multimodal teacher model and the image feature extracted by the student model in other scenes, or the point cloud feature extracted by the multimodal teacher model and the image feature extracted by the student model at non-corresponding spatial positions in the same scene; Furthermore, the preset contrast loss function is: ； in, represents the contrast loss result, represents the cosine similarity function, represents the point cloud features in the multimodal teacher model, represents the positive sample feature, represents the negative sample feature, represents the scaling factor, Represents the similarity strength of the scaled positive sample feature pair, Represents the similarity strength of the scaled negative sample feature pair, Represents the sum of the similarity strengths of all scaled negative sample feature pairs.

2. The camera 3D object detection method based on multimodal distillation according to claim 1, characterized in that: The constructing corresponding pseudo point cloud features based on the image features includes: Global geometric compensation is performed on the image features to construct corresponding pseudo point cloud features.

3. The camera 3D object detection method based on multimodal distillation according to claim 1, characterized in that: After constructing the corresponding pseudo point cloud features based on the image features, the method further includes: The pseudo point cloud features in the student model and the point cloud features in the multimodal teacher model are aligned using a cross-modal adaptive smoothing distillation method.

4. The camera 3D object detection method based on multimodal distillation according to claim 1, characterized in that: The method of using multimodal adaptive smooth distillation to align student model features and teacher model features includes: Normalize the student model features and the teacher model features to obtain normalized student model features and normalized teacher model features; Calculating the cosine similarity between the normalized student model features and the normalized teacher model features; Calculating a corresponding error based on the cosine similarity, and determining whether the error is less than a preset error threshold; When the error is less than the preset error threshold, aligning the student model features and the teacher model features using L2 loss; When the error is not less than the preset error threshold, the student model features and the teacher model features are aligned using L1 loss.

5. The camera 3D object detection method based on multimodal distillation according to claim 1, characterized in that: The student model features are obtained by performing heterogeneous fusion based on the pseudo point cloud features and the image features, including: Perform channel splicing on the pseudo point cloud BEV features and the image BEV features to obtain the spliced ​​input features; Performing global average pooling on the concatenated input features to obtain channel-level global features; Generating a channel attention weight based on the channel-level global feature, and correcting the channel attention weight using a preset bias suppression coefficient of the pseudo point cloud feature to obtain a corrected channel attention weight; The spliced ​​input features are weighted according to the corrected channel attention weights to obtain corresponding weighted features, and student model features are generated based on the weighted features.

6. The camera 3D object detection method based on multimodal distillation according to claim 1, characterized in that: The determining whether the regression output of the multimodal teacher model is better than the regression output of the student model includes: Calculate the first three-dimensional intersection-over-union between the regression output of the multimodal teacher model and the true label, and calculate the second three-dimensional intersection-over-union between the regression output of the student model and the true label; Adding the second three-dimensional intersection-union ratio to a preset difference threshold to obtain a corresponding addition result; Comparing whether the first three-dimensional intersection-union ratio is greater than the addition result to obtain a corresponding comparison result; Determining whether the regression output of the multimodal teacher model is better than the regression output of the student model based on the comparison result; Wherein, judging whether the regression output of the multimodal teacher model is better than the regression output of the student model based on the comparison result includes: When the comparison result indicates that the first three-dimensional intersection-to-union ratio is greater than the addition result, it is determined that the regression output of the multimodal teacher model is better than the regression output of the student model; When the comparison result indicates that the first three-dimensional intersection-to-union ratio is not greater than the addition result, it is determined that the regression output of the multimodal teacher model is not better than the regression output of the student model.

7. The camera 3D object detection method based on multimodal distillation according to any one of claims 1 to 6, characterized in that: After determining whether the regression output of the multimodal teacher model is better than the regression output of the student model, the method further includes: When the regression output of the multimodal teacher model is not better than the regression output of the student model, there is no need to transfer the regression box knowledge distillation of the multimodal teacher model to the student model, and the student model after feature alignment is used as the trained student model for 3D object detection.

8. A camera 3D object detection system based on multimodal distillation, characterized in that: The system comprises: The student model construction module is used to determine the multimodal teacher model trained for 3D object detection using point clouds and multi-view camera images as input, and to construct a student model using multi-view camera images as input; The first distillation module is used to perform cross-modal comparative distillation on the image features in the student model using the point cloud features in the multimodal teacher model during the training process of the student model, thereby obtaining a student model with spatial geometric reasoning capabilities, so that the student model can construct corresponding pseudo point cloud features based on the image features; A second distillation module is configured to align student model features and teacher model features using a multimodal adaptive smoothing distillation method; wherein the student model features are fused features obtained by heterogeneously fusing the pseudo point cloud features and the image features of the student model, and the teacher model features are fused features obtained by channel-wise fusing the point cloud features and the image features of the multimodal teacher model; A judgment module is used to obtain the regression output of the student model and the regression output of the multimodal teacher model during the training process, and to judge whether the regression output of the multimodal teacher model is better than the regression output of the student model; The third distillation module is used to distill the regression box knowledge of the multimodal teacher model into the student model when the regression output of the multimodal teacher model is better than the regression output of the student model, thereby obtaining a trained student model for 3D object detection; Wherein, the first distillation module is specifically used for: Calculate the similarity strength of the positive sample feature pair and the similarity strength of the negative sample feature pair; Using a preset contrast loss function, and based on the similarity strength of the positive sample feature pair and the similarity strength of the negative sample feature pair, a corresponding contrast loss result is calculated; Controlling the student model to implicitly learn a feature space shared with the multimodal teacher model based on the contrastive loss result; Among them, the positive sample feature pair is the point cloud feature extracted by the multimodal teacher model and the image feature extracted at the corresponding spatial position of the student model in the same scene, and the negative sample feature pair is the point cloud feature extracted by the multimodal teacher model and the image feature extracted by the student model in other scenes, or the point cloud feature extracted by the multimodal teacher model and the image feature extracted by the student model at non-corresponding spatial positions in the same scene; Furthermore, the preset contrast loss function is: ； in, represents the contrast loss result, represents the cosine similarity function, represents the point cloud features in the multimodal teacher model, represents the positive sample feature, represents the negative sample feature, represents the scaling factor, Represents the similarity strength of the scaled positive sample feature pair, Represents the similarity strength of the scaled negative sample feature pair, Represents the sum of the similarity strengths of all scaled negative sample feature pairs.

Images