Automatic driving perception method and device based on cross-modal distillation

Abstract

The invention relates to an automatic driving perception method and device based on cross-modal distillation, and the method comprises the steps: enabling a point cloud teacher model to guide an image student model through feature extraction, dynamic attention weight calculation, deep confidence filtering and feature alignment, dual-view comparison distillation, total loss optimization, model training and deployment, and the like. And in combination with a dynamic view attention mechanism and deep confidence filtering, efficient migration of 3D knowledge to a 2D model is realized, and finally, a pure image model has a 3D perception capability close to a point cloud model. Therefore, the problems that in the prior art, a pure vision vehicle type is difficult to meet the high-precision perception requirement under the condition of no laser radar configuration, meanwhile, a fixed weight cannot adjust the learning key point according to the scene complexity, and the mapping noise interference of the same image pixel is large are solved.

Description

Technical Field

[0001] This application relates to the field of cross-modal knowledge distillation technology, and in particular to an autonomous driving perception method and device based on cross-modal distillation. Background Technology

[0002] In the current field of autonomous driving environmental perception, cross-modal knowledge distillation technology is widely used to fuse complementary information from point clouds and images to improve model performance. Existing technologies mostly construct distillation frameworks based on hybrid views (such as Image PlaneView (IPV) and Bird's Eye View (BEV)), establishing feature correspondences between point clouds and images (such as pixel-to-point and superpixel-to-superpoint mappings) to achieve cross-modal transfer of semantic and geometric knowledge. For example, some existing technologies can calculate contrast loss separately for IPV and BEV by fixing the distillation weights, guiding the student model to learn the feature distribution of the teacher model; other technologies can generate sparse depth information through point cloud projection to assist in extending image features into 3D space, thereby strengthening cross-modal alignment. These methods have shown some effectiveness in multi-sensor configurations, but there is still room for optimization in areas such as view weight adaptation and robustness to dynamic scenes. In addition, existing technologies mainly rely on public datasets for training. Their core idea is to establish cross-modal feature associations through geometric projection relationships, and use contrastive learning or knowledge distillation loss to achieve feature alignment, ultimately improving the accuracy of the perception model for tasks such as object detection and semantic segmentation.

[0003] However, existing technologies still have the following drawbacks: 1. Performance limitations of purely visually-oriented vehicles: Perception models relying on LiDAR have high hardware costs, and existing distillation methods cannot fully transfer 3D knowledge of point clouds to image models, making it difficult for pure vision vehicles to meet high-precision perception requirements without LiDAR configuration.

[0004] 2. Poor adaptability of fixed weights: IPV is responsible for learning local details (such as pedestrians in the distance and traffic signs), while BEV is responsible for learning the global layout (such as the positional relationship between vehicles). Fixed weights cannot adjust the learning focus according to the complexity of the scene, resulting in a 5%-20% decrease in target detection accuracy in complex scenes (such as urban congestion) and insufficient accuracy in small target recognition in simple scenes (such as highways).

[0005] 3. High level of mapping noise interference: The same image pixel may correspond to multiple point clouds at different depths (such as overlapping vehicle projection areas). Direct mapping will introduce a lot of noise, resulting in a reduction of semantic segmentation accuracy by 8%-12%, which will affect the accuracy of 3D knowledge transfer.

[0006] In summary, in the existing technologies, pure vision-based vehicle models without LiDAR configurations struggle to meet high-precision perception requirements. Furthermore, fixed weights cannot adjust the learning focus according to scene complexity, and the mapping noise interference of the same image pixel is significant, all of which urgently need to be addressed. Summary of the Invention

[0007] This application provides an autonomous driving perception method and device based on cross-modal distillation to solve the problems in the prior art, such as pure vision vehicles failing to meet high-precision perception requirements without LiDAR configuration, fixed weights failing to adjust learning focus according to scene complexity, and large noise interference in the mapping of the same image pixel.

[0008] The first aspect of this application provides an autonomous driving perception method based on cross-modal distillation, comprising the following steps: acquiring point cloud data and image data of a target vehicle; inputting the preprocessed point cloud data into a pre-constructed teacher model and the image data into a pre-constructed student model to extract corresponding point cloud BEV features, point cloud scene complexity index, and image features; dynamically allocating attention weights corresponding to the image features and the point cloud BEV features based on a preset weight allocation rule and the point cloud scene complexity index; calculating the depth value variance of the point cloud within the projection range corresponding to the superpixel of each image input to the IPV branch of the student model; and performing a preset depth confidence filtering operation based on the depth value variance to obtain... The target point cloud confidence level is obtained by achieving a target point cloud confidence level that meets a preset confidence requirement. Based on the target point cloud confidence level, a preset super-point feature alignment operation is performed on the superpixels to obtain point cloud alignment pairing features. Based on the point cloud alignment pairing features and the attention weights, an IPV view contrast distillation loss function and a BEV view contrast distillation loss function are constructed. Through the IPV view contrast distillation loss function and the BEV view contrast distillation loss function, a corresponding total loss function is constructed. Based on the total loss function, the model parameter gradient of the student model is iteratively calculated and updated until the total loss function meets a preset convergence requirement to obtain the trained student model. The trained student model is then used to perform the corresponding autonomous driving perception operation.

[0009] Optionally, in one embodiment of this application, the step of acquiring point cloud data and image data of the target vehicle, inputting the preprocessed point cloud data into a pre-built teacher model, and inputting the image data into a pre-built student model to extract corresponding point cloud BEV features, point cloud scene complexity index, and image features includes: using the teacher model to perform feature encoding and decoding operations on the point cloud data to obtain a corresponding point cloud feature map, performing global average pooling on the point cloud feature map to obtain a corresponding global feature vector, and inputting the global feature vector into a preset fully connected layer to quantize the global feature vector into a corresponding point cloud scene complexity index; and processing the point cloud data using a preset 3D convolutional layer. The Z-axis of the cloud feature map is downsampled to obtain the corresponding point cloud BEV features. The IPV branch of the student model is used to extract features from the image data to generate the corresponding target dimension feature map. The number of channels in the target dimension feature map is adjusted, and a preset pixel similarity clustering operation is performed on the adjusted target dimension feature map to obtain the image superpixel features in the image features. The target dimension feature map is input to the LSS depth estimation head of the BEV branch of the student model to obtain the corresponding depth map. Based on preset camera intrinsic and extrinsic parameters, the depth map is mapped to a 3D feature map, and depth-oriented pooling is performed on the 3D feature map to generate the image BEV features in the image features.

[0010] Optionally, in one embodiment of this application, the step of dynamically allocating attention weights corresponding to the image features and the point cloud BEV features based on a preset weight allocation rule and the point cloud scene complexity index, and calculating the depth value variance of the point cloud within the projection range corresponding to the superpixel of each image input to the student model IPV branch, performing a preset depth confidence filtering operation based on the depth value variance to obtain a target point cloud confidence that meets the preset confidence requirements, and performing a preset superpixel feature alignment operation on the superpixel based on the target point cloud confidence to obtain point cloud aligned pairing features, includes: acquiring the driving information of the target vehicle, determining the corresponding driving scene based on the driving information, constructing a corresponding weight allocation rule based on the driving scene, and allocating corresponding attention weights to the image features and the point cloud features based on the weight allocation rule and the point cloud scene complexity index; and transforming the pixel coordinates of the superpixels of each image of the IPV branch to the three-dimensional coordinate range of the point cloud coordinate system. Within the image superpixel, the point cloud projection region corresponding to each superpixel is determined, and all point clouds within the projection region are traversed to obtain the depth value of each point cloud. The depth value variance is calculated based on the depth values ​​of all point clouds. It is then determined whether the depth value variance is greater than a preset depth value threshold. If it is greater than the preset depth value threshold, the corresponding point cloud projection region is identified as a depth ambiguity region. The confidence score of all point clouds within the depth ambiguity region is calculated, and the confidence score is compared with a preset confidence threshold. Point clouds with confidence scores less than or equal to the preset confidence threshold are filtered out to obtain the target point cloud confidence score greater than the preset confidence threshold. A preset clustering grouping operation is performed on the target point cloud confidence scores to obtain multiple groups of point clouds. The features of each group of point clouds are averaged and aggregated to generate multiple target point cloud superpixel features. These multiple target point cloud superpixel features are projected onto the image superpixel plane using preset camera intrinsic parameters, ensuring that each target point cloud superpixel feature is spatially aligned with its corresponding image superpixel feature.

[0011] Optionally, in one embodiment of this application, the step of constructing an IPV view contrastive distillation loss function and a BEV view contrastive distillation loss function based on the point cloud alignment and pairing features and the attention weights, and constructing a corresponding total loss function through the IPV view contrastive distillation loss function and the BEV view contrastive distillation loss function, includes: constructing multiple IPV view positive sample data based on the point cloud alignment and pairing features, randomly sampling each IPV view positive sample data to obtain first random sample data, and constructing multiple IPV view negative sample data based on the first random sample data, so as to construct a total loss function based on the multiple IPV view negative sample data and the attention weights. The IPV view contrastive distillation loss function is constructed using multiple IPV view positive sample data and the attention weights. The point cloud BEV features are paired with the image BEV features to obtain corresponding paired features. Multiple BEV view positive sample data are constructed based on the paired features. Each BEV view positive sample data is randomly sampled to obtain corresponding second random sample data. Multiple BEV view negative sample data are constructed based on the second random sample data. The BEV view contrastive distillation loss function is then constructed based on the multiple BEV view negative sample data, the multiple BEV view positive sample data, and the attention weights.

[0012] Optionally, in one embodiment of this application, the mathematical expression of the total loss function is:

[0013] in, This indicates the IPV view compared to the distillation loss function; This indicates the BEV view compared to the distillation loss function; This represents the depth prediction loss function; This represents the weighting coefficients of the depth prediction loss function.

[0014] A second aspect of this application provides an autonomous driving perception device based on cross-modal distillation, comprising: a feature extraction module, used to acquire point cloud data and image data of a target vehicle, and input the preprocessed point cloud data into a pre-built teacher model, and input the image data into a pre-built student model, to extract corresponding point cloud BEV features, point cloud scene complexity index, and image features; and a feature alignment module, used to dynamically allocate attention weights corresponding to the image features and the point cloud BEV features based on a preset weight allocation rule and the point cloud scene complexity index, and calculate the depth value variance of the point cloud within the projection range corresponding to the superpixel of each image input to the IPV branch of the student model, so as to perform preset depth confidence filtering based on the depth value variance. The process involves several steps: First, obtaining a target point cloud confidence level that meets a pre-set confidence requirement. Then, based on this confidence level, performing a pre-set super-point feature alignment operation on the superpixels to obtain point cloud alignment and pairing features. Second, a training module is used to construct an IPV view contrast distillation loss function and a BEV view contrast distillation loss function based on the point cloud alignment and pairing features and the attention weights. Using these two functions, a corresponding total loss function is constructed. Based on this total loss function, the gradient of the student model's model parameters is iteratively calculated and updated until the total loss function meets a pre-set convergence requirement, resulting in a trained student model. Finally, the trained student model is used to perform corresponding autonomous driving perception operations.

[0015] Optionally, in one embodiment of this application, the feature extraction module includes: a quantization unit, configured to perform feature encoding and decoding operations on the point cloud data using the teacher model to obtain a corresponding point cloud feature map, and to perform global average pooling on the point cloud feature map to obtain a corresponding global feature vector, and to input the global feature vector into a preset fully connected layer to quantize the global feature vector into a corresponding point cloud scene complexity index; a downsampling unit, configured to perform downsampling operations on the Z-axis of the point cloud feature map through a preset 3D convolutional layer to obtain the corresponding point cloud BEV feature; and an adjustment unit, configured to utilize the student The model's IPV branch extracts features from image data to generate a corresponding target dimension feature map, adjusts the number of channels in the target dimension feature map, and performs a preset pixel similarity clustering operation on the adjusted target dimension feature map to obtain image superpixel features in the image features. The mapping unit is used to input the target dimension feature map into the LSS depth estimation head of the BEV branch of the student model to obtain a corresponding depth map, and maps the depth map into a three-dimensional feature map based on preset camera intrinsic and extrinsic parameters, and performs depth-direction pooling on the three-dimensional feature map to generate image BEV features in the image features.

[0016] Optionally, in one embodiment of this application, the feature alignment module includes: a weight allocation unit, configured to acquire driving information of the target vehicle, determine the corresponding driving scenario based on the driving information, construct a corresponding weight allocation rule based on the driving scenario, and allocate corresponding attention weights to the image features and the point cloud features based on the weight allocation rule and the point cloud scene complexity index; a projection unit, configured to transform the pixel coordinates of the superpixels of each image of the IPV branch to the three-dimensional coordinate range of the point cloud coordinate system, determine the point cloud projection area corresponding to each superpixel, traverse all point clouds within the point cloud projection area to obtain the depth value of each point cloud, and calculate the depth value variance based on the depth values ​​of all point clouds; and a judgment unit, configured to judge whether the depth value variance is greater than a preset depth. The system includes a threshold value, wherein if the value is greater than the preset depth threshold, the corresponding point cloud projection region is determined to be a depth ambiguity region; a filtering unit, used to calculate the confidence score of all point clouds within the depth ambiguity region, compare the confidence score with a preset confidence threshold, and filter point clouds with confidence scores less than or equal to the preset confidence threshold to obtain the target point cloud confidence score greater than the preset confidence threshold; a grouping unit, used to perform a preset clustering grouping operation on the target point cloud confidence score to obtain multiple groups of point clouds, and to perform average aggregation on the features of each group of point clouds to generate multiple target point cloud super-point features; and an alignment unit, used to project the multiple target point cloud super-point features onto the image superpixel plane using preset camera intrinsic parameters, so that each target point cloud super-point feature is spatially aligned with the corresponding image superpixel feature.

[0017] Optionally, in one embodiment of this application, the training module includes: a first loss function construction unit, configured to construct multiple IPV view positive sample data based on the point cloud alignment pairing features, and randomly sample each IPV view positive sample data to obtain first random sample data, and construct multiple IPV view negative sample data based on the first random sample data, so as to construct the IPV view contrastive distillation loss function based on the multiple IPV view negative sample data, the multiple IPV view positive sample data, and the attention weight; and a second loss function construction unit, configured to pair the point cloud BEV features with the image BEV features to obtain corresponding pairing features, and construct multiple corresponding BEV view positive sample data based on the pairing features, and randomly sample each BEV view positive sample data to obtain corresponding second random sample data, and construct multiple BEV view negative sample data based on the second random sample data, so as to construct the BEV view contrastive distillation loss function based on the multiple BEV view negative sample data, the multiple BEV view positive sample data, and the attention weight.

[0018] Optionally, in one embodiment of this application, the mathematical expression of the total loss function is:

[0019] in, This indicates the IPV view compared to the distillation loss function; This indicates the BEV view compared to the distillation loss function; This represents the depth prediction loss function; This represents the weighting coefficients of the depth prediction loss function.

[0020] A third aspect of this application provides a vehicle, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the autonomous driving perception method based on cross-modal distillation as described in the above embodiments.

[0021] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described autonomous driving perception method based on cross-modal distillation.

[0022] A fifth aspect of this application provides a computer program product, including a computer program that is executed to implement the above-described autonomous driving perception method based on cross-modal distillation.

[0023] Therefore, the embodiments of this application have the following beneficial effects: The embodiments of this application can acquire point cloud data and image data of the target vehicle, input the preprocessed point cloud data into a pre-built teacher model, and input the image data into a pre-built student model to extract the corresponding point cloud BEV features, point cloud scene complexity index, and image features. Based on preset weight allocation rules and point cloud scene complexity index, attention weights corresponding to image features and point cloud BEV features are dynamically allocated, and the depth value variance of the point cloud within the projection range corresponding to the superpixel of each image input to the IPV branch of the student model is calculated. Based on the depth value variance, a preset depth confidence filtering operation is performed to obtain a result that meets the preset confidence requirements. The target point cloud confidence is used to perform pre-defined super-point feature alignment on superpixels to obtain point cloud aligned pairing features. Based on the point cloud aligned pairing features and attention weights, IPV view contrast distillation loss functions and BEV view contrast distillation loss functions are constructed. A corresponding total loss function is then constructed using these two functions. The gradient of the student model's parameters is iteratively calculated and updated based on the total loss function until it meets a pre-defined convergence requirement, resulting in a trained student model. This trained student model is then used to perform corresponding autonomous driving perception operations. This application constructs a dual-model architecture of point cloud teacher and image student, and achieves knowledge transfer through operations such as superpixel pooling, feature alignment, and NCE loss calculation. It also incorporates dynamic weight adjustment and deep confidence filtering mechanisms to optimize model performance, thereby improving the accuracy of 3D knowledge transfer to image models and reducing the hardware cost of autonomous driving. This solves the problems in existing technologies, such as the inability of pure vision vehicles to meet high-precision perception requirements without LiDAR configuration, the inability of fixed weights to adjust the learning focus according to scene complexity, and the large noise interference in the mapping of the same image pixel.

[0024] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0025] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of an autonomous driving perception method based on cross-modal distillation provided according to an embodiment of this application; Figure 2 A schematic diagram of the logical architecture of an autonomous driving perception method based on cross-modal distillation, provided for one embodiment of this application; Figure 3 This is an example diagram of an autonomous driving perception device based on cross-modal distillation according to an embodiment of this application; Figure 4 This is a schematic diagram of the vehicle structure provided in an embodiment of this application.

[0026] Among them, 10-autonomous driving perception device based on cross-modal distillation; 100-acquisition module, 200-analysis module, 300-modulation module; 401-memory, 402-processor, 403-communication interface. Detailed Implementation

[0027] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0028] The following description, with reference to the accompanying drawings, describes an autonomous driving perception method and apparatus based on cross-modal distillation according to embodiments of this application. Addressing the problems mentioned in the background section, this application provides an autonomous driving perception method based on cross-modal distillation. In this method, point cloud data and image data of the target vehicle are acquired. The preprocessed point cloud data is input into a pre-built teacher model, and the image data is input into a pre-built student model to extract corresponding point cloud BEV features, point cloud scene complexity index, and image features. Based on a preset weight allocation rule and the point cloud scene complexity index, attention weights corresponding to the image features and point cloud BEV features are dynamically allocated. The depth variance of the point cloud within the projection range corresponding to the superpixel of each image input to the IPV branch of the student model is calculated. Based on the depth variance, a preset depth is determined. The confidence filtering operation obtains the target point cloud confidence that meets the preset confidence requirements. Based on the target point cloud confidence, a preset super-point feature alignment operation is performed on superpixels to obtain point cloud aligned pairing features. Based on the point cloud aligned pairing features and attention weights, IPV view contrast distillation loss functions and BEV view contrast distillation loss functions are constructed. A corresponding total loss function is then constructed using these two functions. Based on the total loss function, the gradient of the student model's parameters is iteratively calculated and updated until the total loss function meets the preset convergence requirements, resulting in a trained student model. This trained student model is then used to perform corresponding autonomous driving perception operations. This application constructs a dual-model architecture of point cloud teacher and image student, and achieves knowledge transfer through operations such as superpixel pooling, feature alignment, and NCE loss calculation. It also incorporates dynamic weight adjustment and deep confidence filtering mechanisms to optimize model performance, thereby improving the accuracy of 3D knowledge transfer to image models and reducing the hardware cost of autonomous driving. This solves the problems in existing technologies, such as the inability of pure vision vehicles to meet high-precision perception requirements without LiDAR configuration, the inability of fixed weights to adjust the learning focus according to scene complexity, and the large noise interference in the mapping of the same image pixel.

[0029] Specifically, Figure 1 This is a flowchart of an autonomous driving perception method based on cross-modal distillation, provided as an embodiment of this application.

[0030] like Figure 1 As shown, this autonomous driving perception method based on cross-modal distillation includes the following steps: In step S101, point cloud data and image data of the target vehicle are acquired, and the preprocessed point cloud data is input into a pre-built teacher model, and the image data is input into a pre-built student model to extract the corresponding point cloud BEV features, point cloud scene complexity index and image features.

[0031] In this embodiment, a pre-trained 3D point cloud model (parameter frozen) can be used as the corresponding teacher model, and a 2D image model can be used as the student model. Point cloud data is input into the teacher model, and image data is input into the student model to extract the point cloud BEV features corresponding to the point cloud data, obtain the point cloud scene complexity index, and extract the image features corresponding to the image data.

[0032] Therefore, the embodiments of this application provide reliable data guidance and basis for subsequent dynamic view attention weight calculation and loss function construction through dual-model feature extraction operations.

[0033] Optionally, in one embodiment of this application, point cloud data and image data of the target vehicle are acquired, and the preprocessed point cloud data is input into a pre-built teacher model, and the image data is input into a pre-built student model to extract the corresponding point cloud BEV features, point cloud scene complexity index, and image features. This includes: using the teacher model to perform feature encoding and decoding operations on the point cloud data to obtain the corresponding point cloud feature map; performing global average pooling on the point cloud feature map to obtain the corresponding global feature vector; inputting the global feature vector into a preset fully connected layer to quantize the global feature vector into the corresponding point cloud scene complexity index; performing downsampling on the Z-axis of the point cloud feature map through a preset 3D convolutional layer to obtain the corresponding point cloud BEV features; using the IPV branch of the student model to extract features from the image data to generate the corresponding target dimension feature map; adjusting the number of channels in the target dimension feature map; and performing a preset pixel similarity clustering operation on the adjusted target dimension feature map to obtain image superpixel features in the image features; and inputting the target dimension feature map into the LSS (Lift Splat) of the BEV branch of the student model. The Shoot (dimensionality upscaling-projection-mapping) depth estimation head is used to obtain the corresponding depth map. Based on preset camera intrinsic and extrinsic parameters, the depth map is mapped to a 3D feature map, and depth-oriented pooling is performed on the 3D feature map to generate image BEV features in the image features.

[0034] In the specific implementation process, such as Figure 2 As shown, in the embodiments of this application, the point cloud data can first be preprocessed, and the preprocessed point cloud data (i.e., data in the format of N×4, where N is the number of points and 4 represents the x / y / z coordinates and reflectivity) can be input into the pre-trained SR-UNet point cloud 3D model (parameters are frozen to avoid 3D knowledge shift).

[0035] Secondly, the process of extracting point cloud features using a point cloud 3D model in this embodiment is as follows: 1. The input point cloud is feature encoded and decoded through the SR-UNet backbone network (containing 5 downsampling blocks and 5 upsampling blocks), and the output point cloud feature map is output (dimension is 256×64×64, 256 is the number of feature channels, and 64×64 is the feature map size). 2. Perform global average pooling on the point cloud feature map to obtain a global feature vector with dimension 1×256, and input it into a fully connected layer to quantize it into a scene complexity index S (value 0-1, calculated as S=0.4×D+0.3×O+0.3×S). p Where D is the target density; O is the occlusion rate; S p (Point cloud sparsity, all normalized to 0-1). 3. The Z-axis (depth direction) of the point cloud feature map is downsampled by 80 times through a 3D convolutional layer (3×3×3 convolutional kernel, stride 8, padding=1) and compressed into point cloud BEV features (dimension 256×100×100, 100×100 is the BEV grid size, corresponding to the actual spatial range of 50m×50m).

[0036] Furthermore, in this embodiment, the preprocessed RGB image (3×720×1280 dimensions, where 3 represents the color channels and 720×1280 represents the image resolution) can be input into a dual-branch network (IPV branch and BEV branch) built based on MobileNetV3-Large. The process of extracting image features through the dual-branch network in this embodiment is as follows: 1. IPV Branch: The MobileNetV3 backbone network extracts features from the image and outputs a feature map with dimensions of 128×180×320 (i.e., the target dimension feature map). After the number of channels is adjusted to 256 by a 1×1 convolutional layer, it is input into the SLIC superpixel clustering layer (500 superpixels, compactness 10) to cluster the feature map according to pixel similarity, aggregating it into image superpixel features with dimensions of 256×500. 2. BEV Branch: The 128×180×320 feature map output from the MobileNetV3 backbone is input to the LSS depth estimation head (containing three 3×3 convolutional layers), outputting a depth map of dimension 64×180×320 (64 represents the depth range, from 0-50m); combined with camera intrinsic and extrinsic parameters (intrinsic parameters...). f x =900、 f y =900、 u 0=640、 v 0=360, extrinsic rotation matrix R Translation vector TThe 2D features are mapped to 3D features (dimension 256×64×180×320), and the 3D features are then depthwise pooled through the BEVPool layer (Z-axis pooling kernel 8) to compress them into image BEV features of dimension 256×100×100.

[0037] Therefore, the embodiments of this application extract accurate point cloud BEV features and scene complexity index through the teacher model, and efficiently extract image superpixels and BEV features through the student model with two branches. This can adapt to different needs, unify the feature dimensions, facilitate fusion, balance accuracy and efficiency, and improve the reliability and practicality of feature extraction.

[0038] In step S102, based on the preset weight allocation rules and the point cloud scene complexity index, attention weights corresponding to image features and point cloud BEV features are dynamically allocated, and the depth value variance of the point cloud within the projection range corresponding to the superpixel of each image input to the IPV branch of the student model is calculated. Based on the depth value variance, a preset depth confidence filtering operation is performed to obtain the target point cloud confidence that meets the preset confidence requirements. Based on the target point cloud confidence, a preset super-point feature alignment operation is performed on the superpixel to obtain point cloud aligned pairing features.

[0039] Furthermore, embodiments of this application can quantify scene features (target density, occlusion rate, point cloud sparsity) extracted from the point cloud teacher model into a scene complexity index S (value 0-1, the larger the value, the more complex the scene), and then dynamically allocate distillation weights for IPV and BEV (i.e., distillation attention weights for IPV and BEV views) according to the index to ensure that the model focuses on learning the corresponding view in different scenarios. Secondly, embodiments of this application can calculate the depth variance of the point cloud within the image pixel projection range, calculate the point cloud confidence based on the depth variance for ambiguous regions, filter low-confidence noise, and retain only high-confidence point clouds to participate in feature mapping. Furthermore, embodiments of this application can perform superpixel pooling on image features (i.e., clustering pixels into superpixels and aggregating features), and perform superpoint pooling on point cloud features (grouping point clouds according to the image superpixel range and aggregating features) to obtain point cloud superpoint features, and then project the point cloud superpoint features onto the image superpixel plane to achieve spatial alignment.

[0040] Therefore, the embodiments of this application can achieve scene adaptive adjustment of IPV and BEV view distillation weights, so that the model can focus on learning the corresponding view under different complexity scenarios; in addition, the embodiments of this application can also improve the accuracy of 3D knowledge transfer to image models by filtering depth ambiguity noise in point cloud-image mapping.

[0041] Optionally, in one embodiment of this application, attention weights corresponding to image features and point cloud BEV features are dynamically allocated based on a preset weight allocation rule and a point cloud scene complexity index. The variance of the depth values ​​of the point cloud within the projection range corresponding to the superpixels of each image input to the IPV branch of the student model is calculated. Based on the depth value variance, a preset depth confidence filtering operation is performed to obtain a target point cloud confidence score that meets the preset confidence requirements. Based on the target point cloud confidence score, a preset superpixel feature alignment operation is performed to obtain point cloud aligned pairing features. This includes: acquiring the driving information of the target vehicle and determining the corresponding driving scenario based on the driving information; constructing a corresponding weight allocation rule based on the driving scenario; and assigning corresponding attention weights to image features and point cloud features based on the weight allocation rule and the point cloud scene complexity index. The pixel coordinates of the superpixels of each image in the IPV branch are transformed to the three-dimensional coordinate range of the point cloud coordinate system to determine... For each superpixel, a point cloud projection region is defined, and all point clouds within this region are traversed to obtain the depth value of each point cloud. The depth variance is calculated based on the depth values ​​of all point clouds. It is then determined whether the depth variance exceeds a preset depth threshold. If it does, the corresponding point cloud projection region is identified as a depth ambiguity region. The confidence scores of all point clouds within the depth ambiguity region are calculated and compared to a preset confidence threshold. Point clouds with confidence scores less than or equal to the preset threshold are filtered out to obtain target point cloud confidence scores greater than the preset threshold. Preset clustering is performed on the target point cloud confidence scores to obtain multiple groups of point clouds. The features of each group are then averaged and aggregated to generate multiple target point cloud super-point features. These multiple target point cloud super-point features are projected onto the image superpixel plane using preset camera intrinsic parameters, ensuring that each target point cloud super-point feature is spatially aligned with its corresponding image superpixel feature.

[0042] It should be noted that the embodiments of this application can construct corresponding weight allocation rules according to different vehicle driving scenarios, as detailed below: 1. Simple scenarios (S<0.3, such as highways): =0.7, ≈0.3-0.51, focusing on the IPV branch to improve the ability to identify small targets at a distance; 2. Complex scenarios (S>0.7, such as urban traffic congestion): =0.7, ≈0.3-0.49, focusing on the BEV branch, optimizing global perception in dense vehicle scenarios; 3. Moderately complex scenarios (0.3 ≤ S ≤ 0.7): It smoothly decreased from 0.7 to 0.3. The weight is smoothly increased from 0.51 to 0.7 to achieve adaptive weighting during scene transitions.

[0043] Secondly, the embodiments of this application can calculate the IPV view attention weight based on the weight allocation rules. and BEV view attention weight Among them, IPV view attention weight for The smaller S is (the simpler the scenario). Larger values ​​enhance local detail learning; BEV view attention weights for The larger S is, the more complex the scenario. The larger the size, the more it strengthens the learning of the overall layout.

[0044] It is understood that the dynamic weight adjustment mechanism of the embodiments of this application can flexibly allocate the focus of view learning according to the complexity of the driving scenario. For example, in complex scenarios such as urban congestion, the embodiments of this application can more accurately capture global spatial layout information to greatly optimize the target detection effect; in simple scenarios such as highways, the embodiments of this application can focus on local detailed features to effectively improve the recognition ability of small targets at a distance, thereby adapting to the perception needs of different scenarios.

[0045] Furthermore, in this embodiment, for each image superpixel (500 in total) in the IPV branch, the pixel coordinates of the superpixel are converted into a 3D coordinate range of the point cloud coordinate system through inverse camera extrinsic transformation, and the point cloud projection area corresponding to the superpixel is determined; subsequently, this embodiment can traverse all point clouds within the point cloud projection area and calculate the variance of the corresponding depth value (Z-axis coordinate). ,like If a region is identified as a deeply ambiguous region (e.g., overlapping vehicle projections) by setting a pre-defined confidence threshold, it is then classified as such.

[0046] Secondly, the embodiments of this application can be based on formula Calculate the confidence level of the point cloud in the ambiguous region. w The closer the value is to 1, the more concentrated the point cloud depth is, and the higher the reliability. Only the confidence level is retained. w High-confidence point clouds (i.e., target point cloud confidence) >0.5 are used to filter out low-confidence noisy point clouds (retaining approximately 75%-80% of the effective point clouds).

[0047] Furthermore, embodiments of this application group the filtered high-confidence point clouds according to the clustering range of image superpixels (each superpixel corresponds to a group of point clouds), and average aggregate the features of each group of point clouds (dimension is 256×N, where N is the number of point clouds in the group) to generate point cloud superpoint features with a dimension of 256×500.

[0048] Furthermore, in this embodiment, the point cloud superpoint features can be projected onto the image superpixel plane using camera intrinsic parameters, ensuring that each point cloud superpoint feature and the corresponding image superpixel feature are spatially aligned (alignment error ≤ 1 pixel), thereby providing spatial consistency assurance for subsequent IPV distillation.

[0049] Therefore, the embodiments of this application can accurately identify deep ambiguous regions in point cloud-image mapping based on the deep confidence filtering mechanism, and filter low-confidence noisy point clouds, reducing the interference of invalid information on model learning, thereby significantly improving the accuracy of semantic segmentation, especially in scenarios where mapping ambiguity is easily generated, such as vehicle overlap.

[0050] In step S103, based on point cloud alignment and pairing features and attention weights, an IPV view contrast distillation loss function and a BEV view contrast distillation loss function are constructed. Then, a corresponding total loss function is constructed using the IPV view contrast distillation loss function and the BEV view contrast distillation loss function. Based on the total loss function, the gradient of the model parameters of the student model is iteratively calculated and updated until the total loss function meets the preset convergence requirements, so as to obtain the trained student model. The trained student model is then used to perform the corresponding autonomous driving perception operation.

[0051] Subsequently, embodiments of this application can perform knowledge distillation of the IPV view by combining the aligned point cloud superpoint features and image superpixel features with NCE loss; secondly, embodiments of this application can also generate a depth map from the image features through the LSS depth estimation module, and map the 2D image features into 3D features by combining the camera intrinsic and extrinsic parameters, and then convert them into BEV features through BEVPool operation; the point cloud features are downsampled by 80 times along the Z-axis (depth direction) through 3D convolution and compressed into BEV features, and the two are subjected to knowledge distillation of the BEV view through NCE loss.

[0052] Furthermore, in this embodiment, an IPV view contrast distillation loss function and a BEV view contrast distillation loss function can be constructed, and a corresponding total loss function can be constructed in combination with a preset depth prediction loss function. The gradient of the model parameters of the student model can be iteratively calculated and updated based on the total loss function until the total loss function meets the preset convergence requirements, so as to complete the training of the student model and use the trained student model to perform the corresponding autonomous driving perception operation.

[0053] Therefore, the embodiments of this application can directly deploy the distilled pure image model on vehicles without LiDAR, without having to bear the high hardware cost of LiDAR, thereby significantly reducing the deployment threshold of autonomous driving systems. At the same time, the model inference speed can meet the real-time perception requirements of the vehicle, and the overall perception accuracy is close to the level of LiDAR combined with point cloud model, which can fully support the environmental perception tasks of L2+ level autonomous driving.

[0054] Optionally, in one embodiment of this application, an IPV view contrastive distillation loss function and a BEV view contrastive distillation loss function are constructed based on point cloud alignment pairing features and attention weights. A corresponding total loss function is then constructed using these two functions. This includes: constructing multiple IPV view positive sample data based on point cloud alignment pairing features, randomly sampling each IPV view positive sample data to obtain first random sample data, and constructing multiple IPV view negative sample data based on the first random sample data. An IPV view contrastive distillation loss function is then constructed based on the multiple IPV view negative sample data, the multiple IPV view positive sample data, and attention weights. Point cloud BEV features are paired with image BEV features to obtain corresponding paired features. Multiple BEV view positive sample data are constructed based on the paired features, and each BEV view positive sample data is randomly sampled to obtain corresponding second random sample data. Multiple BEV view negative sample data are then constructed based on the second random sample data. An BEV view contrastive distillation loss function is then constructed based on the multiple BEV view negative sample data, the multiple BEV view positive sample data, and attention weights.

[0055] In actual implementation, the embodiments of this application can be based on general distillation logic, with "teacher-student" feature alignment as the core. By constructing positive sample pairs (point cloud teacher features and image student features in the corresponding view) and negative sample pairs (non-corresponding features in the same batch), the student model learns the feature distribution of the teacher model using NCE loss (noise contrast estimation loss). The dynamic attention weight of the corresponding view is incorporated into the loss calculation to ensure efficient knowledge transfer in key scenarios.

[0056] First, for IPV view comparison distillation, embodiments of this application can use the super-point features of the aligned point cloud. Image superpixel features The pairing constructs multiple corresponding IPV view positive sample data, and the superpixel features of other images within the same batch are randomly sampled for each positive sample. Construct negative sample data for multiple IPV views, and combine them with the dynamic weights of the IPV views. By forcing image superpixel features to align with point cloud superpoint features through the NCE loss formula, the mathematical expression for the NCE loss (i.e., the IPV view contrastive distillation loss function) corresponding to this IPV view contrastive distillation is as follows:

[0057] Where M is the number of superpixels / superpoints; is the temperature coefficient (controlling the steepness of the feature distribution); K is the number of negative samples corresponding to each positive sample.

[0058] Secondly, for BEV view comparison distillation, point cloud BEV features can be used. Image BEV features Multiple BEV view positive sample data are constructed by pairing, and BEV features of other images within the same batch are randomly sampled based on each positive sample. Construct multiple negative sample data for BEV views and combine them with dynamic weights of BEV views. By forcing the BEV features of the image to align with the BEV features of the point cloud through the NCE loss formula, the mathematical expression of the NCE loss (i.e., the BEV view-to-view distillation loss function) corresponding to this BEV view-to-view distillation is as follows:

[0059] Where H × W is the BEV feature grid size; is the temperature coefficient for the BEV branch; K is the number of negative samples corresponding to each positive sample (consistent with the IPV branch).

[0060] Therefore, the embodiments of this application construct positive and negative sample pairs of IPV and BEV views, and construct a loss function by combining NCE loss and dynamic attention weights. This can enhance the student model's learning of the teacher model's feature distribution, force image features to align with point cloud features, and ensure efficient transfer of key scene knowledge, thereby improving the accuracy and effectiveness of the student model's feature learning.

[0061] Optionally, in one embodiment of this application, the mathematical expression for the total loss function is:

[0062] in, This represents the IPV view compared to the distillation loss function; This indicates the BEV view compared to the distillation loss function; This represents the depth prediction loss function; This represents the weighting coefficients of the depth prediction loss function.

[0063] It should be noted that the embodiments of this application construct the corresponding total loss function by fusing the IPV view contrast distillation loss function, the BEV view contrast distillation loss function, and the depth prediction loss function. The mathematical expression for the total loss function is:

[0064] in, This indicates the IPV view comparison distillation loss function (i.e., the IPV branch NCE loss, which incorporates dynamic weights). This reflects the alignment error between point cloud superpoint features and image superpixel features. This indicates the BEV view versus distillation loss function (i.e., the BEV branch NCE loss, which incorporates dynamic weights). (This reflects the alignment error between point cloud BEV features and image BEV features). This represents the depth prediction loss function; The weight coefficients of the depth prediction loss function are used to balance the importance of depth estimation accuracy and distillation loss, so as to ensure that depth information supports the BEV feature transformation.

[0065] In the embodiments of this application, the depth prediction loss function The deviation between the depth estimation result of the image student model and the true depth of the point cloud is quantified using L1 loss, and its mathematical expression is as follows:

[0066] Where B is the training batch size; This refers to the depth map dimensions. The predicted depth output by the image model; The true depth generated by projecting the point cloud onto the image plane.

[0067] Furthermore, during training, the total loss function can calculate the gradients of each parameter of the image student model (such as the convolutional weights of the backbone network, the superpixel clustering parameters, and the depth estimation head parameters) through the backpropagation algorithm, and update the parameters in conjunction with the Adam optimizer (with a preset learning rate strategy and weight decay). In addition, since the parameters of the point cloud teacher model are always frozen, 3D knowledge shift is avoided, ensuring that the total loss converges stably to the optimal value, and finally realizing the effective learning of 3D perceptual knowledge by the image model.

[0068] In summary, the embodiments of this application, through operations such as feature extraction, dynamic attention weight calculation, depth confidence filtering and feature alignment, dual-view comparison distillation, total loss optimization, model training and deployment, enable the point cloud teacher model to guide the image student model. Combined with the dynamic view attention mechanism and depth confidence filtering, it achieves efficient transfer of 3D knowledge to the 2D model, enabling the pure image model to have 3D perception capabilities close to those of the point cloud model, and is well applicable to vehicles without LiDAR.

[0069] According to the autonomous driving perception method based on cross-modal distillation proposed in this application, point cloud data and image data of the target vehicle are acquired. The preprocessed point cloud data is input into a pre-built teacher model, and the image data is input into a pre-built student model to extract corresponding point cloud BEV features, point cloud scene complexity index, and image features. Based on preset weight allocation rules and the point cloud scene complexity index, attention weights corresponding to image features and point cloud BEV features are dynamically allocated. The depth value variance of the point cloud within the projection range corresponding to the superpixel of each image input to the IPV branch of the student model is calculated. Based on the depth value variance, a preset depth confidence filtering operation is performed to obtain... The process involves obtaining a target point cloud confidence level that meets pre-set confidence requirements, and then performing pre-set super-point feature alignment on superpixels based on this confidence level to obtain point cloud aligned pairing features. Based on these point cloud aligned pairing features and attention weights, an IPV view contrast distillation loss function and a BEV view contrast distillation loss function are constructed. A corresponding total loss function is then constructed using these two functions. The gradient of the student model's parameters is iteratively calculated and updated based on the total loss function until it meets a pre-set convergence requirement, resulting in a trained student model. This trained student model is then used to perform corresponding autonomous driving perception operations. This application constructs a dual-model architecture of point cloud teacher and image student, and achieves knowledge transfer through operations such as superpixel pooling, feature alignment, and NCE loss calculation. It also incorporates dynamic weight adjustment and deep confidence filtering mechanisms to optimize model performance, thereby improving the accuracy of 3D knowledge transfer to image models and reducing the hardware cost of autonomous driving.

[0070] Secondly, with reference to the accompanying drawings, an autonomous driving perception device based on cross-modal distillation according to an embodiment of this application is described.

[0071] Figure 3 This is a block diagram of an autonomous driving perception device based on cross-modal distillation, according to an embodiment of this application.

[0072] like Figure 3 As shown, the autonomous driving perception device 10 based on cross-modal distillation includes: a feature extraction module 100, a feature alignment module 200, and a training module 300.

[0073] The feature extraction module 100 is used to acquire point cloud data and image data of the target vehicle, and input the preprocessed point cloud data into a pre-built teacher model and the image data into a pre-built student model to extract the corresponding point cloud BEV features, point cloud scene complexity index and image features.

[0074] The feature alignment module 200 is used to dynamically allocate attention weights corresponding to image features and point cloud BEV features based on preset weight allocation rules and point cloud scene complexity index, and calculate the depth value variance of the point cloud within the projection range corresponding to the superpixel of each image input to the IPV branch of the student model. Based on the depth value variance, a preset depth confidence filtering operation is performed to obtain the target point cloud confidence that meets the preset confidence requirements. Based on the target point cloud confidence, a preset super-point feature alignment operation is performed on the superpixel to obtain point cloud aligned pair features.

[0075] The training module 300 is used to construct IPV view contrast distillation loss function and BEV view contrast distillation loss function based on point cloud alignment and pairing features and attention weights. It also constructs the corresponding total loss function based on the IPV view contrast distillation loss function and the BEV view contrast distillation loss function. Based on the total loss function, the gradient of the model parameters of the student model is iteratively calculated and updated until the total loss function meets the preset convergence requirements to obtain the trained student model. The trained student model is then used to perform the corresponding autonomous driving perception operation.

[0076] Optionally, in one embodiment of this application, the feature extraction module 100 includes: a quantization unit, a downsampling unit, an adjustment unit, and a mapping unit.

[0077] The quantization unit is used to perform feature encoding and decoding operations on point cloud data using the teacher model to obtain the corresponding point cloud feature map, and to perform global average pooling on the point cloud feature map to obtain the corresponding global feature vector. The global feature vector is then input into a preset fully connected layer to quantize the global feature vector into the corresponding point cloud scene complexity index.

[0078] The downsampling unit is used to perform downsampling operations on the Z-axis of the point cloud feature map through a preset 3D convolutional layer to obtain the corresponding point cloud BEV features.

[0079] The adjustment unit is used to extract features from image data using the IPV branch of the student model to generate a corresponding target dimension feature map, adjust the number of channels in the target dimension feature map, and perform a preset pixel similarity clustering operation on the adjusted target dimension feature map to obtain image superpixel features in the image features.

[0080] The mapping unit is used to input the target dimension feature map into the LSS depth estimation head of the BEV branch of the student model to obtain the corresponding depth map, and based on the preset camera intrinsic and extrinsic parameters, to map the depth map into a 3D feature map, and to perform depth-direction pooling on the 3D feature map to generate the image BEV feature in the image features.

[0081] Optionally, in one embodiment of this application, the feature alignment module 100 includes: a weight allocation unit, a projection unit, a judgment unit, a filtering unit, a grouping unit, and an alignment unit.

[0082] The weight allocation unit is used to acquire the driving information of the target vehicle and determine the corresponding driving scenario based on the driving information. Based on the driving scenario, it constructs the corresponding weight allocation rule and assigns corresponding attention weights to the image features and point cloud features based on the weight allocation rule and the point cloud scene complexity index.

[0083] The projection unit is used to transform the pixel coordinates of the superpixels of each image in the IPV branch to the three-dimensional coordinate range of the point cloud coordinate system, so as to determine the point cloud projection area corresponding to each superpixel, and traverse all point clouds within the point cloud projection area to obtain the depth value of each point cloud, and calculate the depth value variance based on the depth values ​​of all point clouds.

[0084] The judgment unit is used to determine whether the depth value variance is greater than a preset depth value threshold. If it is greater than the preset depth value threshold, the corresponding point cloud projection area is determined to be a depth ambiguity area.

[0085] The filtering unit is used to calculate the confidence score of all point clouds in the deep ambiguity region, compare the confidence score with a preset confidence threshold, and filter the point clouds corresponding to confidence scores that are less than or equal to the preset confidence threshold in order to obtain the target point cloud confidence score that is greater than the preset confidence threshold.

[0086] The grouping unit is used to perform a preset clustering grouping operation on the confidence of the target point cloud to obtain multiple groups of point clouds, and to average and aggregate the features of each group of point clouds to generate multiple target point cloud super-point features.

[0087] The alignment unit is used to project multiple target point cloud superpoint features onto the image superpixel plane using preset camera intrinsic parameters, so that each target point cloud superpoint feature is spatially aligned with the corresponding image superpixel feature.

[0088] Optionally, in one embodiment of this application, the training module 300 includes: a first loss function construction unit and a second loss function construction unit.

[0089] The first loss function construction unit is used to construct multiple IPV view positive sample data based on point cloud alignment and pairing features, and to randomly sample each IPV view positive sample data to obtain first random sample data. It also constructs multiple IPV view negative sample data based on the first random sample data, and constructs an IPV view contrastive distillation loss function based on the multiple IPV view negative sample data, the multiple IPV view positive sample data, and attention weights.

[0090] The second loss function construction unit is used to pair point cloud BEV features with image BEV features to obtain corresponding paired features, and construct multiple BEV view positive sample data based on the paired features. It also randomly samples each BEV view positive sample data to obtain corresponding second random sample data, and constructs multiple BEV view negative sample data based on the second random sample data. Finally, it constructs a BEV view contrastive distillation loss function based on the multiple BEV view negative sample data, the multiple BEV view positive sample data, and attention weights.

[0091] Optionally, in one embodiment of this application, the mathematical expression for the total loss function is:

[0092] in, This represents the IPV view compared to the distillation loss function; This indicates the BEV view compared to the distillation loss function; This represents the depth prediction loss function; This represents the weighting coefficients of the depth prediction loss function.

[0093] It should be noted that the foregoing explanation of the autonomous driving perception method embodiment based on cross-modal distillation also applies to the autonomous driving perception device based on cross-modal distillation in this embodiment, and will not be repeated here.

[0094] The autonomous driving perception device based on cross-modal distillation proposed in this application includes a feature extraction module 100, used to acquire point cloud data and image data of the target vehicle, and input the preprocessed point cloud data into a pre-built teacher model and the image data into a pre-built student model to extract the corresponding point cloud BEV features, point cloud scene complexity index, and image features; and a feature alignment module 200, used to dynamically allocate attention weights corresponding to image features and point cloud BEV features based on preset weight allocation rules and point cloud scene complexity index, and calculate the depth value variance of the point cloud within the projection range corresponding to the superpixel of each image input to the IPV branch of the student model, so as to perform preset depth confidence based on the depth value variance. The system employs a depth-based filtering operation to obtain the target point cloud confidence score that meets pre-set confidence requirements. Based on this confidence score, it performs a pre-set super-point feature alignment operation on superpixels to obtain point cloud aligned pairing features. The training module 300 constructs IPV view contrast distillation loss functions and BEV view contrast distillation loss functions based on the point cloud aligned pairing features and attention weights. It then constructs a corresponding total loss function using these functions. Based on this total loss function, iteratively calculates and updates the model parameter gradients of the student model until the total loss function meets a pre-set convergence requirement, resulting in a trained student model. This trained student model is then used to perform corresponding autonomous driving perception operations. This application constructs a dual-model architecture of point cloud teacher and image student, and achieves knowledge transfer through operations such as superpixel pooling, feature alignment, and NCE loss calculation. It also incorporates dynamic weight adjustment and depth-based confidence filtering mechanisms to optimize model performance, thereby improving the accuracy of 3D knowledge transfer to image models and reducing the hardware cost of autonomous driving.

[0095] Figure 4 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include: The memory 401, the processor 402, and the computer program stored on the memory 401 and capable of running on the processor 402.

[0096] When the processor 402 executes the program, it implements the autonomous driving perception method based on cross-modal distillation provided in the above embodiments.

[0097] Furthermore, the vehicle also includes: Communication interface 403 is used for communication between memory 401 and processor 402.

[0098] The memory 401 is used to store computer programs that can run on the processor 402.

[0099] Memory 401 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0100] If the memory 401, processor 402, and communication interface 403 are implemented independently, then the communication interface 403, memory 401, and processor 402 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0101] Optionally, in a specific implementation, if the memory 401, processor 402, and communication interface 403 are integrated on a single chip, then the memory 401, processor 402, and communication interface 403 can communicate with each other through an internal interface.

[0102] Processor 402 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0103] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described autonomous driving perception method based on cross-modal distillation.

[0104] This application also provides a computer program product, including a computer program, which, when executed, is used to implement the above-described autonomous driving perception method based on cross-modal distillation.

[0105] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0106] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0107] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0108] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0109] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0110] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0112] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. An autonomous driving perception method based on cross-modal distillation, characterized in that, Includes the following steps: The point cloud data and image data of the target vehicle are acquired, and the preprocessed point cloud data is input into a pre-built teacher model, and the image data is input into a pre-built student model to extract the corresponding point cloud BEV features, point cloud scene complexity index and image features. Based on the preset weight allocation rules and the point cloud scene complexity index, attention weights corresponding to the image features and the point cloud BEV features are dynamically allocated. The depth value variance of the point cloud within the projection range corresponding to the superpixel of each image input to the IPV branch of the student model is calculated. Based on the depth value variance, a preset depth confidence filtering operation is performed to obtain the target point cloud confidence that meets the preset confidence requirements. Based on the target point cloud confidence, a preset superpixel feature alignment operation is performed on the superpixel to obtain point cloud aligned pairing features. Based on the point cloud alignment and pairing features and the attention weights, an IPV view contrast distillation loss function and a BEV view contrast distillation loss function are constructed. A corresponding total loss function is then constructed using the IPV view contrast distillation loss function and the BEV view contrast distillation loss function. The gradient of the model parameters of the student model is iteratively calculated and updated based on the total loss function until the total loss function meets the preset convergence requirement, so as to obtain the trained student model. The trained student model is then used to perform the corresponding autonomous driving perception operation.

2. The method according to claim 1, characterized in that, The process of acquiring point cloud data and image data of the target vehicle, inputting the preprocessed point cloud data into a pre-built teacher model, and inputting the image data into a pre-built student model to extract corresponding point cloud BEV features, point cloud scene complexity index, and image features includes: The teacher model is used to perform feature encoding and decoding operations on the point cloud data to obtain the corresponding point cloud feature map. The point cloud feature map is then subjected to global average pooling to obtain the corresponding global feature vector. The global feature vector is then input into a preset fully connected layer to quantize the global feature vector into the corresponding point cloud scene complexity index. The Z-axis of the point cloud feature map is downsampled by a preset 3D convolutional layer to obtain the corresponding point cloud BEV features. The IPV branch of the student model is used to extract features from the image data to generate a corresponding target dimension feature map. The number of channels in the target dimension feature map is adjusted, and a preset pixel similarity clustering operation is performed on the adjusted target dimension feature map to obtain the image superpixel features in the image features. The target dimension feature map is input into the LSS depth estimation head of the BEV branch of the student model to obtain the corresponding depth map. Based on the preset camera intrinsic and extrinsic parameters, the depth map is mapped into a three-dimensional feature map, and depth-oriented pooling is performed on the three-dimensional feature map to generate the image BEV feature in the image features.

3. The method according to claim 2, characterized in that, The process dynamically allocates attention weights corresponding to the image features and the point cloud BEV features based on preset weight allocation rules and the point cloud scene complexity index. It also calculates the depth variance of the point cloud within the projection range corresponding to the superpixels of each image input to the student model's IPV branch. Based on this depth variance, a preset depth confidence filtering operation is performed to obtain a target point cloud confidence that meets preset confidence requirements. Finally, based on the target point cloud confidence, a preset superpixel feature alignment operation is performed on the superpixels to obtain point cloud aligned pairing features, including: The driving information of the target vehicle is obtained, and the corresponding driving scenario is determined based on the driving information. Based on the driving scenario, a corresponding weight allocation rule is constructed, and based on the weight allocation rule and the point cloud scene complexity index, corresponding attention weights are assigned to the image features and the point cloud features. The pixel coordinates of the superpixels of each image in the IPV branch are transformed to the three-dimensional coordinate range of the point cloud coordinate system to determine the point cloud projection area corresponding to each superpixel. All point clouds within the point cloud projection area are traversed to obtain the depth value of each point cloud, and the depth value variance is calculated based on the depth values ​​of all point clouds. Determine whether the variance of the depth value is greater than a preset depth value threshold. If it is greater than the preset depth value threshold, determine that the corresponding point cloud projection region is a depth ambiguity region. Calculate the confidence score of all point clouds within the deep ambiguity region, compare the confidence score with a preset confidence threshold, and filter out point clouds with confidence scores less than or equal to the preset confidence threshold to obtain the confidence score of the target point cloud greater than the preset confidence threshold. The confidence scores of the target point cloud are subjected to a preset clustering grouping operation to obtain multiple groups of point clouds, and the features of each group of point clouds are averaged and aggregated to generate multiple target point cloud superpoint features. The superpoint features of the multiple target point clouds are projected onto the superpixel plane of the image using preset camera intrinsic parameters, so that each superpoint feature of the target point cloud is spatially aligned with the corresponding superpixel feature of the image.

4. The method according to claim 3, characterized in that, Based on the point cloud alignment and pairing features and the attention weights, an IPV view contrastive distillation loss function and a BEV view contrastive distillation loss function are constructed. Then, using the IPV view contrastive distillation loss function and the BEV view contrastive distillation loss function, a corresponding total loss function is constructed, including: Based on the point cloud alignment and pairing features, multiple IPV view positive sample data are constructed, and each IPV view positive sample data is randomly sampled to obtain first random sample data. Multiple IPV view negative sample data are constructed based on the first random sample data. Based on the multiple IPV view negative sample data, the multiple IPV view positive sample data, and the attention weight, the IPV view contrastive distillation loss function is constructed. The point cloud BEV features are paired with the image BEV features to obtain corresponding paired features. Multiple BEV view positive sample data are constructed based on the paired features. Each BEV view positive sample data is randomly sampled to obtain corresponding second random sample data. Multiple BEV view negative sample data are constructed based on the second random sample data. The BEV view contrastive distillation loss function is constructed based on the multiple BEV view negative sample data, the multiple BEV view positive sample data, and the attention weight.

5. The method according to claim 4, characterized in that, The mathematical expression for the total loss function is: in, This indicates the IPV view compared to the distillation loss function; This indicates the BEV view compared to the distillation loss function; This represents the depth prediction loss function; This represents the weighting coefficients of the depth prediction loss function.

6. An autonomous driving perception device based on cross-modal distillation, characterized in that, include: The feature extraction module is used to acquire point cloud data and image data of the target vehicle, and input the preprocessed point cloud data into a pre-built teacher model, and input the image data into a pre-built student model to extract the corresponding point cloud BEV features, point cloud scene complexity index and image features. The feature alignment module is used to dynamically allocate attention weights corresponding to the image features and the point cloud BEV features based on preset weight allocation rules and the point cloud scene complexity index, and calculate the depth value variance of the point cloud within the projection range corresponding to the superpixel of each image input to the student model IPV branch. Based on the depth value variance, a preset depth confidence filtering operation is performed to obtain the target point cloud confidence that meets the preset confidence requirements. Based on the target point cloud confidence, a preset superpixel feature alignment operation is performed on the superpixel to obtain point cloud aligned pair features. The training module is used to construct IPV view contrast distillation loss function and BEV view contrast distillation loss function based on the point cloud alignment and pairing features and the attention weights. It also constructs a corresponding total loss function based on the IPV view contrast distillation loss function and the BEV view contrast distillation loss function. The module iteratively calculates and updates the model parameter gradient of the student model based on the total loss function until the total loss function meets the preset convergence requirement, so as to obtain the trained student model. The trained student model is then used to perform the corresponding autonomous driving perception operation.

7. The apparatus according to claim 6, characterized in that, The feature extraction module includes: The quantization unit is used to perform feature encoding and decoding operations on the point cloud data using the teacher model to obtain the corresponding point cloud feature map, and to perform global average pooling on the point cloud feature map to obtain the corresponding global feature vector. The global feature vector is then input into a preset fully connected layer to quantize the global feature vector into the corresponding point cloud scene complexity index. The downsampling unit is used to perform a downsampling operation on the Z-axis of the point cloud feature map through a preset 3D convolutional layer to obtain the corresponding point cloud BEV features. The adjustment unit is used to extract features from the image data using the IPV branch of the student model to generate a corresponding target dimension feature map, adjust the number of channels of the target dimension feature map, and perform a preset pixel similarity clustering operation on the adjusted target dimension feature map to obtain the image superpixel features in the image features. The mapping unit is used to input the target dimension feature map into the LSS depth estimation head of the BEV branch of the student model to obtain the corresponding depth map, and based on the preset camera intrinsic and extrinsic parameters, map the depth map into a three-dimensional feature map, and perform depth-direction pooling on the three-dimensional feature map to generate the image BEV features in the image features.

8. A vehicle, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, the processor executing the program to implement the autonomous driving perception method based on cross-modal distillation as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the autonomous driving perception method based on cross-modal distillation as described in any one of claims 1-5.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the autonomous driving perception method based on cross-modal distillation as described in any one of claims 1-5.

Images