Knowledge distillation-based lightweight vehicle-mounted multi-source sensing data fusion method

Abstract

The invention discloses a knowledge distillation-based lightweight vehicle-mounted multi-source perception data fusion method, which comprises the following steps of: firstly, constructing a multi-modal teacher model, respectively processing visible light, infrared and radar data through a feature extraction network, and adaptively calculating associated weights with infrared and radar features by utilizing an attention mechanism taking visible light as query; and deep fusion of cross-modal features is realized. Then, a lightweight student model is designed, efficient networks such as MobileNetV3, weight sharing and lightweight PointNet are adopted for feature extraction, and feature fusion is achieved through depth separable convolution and a channel attention module; and finally, through a dynamic modal mask knowledge distillation technology, the fusion ability of the teacher model is migrated to the student model, so that the calculation complexity is significantly reduced while high precision is maintained. According to the method, the contradiction between the high computing power demand, the real-time performance and the reliability of the existing fusion method on the vehicle-mounted platform is effectively solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent driving embedded perception system, and particularly relates to a lightweight vehicle-mounted multi-source perception data fusion method based on knowledge distillation. BACKGROUND

[0002] Multi-source data fusion of vehicle-mounted perception information has become one of the key technologies to improve the environmental perception capability of vehicles. In the intelligent driving system, in order to realize comprehensive perception of the surrounding environment, it is usually necessary to integrate multiple sensors such as cameras, laser radars, millimeter wave radars, ultrasonic sensors, etc. to make up for the shortcomings of single sensor in terms of perception range, accuracy and robustness. However, the data collected by different sensors has different characteristics. For example, the camera can provide rich visual information, but it is easily affected by light changes and occlusions; the laser radar can provide high-precision three-dimensional point cloud data, but the cost is high and it is easily limited by weather conditions; the millimeter wave radar can effectively detect long-distance targets, but it is difficult to obtain detailed shape information of the targets.

[0003] In order to effectively integrate these heterogeneous data, current researchers have proposed various fusion strategies, mainly including data-level fusion, feature-level fusion and decision-level fusion. Among them, data-level fusion directly fuses the original sensor data, although it can retain more original information, but the requirement for computing resources is extremely high, which is difficult to meet the real-time demand; feature-level fusion extracts feature vectors after data preprocessing for fusion, which is suitable for feature matching of different modal data, but part of the original data information may be lost; decision-level fusion fuses the preliminary perception results of each sensor to improve the robustness and accuracy of the overall system, which is currently more common in commercial applications. In recent years, with the development of deep learning technology, multi-source perception data fusion methods based on neural networks have attracted widespread attention. For example, Faster R-CNN, VoxelNet, etc. These methods have achieved good results in specific scenarios, but still face many challenges in practical applications, such as high computing power demand, data domain bias, resolution conflict, and insufficient use of time information.

[0004] In order to cope with these challenges, researchers have begun to explore lightweight and efficient perception fusion methods, although some progress has been made, but there are still some problems to be solved. First, existing methods often have difficulty in fully exploiting the complementary advantages between different sensors when dealing with multi-modal data, resulting in limited accuracy and robustness of the perception results. Second, due to the limitation of computing resources, many efficient fusion algorithms are difficult to deploy in resource-constrained vehicle-mounted environments, affecting the real-time performance and reliability of the system. Finally, how to reduce the computational complexity and energy consumption of the system while ensuring the accuracy of perception is still the focus and difficulty of current research. SUMMARY

[0005] Therefore, the application provides a lightweight vehicle-mounted multi-source perception data fusion method based on knowledge distillation, which can effectively integrate multi-modal data, significantly reduce the computing cost while ensuring the perception accuracy.

[0006] The technical scheme of the application is as follows: A lightweight vehicle-mounted multi-source perception data fusion method based on knowledge distillation comprises the following steps: A multi-modal teacher model is constructed to synchronously process visible light, infrared and radar data, obtain feature representations of each modality through respective feature extraction networks, and interact with infrared and radar features based on an attention mechanism, calculate correlation weights, and then generate robust fusion features through weighted fusion by taking visible light features as queries; A lightweight student model is designed, which uses a lightweight network with higher computing efficiency to extract features of each modality, and realizes effective fusion of multi-modal features through feature splicing, depth separable convolution and channel attention module; A dynamic mask knowledge distillation technology is used, a dynamic mask is independently generated for each modality of each training sample and applied to the input of the student model, the joint supervision of the teacher model and the student model at the feature level and the output level is constructed, the feature alignment loss, the knowledge distillation loss and the task loss are calculated, and the student model parameters are optimized by the total loss; The trained student model is used for real-time fusion and target recognition of multi-sensor data in a vehicle-mounted environment.

[0007] Further, the feature extraction and fusion in the multi-modal teacher model specifically comprises: The visible light image and the infrared image are respectively input into a feature extraction backbone network to output corresponding feature tensors; the radar point cloud data is input into a point cloud backbone network to output a bird's eye view feature tensor; The visible light feature is projected into a query vector through a learnable projection matrix, and the infrared feature and the radar feature are respectively projected into a key vector and a value vector of each modality; The scaled dot product of the query vector and the transpose of each key vector is calculated, and after Softmax normalization, the correlation weight matrix of the visible light feature and the infrared and radar features is obtained; The correlation weight matrix is used to respectively perform weighted aggregation on the spatial dimensions of the infrared value vector and the radar value vector, add the aggregated features to obtain cross-modal fusion features.

[0008] Further, the feature extraction and fusion in the lightweight student model specifically comprises: The visible light data adopts a MobileNetV3-Small network for feature extraction, the infrared data adopts a lightweight module sharing weights with the visible light network for processing, and the radar data adopts a lightweight PointNet network based on a maximum pooling operation for feature extraction; The extracted three-modal features are spliced along the channel dimension; The spliced features are sequentially subjected to a depth separable convolution, a channel attention weighting based on a compression excitation mechanism, and a 1x1 convolution dimension reduction operation, and finally the fused features are output.

[0009] Further, in the dynamic mask knowledge distillation technology, the dynamic mask is independently generated for each modality of visible light, infrared, and radar, and the mask value is a binary variable subject to Bernoulli distribution; the feature alignment loss is measured by a maximum mean difference method to measure the distribution difference between the student model and the teacher model at the specified intermediate layer feature map; and the knowledge distillation loss is the KL divergence between the softened teacher and student prediction distributions by a temperature parameter.

[0010] Further, the task loss is composed of a Focal Loss classification loss and a Smooth L1 regression loss; and the total loss is a weighted sum of the feature alignment loss, the knowledge distillation loss, and the task loss, and the student model parameters are optimized by a gradient descent method.

[0011] Advantages 1. The present application migrates the high-precision fusion capability of a complex teacher model to a lightweight student model through a "teacher-student" knowledge distillation framework. The student model adopts structures such as MobileNetV3, weight sharing, and lightweight PointNet, and uses efficient modules such as depth separable convolution and channel attention in the fusion stage, which greatly reduces the computational complexity and parameter quantity of the model. This enables the present application to efficiently run on a resource-limited vehicle-mounted computing platform while ensuring a perception accuracy close to that of the teacher model, meeting the stringent real-time requirements of intelligent driving systems.

[0012] 2. The present application enhances the effectiveness of multi-modal feature fusion, and fully utilizes the complementary advantages of sensors An attention mechanism-based cross-modal feature fusion method is innovatively designed. In the teacher model, the visible light features are used as queries to dynamically calculate the correlation weights with the infrared and radar features, thereby selectively and adaptively fusing the complementary information of different modalities. In the student model, the features are spliced and then subjected to secondary fusion optimization by efficient depth separable convolution + attention mechanism. This multi-level, adaptive fusion strategy can more fully exploit the synergistic effect between heterogeneous data, effectively improving the perception ability and robustness of the target in complex, occluded, or poor lighting environments.

[0013] 3.The application improves the fault tolerance and stability of the system to sensor failure and abnormal conditions By introducing a dynamic modal mask mechanism in the knowledge distillation stage, the scenarios of single or multiple sensor data missing or failure are simulated. The student model is forced to learn to extract key information from the remaining valid modalities and make reliable inferences even under incomplete input. This training strategy greatly enhances the adaptability and performance robustness of the final deployed model under fault conditions, ensuring higher reliability of the vehicle-mounted perception system in real and variable driving environments. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The teacher model-student model overall flowchart of the application.

[0015] Figure 2 The teacher model training schematic of the application.

[0016] Figure 3 The student model dynamic mask distillation flowchart of the application. DETAILED DESCRIPTION

[0017] As Figure 1 shown, the knowledge distillation-based lightweight vehicle-mounted multi-source perception data fusion method involves three main modules: teacher model, student model, and dynamic mask module. The student model is an optimized recognition network based on the structure and parameters of the teacher model. For example, in the teacher model, the feature extraction of visible light and infrared data uses the Swin-Transformer Tiny or YOLO structure, and the radar data uses the PointFormer network. The student model uses a lightweight feature extraction network based on the teacher model, such as MobileNetV3-Small structure for infrared and visible light data, and PointNet network with MaxPool operation for radar data. Through the above operations, the computational power requirement of the recognition network on the vehicle-mounted physical equipment in practical applications can be maximally reduced. Although the recognition accuracy of the student model is reduced, the reduced recognition accuracy is corrected through parameter sharing and knowledge distillation of the teacher model, while ensuring the recognition efficiency.

[0018] The teacher model training process based on vehicle-mounted heterogeneous sensor data involves Figure 2 As shown in the figure, the training of the teacher model is the basis for multi-source sensor data fusion. The application extracts features from visible light and infrared radar data collected by vehicle-mounted sensors, combines attention and feature weighting fusion mechanisms, and realizes feature fusion of vehicle-mounted multi-source heterogeneous information. The implementation logic of the application is as follows: Step 1: Multi-modal teacher model construction Step 1.1: Multi-source sensor data preprocessing ① Obtain the synchronized raw data of the vehicle-mounted visible light sensor, infrared sensor, and millimeter wave radar; ② Perform size normalization processing on the visible light image, adjust the resolution to 640x640 pixels, and perform RGB channel standardization; ③ Perform linear mapping of the radiation value on the infrared image, convert the thermal radiation intensity value to the [0, 255] grayscale range, and align to 640x640 resolution through bilinear interpolation; ④ Perform voxelization processing on the millimeter wave radar point cloud, construct a three-dimensional grid with a cube side length of 0.1 meters, generate a bird's eye view (BEV) feature map, and the size is fixed at 20x20x256.

[0019] Step 1.2: Multi-modal data feature extraction ① Input the visible light image into the feature extraction backbone network, such as Swin-Transformer Tiny or YOLO, and output a feature tensor with dimensions of 1024x7x7 “vis” is the abbreviation of “visible” and represents the visible light (visible light) modality; ② Input the infrared image into the same feature extraction network, and output a feature tensor with dimensions of 1024x7x7 “ir” is the abbreviation of “infrared”; ③ Input the radar BEV feature map into the point cloud data backbone network, such as PointFormer, and output a feature tensor with dimensions of 256x20x20 .

[0020] Step 1.3: Cross-modal feature fusion ① Perform feature projection through a learnable parameter matrix, where the Query vector, infrared Key vector, radar Key vector, infrared Value vector, and radar Value vector are calculated as follows:

[0021]

[0022]

[0023]

[0024]

[0025] where Q represents Query Vector, which is projected from the visible light feature tensor, used for subsequent calculation of association weight with other modalities, is a learnable projection matrix, obtained through model training, which maps the visible light feature tensor to the "query" feature space. represents the Key Vector for Infrared of the infrared modality, which is used to calculate the association weight with the Query Vector. is a learnable projection matrix dedicated to the infrared modality, which maps the infrared feature tensor to the "key" feature space. represents the Key Vector for Radar of the radar modality, which is also used to calculate the association weight with the Query Vector. represents the Value Vector for Infrared of the infrared modality, which is the "information carrier" for subsequent weighted aggregation. is a learnable projection matrix dedicated to the infrared modality, which maps the infrared feature tensor to the "value" feature space. and have the same meaning as above.

[0026] ② Based on the Query Vector and the Key Vector transpose matrix, the association weight of the visible light feature with other modalities is calculated:

[0027]

[0028] where, represents the association weight matrix of the visible light feature and the infrared feature, which is used to measure the association degree of each position of the visible light feature and the infrared feature. represents the association weight matrix of the visible light feature and the radar feature, which is used to measure the association degree of each position of the visible light feature and the radar feature. Softmax is a standard classifier, which is an activation function, and its role is to normalize the calculation results to a probability distribution between 0 and 1, ensuring that the sum of the weights is 1, which is convenient for subsequent weighted aggregation. : is a scaling factor (256 is the dimension of the projected feature, i.e. "head dimension"), which is used to alleviate the problem of gradient vanishing caused by excessive values after matrix multiplication, and to make the distribution of attention weights more uniform.

[0029] ③ Based on the association weight, the multi-modal data features are weighted and fused:

[0030]

[0031]

[0032] wherein represents the infrared feature aggregation, i.e. the infrared fusion feature at position weighted by the correlation weight. is the correlation weight between visible light and infrared (from the matrix in the previous step), representing the correlation strength between the visible light position and the infrared position . is the value vector of the infrared modality, the infrared value feature at position . has the same meaning as above, representing the radar feature aggregation. represents the initial fusion feature.

[0033] Step 1.4: Fusion feature normalization processing

[0034]

[0035]

[0036] wherein , represents the channel dimension mean and variance, , are learnable parameters.

[0037] The above step is characterized by generating Query vector, Key vector and Value vector through projection matrix, calculating scaled dot product attention weight, and outputting fusion feature after weighted aggregation and layer normalization.

[0038] Step 2: Lightweight child model design Step 2.1: Backbone network structure optimization ① Perform convolution layer channel compression on the visible light data feature network, for example, use MobileNetV3-Small structure as follows:

[0039] wherein is the 640x640x3 normalized visible light data; ② Perform structure optimization on the infrared data feature network based on weight sharing of the visible light network;

[0040] wherein , is the pre-training weight of the visible light backbone network, Hard-Swish activation function, is 640x640x1 normalized infrared data; ③ The radar data feature network adopts structural compression, for example, using max pooling PointNet network:

[0041] wherein MLP is a fully connected neural network, and the structure is , is a spatial interpolation layer, which converts the point cloud feature into a BEV grid, is 200x200x3 normalized point cloud data.

[0042] Step 2.2: Deep separable feature fusion ① Visible light, infrared, and radar multi-modal data features are spliced, and three modal features are spliced along the channel dimension, with an output dimension of 256;

[0043] wherein Concat represents combining the feature arrays of multi-modal data into an array ; ② Spatial features are extracted through 3x3 convolution operations, and the group number is equal to the input channel number 256. The fusion mode is output point by point through 1x1 convolution.

[0044]

[0045] wherein DSC represents a deep separable convolution method; ③ The fusion mode is compressed along the spatial dimension to obtain statistical features :

[0046] ④ Then, the importance weight of each channel is generated based on the attention mechanism :

[0047] wherein is used for full connection dimension reduction from 128 to 32, is used for full connection dimension increase from 32 to 128; then, feature recalibration is performed through feature channel multiplication, to enhance key features and suppress noise:

[0048] Step 2.3: Feature output transformation ⑤ Based on one-dimensional convolution, the 128 channels are compressed to 64 channels, to further reduce the subsequent calculation amount:

[0049] Step 2 is characterized by a weight-shared infrared-visible light dual-branch processing architecture, and a fusion module based on deep separable convolution and compressed attention.

[0050] Step 3: Student model dynamic mask distillation On the basis of the lightweight student model (step 2 design), the fusion ability of the teacher model is transferred to the student model through knowledge distillation technology, as shown in Figure 3 .

[0051] Step 3.1: Forward propagation of teacher model ① Load multi-modal samples from the vehicle sensor perception dataset synchronously:

[0052] Wherein B is the size of the data obtained in each batch, Y represents the data label; ② Based on the multi-modal teacher model constructed in step 1, predict the multi-modal data recognition result:

[0053] Wherein Tech represents the constructed teacher model, represents the intermediate feature map of the 3-5 convolution layers of the teacher model, represents the prediction result; Step 3.2: Dynamic mask generation and application ① For each modality, independently generate a dynamic mask using a binary variable:

[0054] Wherein the binary variable The value can be changed according to the needs in actual application, for example The probability is 0.5, The probability is 0.5; ② Cover the dynamic mask to the sample of the student model:

[0055] Wherein represents the student model data sample after mask covering; Step 3.3: Student model inference ① Based on the forward calculation of the student model constructed in step 2, predict the multi-modal data recognition result:

[0056] wherein represents the constructed student model, corresponding to the 3-5 layer convolutional features of the teacher model, is the student model prediction result; ② Calculate the feature alignment loss function based on student feature similarity, teacher feature similarity, and student-teacher cross similarity:

[0057] Step 3.4: Knowledge distillation ① Soften the teacher prediction result:

[0058] wherein The value can be set according to actual needs, for example

[0059] ② Classify the student prediction result:

[0060] ③ Calculate the knowledge distillation loss:

[0061] Step 3.5: Calculate the multi-modal recognition task loss ① Classification loss calculation:

[0062] wherein , The parameters can be set according to actual needs, for example ; ② Regression loss calculation:

[0063] wherein represents the predicted value; ③ Calculate the task loss based on classification and regression loss:

[0064] Step 3.6: Backpropagation and parameter optimization ① Calculate the loss function weighted result:

[0065] ② Student model gradient calculation:

[0066] ③ Gradient descent updates student model parameters

[0067] Step 3.7: Iterative training ① Traverse all training batches (1 epoch) ② Repeat to the maximum epoch (default 50) ③ Stop training when the validation set mAP does not improve for 5 rounds in a row ④ Test the validation set performance every epoch ⑤ Save the best model parameters of the student model:

[0068] Step three is characterized in that a modal mask is independently generated for each training sample, and the data after the mask is input into the student model for inference; calculate the feature alignment loss, knowledge distillation loss and task loss; dynamically weight the three loss parts according to the training period.

[0069] Step 4: Real-time fusion recognition of vehicle-mounted multi-sensor data based on the optimal student model.

[0070] To sum up, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A lightweight vehicle-mounted multi-source perception data fusion method based on knowledge distillation, characterized in that, include: A multimodal teacher model is constructed to simultaneously process visible light, infrared, and radar data. Feature representations of each modality are obtained through their respective feature extraction networks. Based on the attention mechanism, visible light features are used as queries and interact with infrared and radar features respectively. The association weights are calculated and then weighted and fused to generate robust fused features. Design a lightweight student model that uses a computationally efficient lightweight network to extract features from various modalities, and achieves effective fusion of multimodal features through feature concatenation, depthwise separable convolution, and channel attention modules. The dynamic mask knowledge distillation technique is adopted. By generating dynamic masks independently for each modality of each training sample and applying them to the input of the student model, joint supervision between the teacher model and the student model at the feature level and the output level is constructed. Feature alignment loss, knowledge distillation loss and task loss are calculated, and the student model parameters are optimized with the total loss. The trained student model was used for real-time fusion of multi-sensor data and target recognition in an in-vehicle environment.

2. The method according to claim 1, characterized in that, The feature extraction and fusion in the multimodal teacher model specifically includes: Visible light images and infrared images are input into the feature extraction backbone network, and the corresponding feature tensors are output; radar point cloud data are input into the point cloud backbone network, and the bird's-eye view feature tensor is output. The visible light features are projected into a query vector using a learnable projection matrix, while the infrared and radar features are projected into their respective key and value vectors, respectively. Calculate the scaled dot product of the query vector and the transpose of each key vector, and after Softmax normalization, obtain the association weight matrix of visible light features and infrared and radar features. The infrared value vector and the radar value vector are weighted and aggregated in terms of spatial dimension using the correlation weight matrix, and the aggregated features are added together to obtain cross-modal fusion features.

3. The method according to claim 2, characterized in that, The feature extraction and fusion in the lightweight student model specifically includes: Visible light data is processed using the MobileNetV3-Small network for feature extraction, infrared data is processed using a lightweight module that shares weights with the visible light network, and radar data is processed using a lightweight PointNet network based on max pooling operations for feature extraction. The extracted trimodal features are concatenated along the channel dimension; The concatenated features are sequentially subjected to depthwise separable convolution, channel attention weighting based on compression excitation mechanism, and 1×1 convolution dimensionality reduction operation to output the final fused features.

4. The method according to claim 2 or 3, characterized in that, In the dynamic mask knowledge distillation technique, the dynamic mask is generated independently for each mode of visible light, infrared, and radar, and the mask value is a binary variable that follows a Bernoulli distribution; the feature alignment loss measures the distribution difference between the student model and the teacher model on the specified intermediate layer feature map through the maximum mean difference method; the knowledge distillation loss is the KL divergence between the teacher and student prediction distributions after being softened by the temperature parameter.

5. The method according to claim 1, characterized in that, The task loss consists of Focal Loss classification loss and Smooth L1 regression loss; the total loss is a weighted sum of feature alignment loss, knowledge distillation loss and task loss, and the student model parameters are optimized using gradient descent.

Images