Cross-modal foreground distillation method for multi-view three-dimensional target detection

Abstract

The invention discloses a cross-modal foreground distillation method for multi-view three-dimensional target detection, and belongs to the technical field of automatic driving perception. According to the method, a laser radar or a multi-modal three-dimensional target detection model is used as a teacher model, a multi-view aerial view BEV perception three-dimensional target detection model is used as a student model, and the model performance is improved through multi-scale feature distillation; a space attention mechanism and a foreground mask module are introduced, and the student model can simulate feature responses of the teacher model in different space positions under multiple scales, focus on a foreground target and suppress background noise; the adaptive module ensures the consistency of the multi-scale features in feature dimensions and spatial resolution, and finally, the difference between the features and regression output is measured through loss functions such as mean square error and KL divergence, so that efficient cross-modal and multi-scale foreground feature distillation is realized. Finally, the multi-view three-dimensional target detection precision and efficiency are improved, and meanwhile, the distillation requirements of multi-scale features are fully considered.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving perception technology in computer vision, and in particular to a cross-modal foreground distillation method for multi-view three-dimensional target detection. Background Art

[0002] In recent years, multi-view 3D target detection has attracted much attention in the field of autonomous driving perception and has a wide range of application scenarios. Among them, camera images have dense color and texture information. The 3D detection model based on multi-view bird's eye view (BEV) perception has the advantages of low cost and easy deployment, and has been favored by industry and academia. Especially in complex urban road environments, rich visual information is more conducive to scene understanding and target recognition, which helps the autonomous driving system make accurate decisions.

[0003] Compared with the 3D detection model based on LiDAR, the 3D target detection model based on multi-view BEV perception can achieve good accuracy, but it is difficult to accurately infer depth and other 3D geometric information, resulting in its performance lagging behind the 3D target detection model based on LiDAR. At the same time, 3D target detection based on LiDAR usually has high complexity, large consumption of computing resources, low operating efficiency, and is difficult to support industrial deployment. This is mainly because it is a huge challenge to construct 3D information from 2D images, and complex neural networks are required to learn and reason about this mapping relationship. An intuitive solution is to improve the operating efficiency of the model by compressing the model and reducing the number of model parameters. However, simple compression models often lead to performance degradation. In this case, knowledge distillation, as an effective model compression and acceleration technology, can not only significantly improve the operating efficiency of small models, but also maintain or approach the performance of the original large model while compressing the model, showing great potential.

[0004] Knowledge distillation allows a large model (teacher model) to guide the learning process of a small model (student model), so that the small model can achieve performance close to that of the large model. This method can not only effectively compress the model size, but also maintain the detection accuracy of the model to a certain extent. By using the lidar model as a teacher model to guide the learning of the visual model, the visual model can obtain the precise three-dimensional perception capability of the lidar while maintaining the advantages of low cost and easy deployment of visual sensors. There are essential differences in the modal characteristics of lidar data and camera image data. How to efficiently transfer the knowledge of the teacher model to the learning model through cross-modal knowledge distillation faces severe challenges in practical applications. Summary of the invention

[0005] In response to the problems in the prior art, the present invention provides a cross-modal foreground distillation system and method for multi-view three-dimensional object detection, which solves the shortcomings of existing vision-based three-dimensional object detection models in terms of accuracy, depth information reasoning and computing resource utilization efficiency.

[0006] The present invention proposes a cross-modal foreground distillation method for multi-view three-dimensional target detection, which is applicable to a distillation framework including an adaptive module, a foreground mask module, a spatial attention distillation module, a multi-scale foreground feature distillation module and a responsive distillation module, so as to use a laser radar or a multi-modal three-dimensional target detection model as a teacher model and a multi-view bird's-eye view BEV perception three-dimensional target detection model as a student model; the adaptive module is used to adjust the features of the student model so that the output features are consistent with the feature scale of the teacher model in terms of spatial distribution and semantic information; the foreground mask module is used to generate a foreground mask and a foreground scaling weight, and to identify the target area in the image, and at the same time, to adjust the distillation weights of target areas of different sizes; the spatial attention distillation module is used to generate a spatial attention map and perform knowledge distillation of spatial attention features; the multi-scale foreground feature distillation module is used to significantly improve the student model's perception of the target area by aligning the features of foreground areas of different scales; the responsive distillation module is used to design a distillation method of the loss function through classification and regression tasks respectively, and to perform difference calculation on the classification and regression outputs of the teacher model and the student model to achieve distillation optimization of the student model.

[0007] Specifically, the adaptive module in the distillation framework is composed of an upsampling layer and a convolution layer stacked together. The feature adjustment of the student model includes: first, improving the resolution of the feature through the upsampling layer, and then adjusting the spatial structure and dimensional information of the feature through the convolution layer, so that the output feature is consistent with the feature of the teacher model in terms of spatial distribution and semantic information; the expression processed by the adaptive module is as follows:

[0008]

[0009] in, is the input student model feature, For the adaptive module, It is the feature output after being processed by the adaptive module.

[0010] Specifically, in the foreground mask module of the distillation framework, the foreground mask It is obtained by summing the two-dimensional Gaussian masks generated by the center points of all target area bounding boxes. The formula is as follows:

[0011]

[0012] in, is the standard deviation, Representative bounding boxes, is the total number of bounding boxes, is the center point coordinate of the bounding box;

[0013] Foreground scaling weight Based on the foreground mask The value of is generated, where when the foreground mask value is greater than a certain threshold , the point is considered to belong to the foreground, and the scaling weight is calculated according to the foreground mask bounding box to which it belongs; when the foreground mask value is less than or equal to a certain threshold , the point is considered to belong to the background area, and a scaling weight is generated based on the number of background points; the specific calculation formula is as follows:

[0014]

[0015] in, For point The corresponding foreground mask value; Indicates A foreground region mask; Represents the foreground mask area The height and width of the foreground mask area are found by The corresponding bounding box is obtained; is the threshold set to distinguish foreground from background;

[0016] Foreground Mask and foreground scaling weight As the weighting coefficient in the multi-scale foreground feature distillation task, it can adjust the distillation weights of target areas of different sizes.

[0017] To be more specific, in the spatial attention distillation module of the distillation framework, the specific calculation formula for the spatial attention map is as follows:

[0018] ,

[0019] in, To perform average pooling on features in the channel dimension, For use The function normalizes the average pooling result to obtain the spatial attention map. is the characteristic of the teacher model or the student model, The number of channels, height, and width of the feature are used to calculate the spatial attention maps of the teacher model and the student model respectively using the above formula , and adjust the attention map of the student model through the adaptive layer , then, using the hyperparameters The spatial attention maps of the two models are fused. The specific formula is as follows:

[0020] ,

[0021] The fused spatial attention is used for subsequent multi-scale feature distillation;

[0022] For the knowledge distillation of spatial attention features, the spatial attention maps of the teacher model and the student model are multiplied by their respective original features to obtain the corresponding spatial attention features, thereby achieving spatial weighting of the features and highlighting the features of important areas. The formula is as follows:

[0023] ,

[0024] in, Represent the characteristics of the teacher and student models respectively, are the spatial attention maps of the teacher and student models, is element-wise multiplication, are the spatial attention features of the teacher and student models respectively; then, we use The difference between the spatial attention features of the two loss functions is as follows:

[0025]

[0026] in, are the length and width of the spatial attention feature, respectively. It is an adaptive module.

[0027] More specifically, define The features generated by the encoder at different levels are multi-scale features, which are expressed as: and in, are the number of channels, height, and width that represent the characteristics of the student model respectively; are the number of channels, height, and width of the teacher model respectively;

[0028] In the multi-scale foreground feature distillation module of the distillation framework, the mean square error loss function is used to measure the similarity between the features of the teacher model and the student model. The calculation formula is as follows:

[0029]

[0030] in, are the number of channels, height, and width of the feature, respectively. is the spatial attention map, Represented as the first Layer features, is the foreground mask, Scale the scale weights for the foreground mask.

[0031] To be more specific, in the distillation method that designs loss functions for classification and regression tasks of the responsive distillation module respectively, the classification loss mainly measures the errors of the teacher model and the student model in the classification task, while the regression loss measures their differences in the target attribute regression task;

[0032] In the classification task, the maximum value of the heat map is extracted from the output of the teacher model and the student model, using The loss function measures the absolute error between the maximum values ​​of the heatmaps of the teacher model and the student model. Finally, the absolute error is weighted and normalized with the foreground mask to obtain the final classification loss. , the specific formula is as follows:

[0033]

[0034] in, Represent the maximum values ​​of the heat maps of the teacher and student models, is the foreground mask, is the sum of the mask weights, is a numerical stability term, whose value is ;

[0035] For regression tasks, the regression output is extracted from the output features of the teacher model and the student model using The divergence measures the difference between the teacher model and the student model, The mathematical expression of divergence is:

[0036] ,in is the output of the teacher model, is the output of the student model; secondly, the different regression outputs are calculated separately Divergence, including position, height, size, rotation, and velocity;

[0037] Finally, The divergence and foreground mask are weighted and normalized to get the final regression loss , the specific formula is as follows:

[0038] in, are the position outputs of the teacher model and the student model respectively, are the height outputs of the teacher model and the student model respectively, are the size outputs of the teacher model and the student model respectively, are the rotation outputs of the teacher model and the student model respectively, are the speed outputs of the teacher model and the student model respectively, are the weights generated by Gaussian mask, is the sum of the mask weights, is a numerical stability term, whose value is .

[0039] Beneficial effects of the present invention:

[0040] 1. Adaptive module gain: The adaptive module adjusts the feature dimension and spatial resolution of the student model to align it with the teacher model features, solving the modal difference problem between lidar and camera image data and ensuring the effectiveness of cross-modal knowledge transfer.

[0041] 2. Foreground mask module gain: By generating foreground masks and foreground scaling weights, it effectively identifies and isolates foreground objects and background areas, avoids background interference, and ensures that the contributions of objects of different scales are more balanced during the distillation process. This module optimizes the key areas of feature distillation and greatly improves the alignment effect and distillation accuracy of target area features.

[0042] 3. Multi-scale foreground feature distillation module gain: This module combines foreground masks and scaling weights to enable the model to perform weighted alignment of foreground target features at different scales and reduce the impact of background noise. By accurately distilling features at different scales, the model's adaptability to multi-scale targets and feature learning effects are enhanced.

[0043] 4. Spatial attention distillation module gain: By calculating the spatial attention map, the features of important areas are weighted and highlighted. The student model can better learn the spatial structure and dimensional information of the teacher model, improve the attention to the target area and feature extraction capabilities, and effectively improve the distillation accuracy.

[0044] 5. Responsive distillation module gain: By optimizing the classification loss and regression loss, the student model can better learn the classification and regression outputs of the teacher model, thereby significantly improving the performance of the student model in 3D object detection tasks, especially the accuracy of classification and positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of a framework of a cross-modal foreground distillation method for multi-view three-dimensional object detection provided by an example of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited to the described contents.

[0047] The present invention proposes a cross-modal foreground distillation method for multi-view three-dimensional target detection, which is suitable for a newly constructed distillation framework, which aims to transfer the knowledge of multi-scale foreground area features, spatial attention information, and high-quality output (classification score, regression result) from the teacher model to the compact student model, wherein multi-scale foreground feature distillation is performed on BEV, spatial attention information is distilled by spatial attention, and classification score and regression result are responsively distilled. At the same time, in order to alleviate the imbalance problem of large and small objects and background areas to distillation loss caused by the background and large foreground objects occupying most of the area of ​​the feature map, a foreground scaling weight is introduced to balance objects of different sizes and reduce background interference. In addition, due to the large differences in the feature expression form and spatial structure of lidar data and camera image data, direct feature distillation will lead to poor distillation effect, so an adaptive module is introduced to correct different modal features.

[0048] Based on multiple perspectives Although the perceptual 3D object detection model has the advantages of low cost and easy deployment, the pure visual model cannot accurately infer depth and other geometric information, resulting in its performance being significantly lower than that of the 3D object detection model based on LiDAR or multi-modality. Knowledge distillation is a general model compression technique that extracts knowledge from the teacher model and transfers this knowledge to a smaller student model. By using the 3D object detection model based on LiDAR or multi-modality as the teacher model during the training process, its rich spatial structure and depth information are transferred to the student model (based on multi-view). Perceptual 3D object detection model), which can help the student model learn cross-modal geometric features, thereby making up for its lack of depth information.

[0049] like Figure 1 As shown, in the embodiment of the present invention, a distillation framework between the teacher model and the student model is constructed. The distillation framework mainly consists of five important parts: adaptive module, foreground mask module, multi-scale foreground feature distillation module, spatial attention distillation module, and responsive distillation module. Using LiDAR or multimodal 3D object detection model as the teacher model, with a bird's-eye view of multiple perspectives The perceptual 3D object detection model is used as a student model;

[0050] The distillation framework starts with the output features of the teacher model and the student model. First, the adaptive module adjusts the feature dimension and spatial resolution of the student model to align it with the features of the teacher model, thereby reducing the problem of feature mismatch. Then, the foreground mask module generates foreground masks and foreground scaling weights. The foreground mask is used to accurately locate the target area of ​​the feature map of the teacher model and the student model, ensuring that the multi-scale foreground feature distillation module can focus on these areas and reduce the interference of the background area on the distillation process. The foreground scaling weight balances the influence of each target in the distillation process by adapting to different target sizes, especially when the sizes of some foreground targets are very different, the larger object plays a dominant role in the calculation of the distillation loss. By applying different adaptive scaling weights to targets of different sizes, it ensures that all types of targets contribute equally to the distillation loss, avoids the background and large targets dominating the distillation loss calculation, and improves the feature alignment effect and distillation accuracy of targets of different sizes. Then, the spatial attention distillation module generates spatial attention maps for the teacher and student models respectively, and uses them to weight the original features to highlight the important regional features. Finally, the responsive distillation module transfers knowledge through classification loss and regression loss based on the classification and regression outputs of the teacher model and the student model, respectively, significantly improving the classification and regression performance of the student model in the object detection task.

[0051] The distillation framework introduces an adaptive module composed of upsampling layers and convolutional layers to adjust the features of the student model. Specifically, the adaptive module takes the features of the student model as input and adjusts its feature dimension and spatial resolution through a multi-layer structure to match the features of the teacher model. The process is expressed as:

[0052]

[0053] in, is the input student model feature, For the adaptive module, It is the feature output after being processed by the adaptive module.

[0054] In this distillation framework, the foreground mask module is used to generate the foreground mask and foreground scaling weight of the feature map, which are used to identify the target area in the image. At the same time, the distillation weights are adjusted for target areas of different sizes.

[0055] Further, generate the foreground mask The specific scheme is as follows: First, for a single target border, a two-dimensional Gaussian mask with the target center as the origin is generated according to the true value , calculate the Gaussian masks of all target bounding boxes and combine them into the final foreground mask. The specific formula is as follows:

[0056]

[0057] in, is the standard deviation, Representative bounding boxes, is the total number of bounding boxes, The coordinates of the center point of the bounding box.

[0058] Furthermore, the foreground scale weight Based on the foreground mask The foreground area in is generated. When the foreground mask value is greater than a certain threshold , the point is considered to belong to the foreground, and the scaling weight is calculated according to the foreground mask bounding box to which it belongs; when the foreground mask value is less than or equal to a certain threshold , the point is considered to belong to the background area, and a scaling weight is generated based on the number of background points. The specific formula is as follows:

[0059]

[0060] in, For point The corresponding foreground mask value; Indicates A foreground region mask; Represents the foreground mask area The height and width of the foreground mask area are found by The corresponding bounding box is obtained; is the threshold value set to distinguish foreground from background; foreground mask and foreground scaling weight As a weighting coefficient in the multi-scale foreground feature distillation task.

[0061] The spatial attention distillation module in this distillation framework consists of two important parts: calculating the spatial attention map and spatial attention distillation.

[0062] First, the spatial attention maps of the teacher model and student model features are calculated respectively, that is, the features are averaged, dimensionally transformed and normalized to obtain the attention mask. The calculation formulas are:

[0063] ,

[0064] in, To perform average pooling on features in the channel dimension, For use The function normalizes the average pooling result to obtain the spatial attention map. is the characteristic of the teacher model or the student model, The number of channels, height, and width of the feature. Use the above formula to calculate the spatial attention map of the teacher model and the student model respectively. , and the attention map of the student model is adjusted to .

[0065] Then use hyperparameters Spatial attention map for two models and After fusion, the final fused spatial attention map A is obtained. The calculation formulas are:

[0066] ,

[0067] Among them, the fused spatial attention map is used for subsequent multi-scale feature distillation.

[0068] For spatial attention distillation, the spatial attention maps of the teacher model and the student model are multiplied by their respective original features to obtain the corresponding spatial attention features. This implements spatial weighting of features and highlights the features of important areas. The formula is as follows:

[0069] ,

[0070] in, They are respectively represented as the teacher model features and the student model features after being corrected by the adaptive module. are the spatial attention maps of the teacher and student models, is element-wise multiplication, are the spatial attention features of the teacher and student models respectively. Then, we use Loss function calculates spatial attention distillation loss , measures the difference in the spatial attention characteristics of the two. The specific formula is as follows:

[0071]

[0072] in, are the length and width of the spatial attention feature, respectively. It is an adaptive module.

[0073] The multi-scale foreground feature distillation module in this distillation framework defines The features generated by the encoder at different levels are multi-scale features, which are expressed as: and .in, are the number of channels, height, and width that represent the characteristics of the student model respectively; are the number of channels, height, and width of the teacher model respectively. Using the adaptive module The multi-scale features of the student model are adjusted in feature dimension and spatial resolution to align with the scale features of the teacher model. Subsequently, the foreground mask module is used to generate the foreground mask. and foreground scaling weight , used for feature weighting. Finally, the mean square error loss function is used to measure the difference between the output features of the teacher model and the student model based on the multi-scale foreground feature distillation module. The formula is as follows:

[0074]

[0075]

[0076] in, are the number of channels, height, and width of the feature, respectively. is the spatial attention map, Represented as the first Layer features, is the foreground mask, Scale the scale weights for the foreground mask.

[0077] The responsive distillation module of the distillation framework is used to optimize the distillation of the student model by calculating the difference between the classification and regression outputs of the teacher model and the student model. The distillation method mainly designs loss functions for classification and regression tasks respectively. The classification loss mainly measures the errors of the teacher model and the student model in the classification task, while the regression loss measures their differences in the target attribute regression task.

[0078] In the classification task, the maximum value of the heat map is extracted from the output of the teacher model and the student model, using The loss function measures the absolute error between the teacher model and the student model. Finally, the loss value is weighted and normalized by the foreground mask to obtain the classification loss , the specific formula is as follows:

[0079]

[0080] in, Represent the maximum values ​​of the heat maps of the teacher and student models, is the foreground mask, is the sum of the mask weights, is a numerical stability term, whose value is .

[0081] For regression tasks, the regression output is extracted from the output of the teacher model and the student model, and the KL divergence is used to measure the difference between the teacher model and the student model. The mathematical expression of divergence is: ,in is the output of the teacher model, is the output of the student model. KL divergence is calculated for different regression outputs, including position, height, size, rotation, and speed, and finally weighted and standardized with the Gaussian mask to obtain the final regression loss , the formula is as follows:

[0082] in, are the position outputs of the teacher model and the student model respectively, are the height outputs of the teacher model and the student model respectively, are the size outputs of the teacher model and the student model respectively, are the rotation outputs of the teacher model and the student model respectively, are the speed outputs of the teacher model and the student model respectively, are the weights generated by Gaussian masks, is the sum of the mask weights, is a numerical stability term, whose value is .

[0083] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields, and the above specific embodiments are only illustrative and instructive, rather than restrictive. A person of ordinary skill in the art can also make many forms under the guidance of this specification and without departing from the scope of protection of the claims of the present invention, all of which belong to the protection of the present invention.

Claims

1. A cross-modal foreground distillation method for multi-view 3D object detection, characterized in that: The cross-modal foreground distillation method is applicable to a distillation framework including an adaptive module, a foreground mask module, a spatial attention distillation module, a multi-scale foreground feature distillation module and a responsive distillation module; a laser radar or a multi-modal 3D target detection model is used as a teacher model, and a multi-view bird's-eye view BEV perception 3D target detection model is used as a student model; The adaptive module is used to adjust the features of the student model so that the output features are consistent with the scale of the teacher model features in terms of spatial distribution and semantic information; The foreground mask module is used to generate foreground masks and foreground scaling weights, identify target regions in the image, and adjust the distillation weights of target regions of different sizes; The spatial attention distillation module is used to generate spatial attention maps and perform knowledge distillation of spatial attention features; Multi-scale foreground feature distillation module, which is used to significantly improve the student model's perception of the target area by aligning the features of foreground areas of different scales; The responsive distillation module is used to design the distillation method of the loss function through classification and regression tasks respectively, calculate the difference between the classification and regression outputs of the teacher model and the student model, and realize the distillation optimization of the student model.

2. The cross-modal prospect distillation method according to claim 1, characterized in that: The adaptive module in the distillation framework is composed of upsampling layers and convolutional layers stacked together. The expression processed by the adaptive module is as follows: ； in, is the input student model feature, For the adaptive module, It is the feature output after being processed by the adaptive module.

3. The cross-modal prospect distillation method according to claim 1, characterized in that: In the foreground mask module of the distillation framework, the foreground mask It is obtained by summing the two-dimensional Gaussian masks generated by the center points of all target area bounding boxes. The formula is as follows: ； in, is the standard deviation, Representative bounding boxes, is the total number of bounding boxes, is the center point coordinate of the bounding box; Foreground scaling weight Based on the foreground mask The specific calculation formula is as follows: ； in, For point The corresponding foreground mask value; Indicates A foreground region mask; Represents the foreground mask area The height and width of the foreground mask area are found by The corresponding bounding box is obtained; is the threshold set to distinguish foreground from background; Foreground Mask and foreground scaling weight As the weighting coefficient in the multi-scale foreground feature distillation task, it can adjust the distillation weights of target areas of different sizes.

4. The cross-modal prospect distillation method according to claim 1, characterized in that: In the spatial attention distillation module of the distillation framework, the specific calculation formula for the spatial attention map is as follows: ， ； in, To perform average pooling on features in the channel dimension, For use The function normalizes the average pooling result to obtain the spatial attention map. is the characteristic of the teacher model or the student model, The spatial attention maps of the teacher model and the student model are calculated using the above formulas, where the number of channels, height, and width of the feature are , and adjust the attention map of the student model through the adaptive layer , then, using the hyperparameters The spatial attention maps of the two models are fused. The specific formula is as follows: ， ； The fused spatial attention is used for subsequent multi-scale feature distillation; For the knowledge distillation of spatial attention features, the spatial attention maps of the teacher model and the student model are multiplied by their respective original features to obtain the corresponding spatial attention features, thereby achieving spatial weighting of the features and highlighting the features of important areas. The specific formula is as follows: ， ； in, Represent the characteristics of the teacher and student models respectively, are the spatial attention maps of the teacher and student models, is element-wise multiplication, are the spatial attention features of the teacher and student models respectively; then, we use The difference between the spatial attention features of the two loss functions is as follows: ； in, are the length and width of the spatial attention feature, respectively. It is an adaptive module.

5. The cross-modal prospect distillation method according to claim 1, characterized in that: definition The features generated by the encoder at different levels are multi-scale features, which are expressed as: and in, are the number of channels, height, and width that represent the characteristics of the student model respectively; are the number of channels, height, and width of the teacher model respectively; In the multi-scale foreground feature distillation module of the distillation framework, the mean square error loss function is used to measure the similarity between the features of the teacher model and the student model. The calculation formula is as follows: ； in, are the number of channels, height, and width of the feature, respectively. is the spatial attention map, Represented as the first Layer features, is the foreground mask, Scale the scale weights for the foreground mask.

6. The cross-modal prospect distillation method according to claim 1, characterized in that: In the distillation method that designs loss functions for classification and regression tasks of the responsive distillation module, the classification loss mainly measures the errors of the teacher model and the student model in the classification task, while the regression loss measures their differences in the target attribute regression task; In the classification task, the maximum value of the heat map is extracted from the output of the teacher model and the student model, using The loss function measures the absolute error between the maximum values ​​of the heatmaps of the teacher model and the student model. Finally, the absolute error is weighted and normalized with the foreground mask to obtain the final classification loss. , the specific formula is as follows: ； in, Represent the maximum values ​​of the heat maps of the teacher and student models, is the foreground mask, is the sum of the mask weights, is a numerical stability term, whose value is ; For regression tasks, the regression output is extracted from the output features of the teacher model and the student model using The divergence measures the difference between the teacher model and the student model, The mathematical expression of divergence is: ,in is the output of the teacher model, is the output of the student model; secondly, the different regression outputs are calculated separately Divergence, including position, height, size, rotation, and velocity; Finally, the KL divergence and the foreground mask are weighted and normalized to obtain the final regression loss , the specific formula is as follows: ； in, are the position outputs of the teacher model and the student model respectively, are the height outputs of the teacher model and the student model respectively, are the size outputs of the teacher model and the student model respectively, are the rotation outputs of the teacher model and the student model respectively, are the speed outputs of the teacher model and the student model respectively, are the weights generated by Gaussian masks, is the sum of the mask weights, is a numerical stability term, whose value is .

Images