Target detection method based on binary self-distillation Transform

Abstract

The invention relates to a target detection method based on binary self-distillation Transform, and belongs to the technical field of model compression and target detection. The method comprises the following steps: inputting a training image into a constructed binary Transform student network for forward propagation, synchronously extracting full-precision QKV and binary QKV in the network, constructing a teacher feature similarity matrix and a student feature similarity matrix, and aligning feature distribution through value exchange to obtain feature self-distillation loss; performing dynamic smoothing processing on the target class labels, and performing weak supervision in combination with non-target class soft labels to obtain label self-distillation loss; and carrying out dynamic weighted fusion on the target detection loss, the feature self-distillation loss and the label self-distillation loss of the training image to obtain total loss, optimizing binary Transform student network parameters based on the total loss until the model is converged, and obtaining a binary target detection Transform model for target detection. The objective of the invention is to solve the technical problem that a full-precision model cannot be pre-trained in the prior art, thereby improving the target detection precision.

Description

TECHNICAL FIELD

[0001] The application relates to a target detection method based on a binary self-distillation Transformer, and belongs to the technical field of model compression and target detection. BACKGROUND

[0002] As a core task of computer vision, target detection aims to locate and identify objects of interest in images, and is a key technology for many real-world applications such as autonomous driving, video surveillance, and intelligent robots. With the rapid extension of application scenarios to mobile terminals and edge devices (such as smartphones, drones, and vehicle-mounted cameras), there are stringent requirements for low latency, low power consumption, and high energy efficiency for target detection models. However, mainstream high-precision detection models usually have high computational complexity and large parameter quantities, and rely on powerful GPU computing power, making it difficult to run directly on resource-constrained edge devices in real time.

[0003] To address the above challenges, researchers have proposed various model compression and acceleration schemes. Among them, binarization quantizes network weights and activation values to 1 bit, converting floating-point multiplication and addition operations into efficient bit operations, which can theoretically achieve tens of times of model compression and acceleration, and is considered a promising solution for edge deployment. At the same time, knowledge distillation, as an effective performance enhancement method, is often used to transfer the knowledge of large teacher models to lightweight student models to make up for the performance loss caused by compression. However, existing binary Transformer methods mostly rely on such external distillation, but pre-training a high-performance full-precision teacher network itself requires a large amount of computational resources and time, with high training costs, making it difficult to deploy in scenarios where data sets are difficult to access or device resources are limited.

[0004] Based on the above background, the application proposes a target detection method based on a binary self-distillation Transformer, aiming to fully exploit and utilize the knowledge potential of the model itself, significantly improve the performance of the binary model without introducing an external teacher, and more accurately identify small-sized objects and locate object boundaries. SUMMARY

[0005] The application aims to provide a target detection method based on a binary self-distillation Transformer, aiming to solve the technical problem that existing technologies cannot pre-train full-precision models, thereby improving target detection accuracy.

[0006] To achieve the above object, the technical scheme of the present application is: a target detection method based on binary self-distillation Transform, which solves the problem of performance degradation in the prior art when lacking external teachers, improves the practicability and accuracy of the model on edge devices, so that it can more reliably identify complex scenes and small-scale targets, more accurately locate object boundaries, and ultimately realize high-precision, low-delay real-time detection on resource-constrained edge devices such as mobile phones and cameras, directly serving key scenes such as automatic driving safety perception, industrial vision quality inspection, and smart retail analysis, the method comprising the following steps: Step 1: binary processing of attention layers and linear layers in the target detection Transform model to construct a binary Transform student network; Step 2: input the training image into the binary Transform student network for forward propagation, simultaneously extract full-precision QKV and binary QKV in the binary Transform student network, and output the prediction result of the binary Transform student network, wherein the prediction result includes target class label and non-target class label; Step 3: constructing a teacher feature similarity matrix based on the teacher feature, constructing a student feature similarity matrix based on the student feature, and aligning the feature distribution through value exchange to obtain a feature self-distillation loss; Step 4: processing the target class label through dynamic smoothing, and combining with the non-target class soft label for weak supervision to generate target class label loss, non-target class label loss and weak supervision loss and fusion to obtain label self-distillation loss; Step 5: calculating the target detection loss of the training image; wherein the target detection loss includes classification loss, bounding box regression loss and object confidence loss; Step 6: dynamically weighting and fusing the target detection loss, the feature self-distillation loss and the label self-distillation loss to obtain a total loss, and iteratively optimizing the binary Transform student network parameters based on the total loss through back propagation until the model converges to obtain a binary target detection Transform model; Step 7: inputting the image data set to be detected into the binary target detection Transform model to obtain the image with detection frame output by the binary target detection Transform model, and completing target detection.

[0007] Optionally, the extraction of full-precision QKV and binary QKV in the binary Transform student network is specifically: linearly project the binary transformer student network layer, where, is the expression: where, is the full precision layer, , is the full precision projection weight matrix of the layer, is the output feature of the layer; binaryzation of the full precision using the sign function to obtain the binary layer, is the expression: where, is the binary layer, , is the sign function.

[0008] Optionally, the Step 3 is specifically: for the layer, the student feature is , the teacher feature is denoted by , and the expression is: L2 normalization is performed on the teacher feature and the student feature to obtain , and the expression is: where, is the normalized teacher feature, is the normalized student feature; a similarity matrix of the normalized teacher feature and the student feature is calculated, and the expression is: where, is the transposed matrix representation; the values in the teacher feature similarity matrix are replaced with the values of the student feature similarity matrix , but the sign of the teacher matrix is kept, and the expression after the value replacement is denoted by , and the expression is: where, denotes element-wise multiplication, This indicates that only symbols are extracted; Finally, the characteristic self-distillation loss is obtained. The expression is: in, Let f denote the f-norm.

[0009] Optionally, Step 4 specifically includes: The raw output of the linear layer of the Transformer model for object detection Transform the target class probability distribution using softmax. The conversion expression is: Calculate the dynamic smoothing factor The expression is: Where epoch is the current training epoch number, and total_epochs is the total number of training epochs; Probability distribution of target class Perform smoothing to obtain smooth labels. The expression is: in, Total number of categories; Constructing teacher soft tags by combining dynamic smoothing factors The target class label loss is obtained, expressed as: in, The mean represents the weight of the true label. Loss is the target class label; For non-target class labels, full-precision features are extracted from the intermediate layers of the binary Transformer student network using a hook function. The original output logit-mid of the intermediate layer is obtained, and the expression is: reserve The confidence score is assigned to the target class label, and the rest... Distribute evenly to non-target class tags to construct softened realistic tags. The result is a loss due to weak supervision. The expression is: in, These are hyperparameters that change with training. Let KL divergence be the KL divergence. For the first i Layer labels, In order to seek and Information entropy; Rank each non-target category The theoretical distribution in Zipf's law Alignment yields non-target class soft tags. And calculate the loss for non-target class labels, the expression is: Among them, using Indicates a non-target class. Indicates the target class. For the final output layer to the first The predicted probabilities of each category, For the final output layer to the first The predicted probabilities of each category, For non-target class label loss, Batchsize is the model's batch size, representing the number of training samples processed simultaneously in one forward and backward propagation. The target class label loss, the non-target class label loss, and the weak supervision loss are weighted and summed to obtain the label self-distillation loss. The expression is: in, These are the weight parameters.

[0010] Optionally, the expression for the target detection loss is: in, For target detection loss, For classifying losses, For bounding box regression loss, For object confidence loss, These are the weight parameters.

[0011] Optionally, the expression for the total loss is: in, For the total loss, For target detection loss, Characteristic of self-distillation loss, For label self-distillation loss, These are dynamic parameters used to adjust the weight of each loss at different training stages; When the model's accuracy improvement on the validation set is less than a preset improvement threshold, or the total loss drops below a preset convergence threshold and the change does not exceed a preset change threshold in several consecutive training epochs, the model is determined to have converged, and the converged binary object detection Transformer model is saved.

[0012] The beneficial effects of this invention are: it can efficiently generate high-performance, low-power lightweight models without relying on expensive pre-trained teacher models, significantly reducing training costs and deployment thresholds, solving the problem of frequent false detections and missed detections when detecting objects on edge devices, and generating detection boxes that can more accurately include the objects to be detected when detecting small-volume objects. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the overall framework of the present invention; Figure 2 This is a schematic diagram of the self-distillation feature of the present invention; Figure 3 This is a schematic diagram of the self-distillation of the label of the present invention. Detailed Implementation

[0014] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0015] Example 1: As Figure 1 As shown, a target detection method based on binary self-distillation Transformer includes the following steps: Step 1: Binarize the attention layer and linear layer in the object detection Transformer model to construct a binary Transformer student network; Step 2: Input the training image into the binary Transformer student network for forward propagation, simultaneously extract the full-precision QKV and binary QKV from the binary Transformer student network, use the full-precision QKV as teacher features and the binary QKV as student features, and output the prediction result of the binary Transformer student network; wherein, the prediction result includes target class label and non-target class label; Optionally, the extraction of the full-precision QKV and binary QKV in the binary Transformer student network specifically involves: For the binary Transformer student network, the first The layers are linearly projected, where... The expression is: in, It is the first Full precision of the layer , It is the first The full-precision projection weight matrix of the layer, It is the first The output features of the layer; Using sign functions for full precision Binarization is performed to obtain the first... Binary layers The expression is: in, For the first Binary layers , It is a symbolic function.

[0016] Optionally, in this embodiment, taking a specific autonomous driving target detection dataset as an example, the training images are a batch of images in the training set, such as 4 images (Batch Size=4), and the corresponding ground truth annotations, including bounding box coordinates and class labels.

[0017] Step 3: Construct a teacher feature similarity matrix based on the teacher features, construct a student feature similarity matrix based on the student features, and align the feature distribution through value exchange to obtain the feature self-distillation loss; Optionally, for the first Layer, student characteristics are Teacher characteristics The expression is: L2 normalization was performed on teacher and student characteristics to obtain the following results. In this example, we take Query as an example. The Key and Value are similar, and the expression is: in, For the normalized teacher characteristics, These are the normalized student characteristics; Calculate the similarity matrix between normalized teacher and student features. The expression is: in, Represented as a transpose matrix; Teacher feature similarity matrix The numerical values ​​in the matrix are based on the student feature similarity matrix. The values ​​are replaced, but the sign of the teacher matrix is ​​preserved. The teacher characteristics after transformation are expressed as follows: in, This indicates element-wise multiplication. This means that only symbols are extracted. It can be understood that by replacing values, teachers and students can be aligned numerically, which greatly reduces the initial differences. Finally, mean squared error is used to measure the difference between the exchanged teacher feature similarity matrix and the student feature similarity matrix, ultimately yielding the feature self-distillation loss. The expression is: in, Let f denote the f-norm.

[0018] It is understandable that, such as Figure 2 As shown, in this embodiment, the goal of the binary self-distillation feature part is to use full-precision QKV features as the teacher to guide the student network of binarized QKV features, and to reduce the teacher-student difference through a value exchange strategy. During the training process, since the ability to accept feature knowledge varies at different training stages, this embodiment only adds QKV features in the early stage of training. After training is completed to half of the full stage, V is added for distillation due to the improvement in model performance. The resulting loss will be used in the subsequent calculation of the total loss, directly guiding the network to learn how to extract more discriminative spatial relationship features using binarized weights. This is crucial for understanding the relative positions between vehicles and background, and between pedestrians and vehicles.

[0019] Step 4: The target class labels are dynamically smoothed, and then combined with non-target class soft labels for weak supervision to generate target class label loss, non-target class label loss and weak supervision loss, which are then fused to obtain label self-distillation loss; Optionally, such as Figure 3 As shown, this embodiment utilizes the network's own prediction output to construct high-quality soft labels, and improves the label learning ability of the binary network through a weak supervision mechanism. Specifically, it uses the original output of the linear layer of the object detection Transformer model to generate the label. Transform the target class probability distribution using softmax. The conversion expression is: Calculate the dynamic smoothing factor The expression is: Where epoch is the current training epoch number, and total_epochs is the total number of training epochs; It is understandable that, as can be seen from the expression of the dynamic smoothing factor, the dynamic smoothing factor decreases with the training cycle. Probability distribution of target class Perform smoothing to obtain smooth labels. The expression is: in, Total number of categories; Constructing teacher soft tags by combining dynamic smoothing factors The target class label loss is obtained, expressed as: in, The weights represent the true labels to restore the confidence level lost due to binary operations, with mean being the average value. Loss is the target class label; For non-target class labels, full-precision features are extracted from the intermediate layers of the binary Transformer student network using a hook function. The original output logit-mid of the intermediate layer is obtained, and the expression is: reserve The confidence score is assigned to the target class label, and the rest... Distribute evenly to non-target class tags to construct softened realistic tags. The result is a loss due to weak supervision. The expression is: in, These are hyperparameters that change with training. Let KL divergence be the KL divergence. For the first i Layer labels, In order to seek and Information entropy; To obtain the non-target class label loss, it is also necessary to construct the non-target class ranking. Combining the predictions from the intermediate and final layers, the relative importance of non-target classes is calculated, specifically by assigning the ranking number of each non-target class to the relevant data. The theoretical distribution in Zipf's law Alignment yields non-target class soft tags. And calculate the loss for non-target class labels, the expression is: Among them, using Indicates a non-target class. Indicates the target class. For the final output layer to the first The predicted probabilities of each category, For the final output layer to the first The predicted probabilities of each category, For non-target class label loss, Batchsize is the model's batch size, representing the number of training samples processed simultaneously in one forward and backward propagation. The target class label loss, the non-target class label loss, and the weak supervision loss are weighted and summed to obtain the label self-distillation loss. The expression is: in, These are the weight parameters.

[0020] It's important to understand that because soft labels dominate the label self-distillation process, the target class label loss should be factored into the weighting parameters. Set to a larger value.

[0021] Understandably, this embodiment constructs a label self-distillation loss so that the final output of the student network should not only predict the target class (vehicle) more accurately in terms of classification probability, but also correctly distinguish the most easily confused non-target classes. Furthermore, the intermediate layers should also learn to identify features to enhance the model's label discrimination ability.

[0022] Step 5: Calculate the target detection loss for the training image; wherein, the target detection loss includes classification loss, bounding box regression loss, and object confidence loss; Optionally, the expression for the target detection loss is: in, For target detection loss, For classifying losses, For bounding box regression loss, For object confidence loss, These are the weight parameters.

[0023] Optionally, in this embodiment, the classification loss can be a binary cross-entropy loss or Focal Loss, used to supervise the identification of object categories within the bounding box; the bounding box regression loss can be a CIoU Loss or GIoU Loss, used to accurately regress the position and size of the target bounding box; and the object confidence loss can be a binary cross-entropy loss, used to determine whether the prior box contains an object.

[0024] It is understood that the target detection loss constructed in this embodiment is the standard supervision signal for all target detectors, ensuring that the model masters the basic tasks of detection, namely localization, classification, and confidence.

[0025] Step 6: Dynamically weight and fuse the target detection loss, the feature self-distillation loss, and the label self-distillation loss to obtain the total loss. Based on the total loss, iteratively optimize the parameters of the binary Transformer student network through backpropagation until the model converges to obtain the binary target detection Transformer model. Optionally, the expression for the total loss is: in, For the total loss, For target detection loss, Characteristic of self-distillation loss, For label self-distillation loss, These are dynamic parameters used to adjust the weight of each loss at different training stages; It's important to understand that in the early stages of training, the focus is on the original task loss, and label self-distillation is introduced. During the middle stages of training, the relationship between these three aspects is balanced, and in the later stages, the focus shifts to feature self-distillation, thereby further improving the model's generalization ability and robustness.

[0026] Optionally, in this embodiment, the iterative optimization uses the chain rule for backpropagation to calculate the gradient of the total loss relative to all trainable parameters (weights) in the binary student network. A pass-through estimator (STE) is used to approximate the gradient, ensuring that the gradient can be effectively propagated back to the full-precision weights before the binary layer. An optimizer (such as AdamW) is used to update the network parameters based on the calculated gradient. When the model's accuracy improvement on the validation set is less than a preset improvement threshold, or the total loss drops below a preset convergence threshold and the change does not exceed a preset change threshold in several consecutive training epochs, the model is considered to have converged, and the converged binary object detection Transformer model is saved.

[0027] Step 7: Input the image dataset to be detected into the binary object detection Transformer model to obtain the image with detection boxes output by the binary object detection Transformer model, thus completing the object detection.

[0028] Specifically, in this embodiment, the binary object detection Transformer model is deployed on an in-vehicle edge computing device. When real-time camera footage is input, the model will output results with accurate bounding boxes and category labels.

[0029] In summary, this invention achieves effective training of a binary object detection Transformer model without relying on an externally pre-trained teacher model through self-knowledge distillation. First, a feature self-distillation path is constructed based on full-precision QKV features. Simultaneously, a label self-distillation mechanism is designed to reconstruct soft labels through dynamic smoothing of target class labels. Furthermore, the cross-comparison of non-target class labels with intermediate layer features compensates for the confidence loss caused by binarization, thereby improving the label learning ability of the binary model. This invention effectively addresses the challenges of data privacy and the inability to pre-train a full-precision model, making it suitable for deployment on resource-constrained edge devices, thus improving object detection accuracy.

[0030] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A target detection method based on binary self-distillation Transformer, characterized in that, The method includes the following steps: Step 1: Binarize the attention layer and linear layer in the object detection Transformer model to construct a binary Transformer student network; Step 2: Input the training image into the binary Transformer student network for forward propagation, simultaneously extract the full-precision QKV and binary QKV from the binary Transformer student network, use the full-precision QKV as teacher features and the binary QKV as student features, and output the prediction result of the binary Transformer student network; wherein, the prediction result includes target class label and non-target class label; Step 3: Construct a teacher feature similarity matrix based on the teacher features, construct a student feature similarity matrix based on the student features, and align the feature distribution through value exchange to obtain the feature self-distillation loss; Step 4: The target class labels are dynamically smoothed, and then combined with non-target class soft labels for weak supervision to generate target class label loss, non-target class label loss and weak supervision loss, which are then fused to obtain label self-distillation loss; Step 5: Calculate the target detection loss for the training image; wherein, the target detection loss includes classification loss, bounding box regression loss, and object confidence loss; Step 6: Dynamically weight and fuse the target detection loss, the feature self-distillation loss, and the label self-distillation loss to obtain the total loss. Based on the total loss, iteratively optimize the parameters of the binary Transformer student network through backpropagation until the model converges to obtain the binary target detection Transformer model. Step 7: Input the image dataset to be detected into the binary object detection Transformer model to obtain the image with detection boxes output by the binary object detection Transformer model, thus completing the object detection.

2. The target detection method based on binary self-distillation Transformer according to claim 1, characterized in that, The extraction of full-precision QKV and binary QKV from the binary Transformer student network specifically involves: For the binary Transformer student network, the first The layers are linearly projected, where... The expression is: ； in, It is the first Full precision of the layer , It is the first The full-precision projection weight matrix of the layer, It is the first The output features of the layer; Using sign functions for full precision Binarization is performed to obtain the first... Binary layers The expression is: ； in, For the first Binary layers , It is a symbolic function.

3. The target detection method based on binary self-distillation Transformer according to claim 2, characterized in that, Step 3 specifically refers to: For the Layer, student characteristics are Teacher characteristics The expression is: ； L2 normalization was performed on teacher and student characteristics to obtain the following results. The expression is: ； in, For the normalized teacher characteristics, These are the normalized student characteristics; Calculate the similarity matrix between normalized teacher and student features. The expression is: ； in, Represented as a transpose matrix; Teacher feature similarity matrix The numerical values ​​in the matrix are based on the student feature similarity matrix. The values ​​are replaced, but the sign of the teacher matrix is ​​preserved. The teacher characteristics after transformation are expressed as follows: in, This indicates element-wise multiplication. This indicates that only symbols are extracted; Finally, the characteristic self-distillation loss is obtained. The expression is: ； in, Let f denote the f-norm.

4. The target detection method based on binary self-distillation Transformer according to claim 1, characterized in that, Step 4 specifically refers to: The raw output of the linear layer of the Transformer model for object detection Transform the target class probability distribution using softmax. The conversion expression is: ； Calculate the dynamic smoothing factor The expression is: ； Where epoch is the current training epoch number, and total_epochs is the total number of training epochs; Probability distribution of target class Perform smoothing to obtain smooth labels. The expression is: ； in, Total number of categories; Constructing teacher soft tags by combining dynamic smoothing factors The target class label loss is obtained, expressed as: ； ； in, The mean represents the weight of the true label. Loss is the target class label; For non-target class labels, full-precision features are extracted from the intermediate layers of the binary Transformer student network using a hook function. The original output logit-mid of the intermediate layer is obtained, and the expression is: ； reserve The confidence score is assigned to the target class label, and the rest... Distribute evenly to non-target class tags to construct softened realistic tags. The result is a loss due to weak supervision. The expression is: ； in, These are hyperparameters that change with training. Let KL divergence be the KL divergence. For the first i Layer labels, In order to seek and Information entropy; Rank each non-target category The theoretical distribution in Zipf's law Alignment yields non-target class soft tags. And calculate the loss for non-target class labels, the expression is: ； ； Among them, using Indicates a non-target class. Indicates the target class. For the final output layer to the first The predicted probabilities of each category, For the final output layer to the first The predicted probabilities of each category, For non-target class label loss, Batchsize is the model's batch size, representing the number of training samples processed simultaneously in one forward and backward propagation. The target class label loss, the non-target class label loss, and the weak supervision loss are weighted and summed to obtain the label self-distillation loss. The expression is: ； in, These are the weight parameters.

5. The target detection method based on binary self-distillation Transformer according to claim 1, characterized in that, The expression for the target detection loss is: ； in, For target detection loss, For classifying losses, For bounding box regression loss, For object confidence loss, These are the weight parameters.

6. The target detection method based on binary self-distillation Transformer according to claim 1, characterized in that, The expression for the total loss is: ； in, For the total loss, For target detection loss, Characteristic of self-distillation loss, For label self-distillation loss, These are dynamic parameters used to adjust the weight of each loss at different training stages; When the model's accuracy improvement on the validation set is less than a preset improvement threshold, or the total loss drops below a preset convergence threshold and the change does not exceed a preset change threshold in several consecutive training epochs, the model is determined to have converged, and the converged binary object detection Transformer model is saved.

Images