DETR target detection model compression method based on combination of pruning and knowledge distillation

Abstract

The invention belongs to the field of computer vision, and particularly relates to a DETR target detection model compression method based on combination of pruning and knowledge distillation, and the method comprises the steps: calculating an attention mechanism of each layer of a Transform structure, and constructing a mask by using the attention mechanism; performing adaptive pruning on parameters of a linear layer and an attention layer of the Transform structure by using a mask to obtain a first compression model of the original model; taking a to-be-compressed encoder with a Transform structure as a student network, and taking the original model as a teacher network to carry out distillation learning; the student network performs distillation updating according to the global distillation loss and the intersection distillation loss to complete compression; according to the method, the calculated amount of the model can be remarkably reduced and the reasoning speed of the model can be greatly improved on the premise that the model does not have obvious precision glide.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a DETR target detection model compression method based on a combination of pruning and knowledge distillation. Background Art

[0002] DETR is a new target detection paradigm proposed in 2020, which belongs to the end-to-end detection method among anchor-free detection methods. The model structure of DETR is very simple and contains three main components: backbone network, Transformer encoder-decoder feature fusion network, and feedforward network. However, the network model is computationally expensive and memory intensive, which hinders its deployment in devices with low memory resources or applications with strict latency requirements. Therefore, the naive idea of ​​performing model compression and acceleration in deep networks without significantly reducing model performance has emerged. Generally speaking, the common techniques are mainly network pruning and knowledge distillation.

[0003] Network pruning can reduce network complexity and solve the problem of overfitting. Network pruning reduces the storage space required for neural networks while maintaining the original accuracy. The commonly used compression framework is a three-stage pipeline: network pruning, parameter quantization, and Huffman coding. Usually one compression framework or multiple compression frameworks are applied simultaneously to reduce the storage requirements of neural networks by 35 to 49 times without affecting the accuracy of the neural network. However, this pruning framework cannot be effectively used in models with Transformer structures. Therefore, a pruning method for Transformer structures is needed to reduce its huge computational overhead and promote the deployment and application of models in edge devices.

[0004] The core idea of ​​knowledge distillation is to guide the training of a small model (student model) through a pre-trained large model (teacher model). This process is like a teacher teaching a student. By transferring the performance of the large model to the small model, the small model can also perform well in resource-constrained environments (such as mobile devices and embedded systems), so it is called a teacher-student framework. Among them, the teacher model is usually a large deep neural network, which is trained on a large amount of training data to achieve high performance. Then, this teacher model is used to generate soft labels (i.e., the predicted probability distribution of the model), and these soft labels are used to train a small student model, which in turn helps the student model learn some implicit knowledge of the teacher model, thereby improving the performance of the student model. However, the traditional knowledge distillation framework is mainly aimed at classification problems and will distill the overall features of the image. Object detection needs to balance the feature differences between the foreground and background, and requires a new distillation method.

[0005] In summary, the model compression method based on network pruning cannot be effectively used in models with Transformer structure. The model compression method based on knowledge distillation will distill image features when applied to images. How to apply network pruning and knowledge distillation to models with Transformer structure for processing images is an urgent problem that technicians in this field need to solve. Summary of the invention

[0006] In order to significantly reduce the computational complexity of the DETR target detection model and greatly improve the inference speed of the model without a significant decrease in accuracy, the present invention proposes a DETR target detection model compression method based on a combination of pruning and knowledge distillation, which prunes and distills the Transformer structure in the target detection model to be compressed, and specifically includes the following steps:

[0007] Calculate the attention mechanism of each layer of the Transformer structure and use the attention mechanism to construct the mask;

[0008] The mask is used to adaptively prune the parameters of the linear layer and attention layer of the Transformer structure to obtain the first compression model of the original model;

[0009] The encoder of the Transformer structure to be compressed is used as the student network, and the original model is used as the teacher network for distillation learning;

[0010] The student network is distilled and updated according to the global distillation loss and the intersection distillation loss to complete the compression.

[0011] Preferably, performing adaptive pruning includes the following steps:

[0012] 101. In the forward pass of Transformer, n vectors are used as input to the linear layer, and the linear layer outputs m vectors. The j-th output of the linear layer is expressed as:

[0013]

[0014] Among them, a j represents the jth output of a linear layer; M j,k Represents the element value of the jth row and column of the mask matrix. When M j,k = 0 means that the mapping weight between the kth input and the jth output of the linear layer is pruned. j,k = 1, indicating that the mapping weight between the kth input and the jth output of the linear layer is retained; W j,k is the mapping weight between the kth input and the jth output of the linear layer; x k Represents the kth input value of the linear layer;

[0015] 102. In the reverse loss propagation, the gradient of the binary step function of the mask matrix is ​​0 everywhere, and the learning process of the significance score of the linear layer can be expressed as:

[0016]

[0017] Among them, L represents the loss function of the Transformer network, M j Represents the mask vector corresponding to the j-th output vector of a linear layer. When M j = 1 means that the j-th output vector of the linear layer is retained. j = 0 means that the j-th output vector of the linear layer is pruned; S j represents the significance score corresponding to the j-th output vector of a linear layer;

[0018] 201. In the forward pass of Transformer, the output of masked multi-head self-attention is expressed as:

[0019]

[0020] Among them, MSA represents the output of the masked multi-head self-attention corresponding to the input data; H is the number of self-attention heads; W proj is the linear transformation matrix; Attn h is the output of the hth attention head in the self-attention mechanism; M h Represents the mask vector corresponding to the h-th attention head of an attention layer;

[0021] 202. The learning process corresponding to the saliency score of the masked multi-head self-attention includes:

[0022]

[0023] Among them, S h represents the significance score corresponding to the hth attention head of an attention layer;

[0024] 301. Prune the Transformer network. During pruning, the augmented Lagrangian method is used to construct regularization on the threshold parameter. The penalty value of the regularization constructed on the threshold parameter is expressed as:

[0025]

[0026] Among them, L p represents the penalty value of regularization; λ1 and λ2 are two Lagrange multipliers; R t is the target pruning ratio; R is the current pruning ratio;

[0027] 302. Threshold parameter β of layer l in Transformer network l The learning process is expressed as:

[0028]

[0029] Among them, n l is the number of parameters in the lth layer of the Transformer network; σ(β l ) represents the threshold parameter β l Normalize; N is the total number of parameters in all layers of the Transformer network.

[0030] Preferably, the process of performing distillation updating according to the global distillation loss and the intersection distillation loss comprises the following steps:

[0031] Compare the feature map output by the model in the teacher network with the feature map of the real target, and calculate the binary mask for separating the foreground and the scaled mask for separating the background respectively;

[0032] Compute spatial and channel attention masks of the model output feature maps;

[0033] Using the mask of the teacher network to guide the student network to learn the focal distillation loss;

[0034] The global information of the feature map output by the model in the teacher network is captured, and the interaction information between the target and the overall image that is missing in the focal distillation learning is compensated by global distillation learning.

[0035] Preferably, the binary mask for separating the foreground is represented as:

[0036]

[0037] Among them, M i,j represents the value of the i-th row and j-th column of the binary mask, (i, j) represents the pixel point in the i-th row and j-th column in the feature map output by the model in the teacher network, and r represents the detection box corresponding to the real target.

[0038] Preferably, the scaling mask for separating the background is expressed as:

[0039]

[0040] Among them, (i, j) represents the pixel in the i-th row and j-th column of the feature map output by the model in the teacher network, r represents the detection box corresponding to the real target, and H r represents the height of the detection box r, W r Represents the width of the detection box r; H represents the height of the feature map output by the model in the teacher network, and W represents the width of the feature map output by the model in the teacher network.

[0041] Preferably, computing the spatial and channel attention masks of the model output feature maps comprises:

[0042] A S (F) = H·W·softmax(G S (F) / T)

[0043]

[0044] A C (F) = C softmax(G C (F) / T)

[0045]

[0046] Among them, A S (F) represents the spatial attention mask of the feature map F output by the model in the teacher network, H represents the height of the feature map output by the model in the teacher network, W represents the width of the feature map output by the model in the teacher network, G S (F) represents the binary mask used to separate the foreground, T is the distillation temperature hyperparameter, C represents the number of channels of the feature map output by the model in the teacher network, and F c A represents the eigenvalue of the cth channel of the feature map, c∈{1,2,…,C}, |·| represents the absolute value; C (F) represents the channel attention mask of the feature map F output by the model in the teacher network, G C (F) represents the scaling mask used to separate the background, F i,j Represents the pixel value of the pixel in the i-th row and j-th column in the feature map.

[0047] Preferably, in the process of using the mask of the teacher network to guide the student network to learn the focal distillation loss, the loss function is expressed as:

[0048] L focal =L fea +L at

[0049]

[0050] Among them, L focal The loss function learned for the focal distillation loss; L fea is the feature loss in focal distillation loss learning, L at is the attention loss in learning the focal distillation loss; γ represents the learnable parameter of the loss; Representing the spatial attention mask of the teacher network Spatial Attention Mask with Student Network The L1 loss between Represents the channel attention mask of the teacher network Channel attention mask with student network L1 loss between them; α and β represent learnable parameters; H, W, and C are the height, width, and number of channels of the feature map output by the model in the teacher network respectively; M i,j represents the value of the i-th row and j-th column in the binary mask matrix; S i,j Represents the value of the i-th row and j-th column in the scaling mask matrix; represents the value of the i-th row and j-th column in the spatial attention mask; Represents the value of the k-th channel attention mask; represents the value of the i-th row and j-th column in the k-th channel of the Transformer encoder output feature map in the teacher model; f(·) is the adaptive layer that reshapes the student network into the teacher network; Represents the value of the i-th row and j-th column in the k-th channel of the Transformer encoder output feature map in the student model.

[0051] Preferably, the global information of the feature graph output by the model in the teacher network is captured by constructing GcBlock, and the process of GcBlock capturing the global information includes:

[0052]

[0053] Among them, R(F) represents the global information of the feature map F output by the model in the teacher network; W v2 (·)、Wv1(·)、W K (·) represents the convolution operation; ReLU(·) is the ReLU activation function; LN(·) represents the layer normalization function; F j represents the jth pixel in the feature map F output by the model in the teacher network, N p Represents the number of pixels in the feature map F output by the model in the teacher network.

[0054] Preferably, when global distillation learning is used to make up for the missing interactive information between the target and the overall image in focal distillation learning, the loss function used is expressed as:

[0055] L global =λ·∑(R(F T )-R(F S )) 2

[0056] Among them, L global represents the loss function of global distillation learning; λ represents the hyperparameter of the balance loss; R(F T ) represents the feature map output by the teacher network Transformer encoder and input into the feature map global information captured by GcBlock; R(FS ) indicates that the feature map output by the student network Transformer encoder is input into the global information of the feature map captured by GcBlock.

[0057] The present invention uses a set of learnable pruning-related parameters to adaptively adjust the Transformer width, solving the problem that models with Transformer structures cannot be pruned effectively; at the same time, the present invention separates the foreground and background so that the student network focuses on the key pixels and channels of the teacher network and reconstructs the relationship between different pixels, solving the problem that traditional knowledge distillation methods cannot balance the differences in foreground and background features. Ultimately, the present invention can significantly reduce the amount of computation of the model and greatly improve the reasoning speed of the model without a significant drop in accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is the overall framework of the target detection model compression method of the present invention;

[0059] Figure 2 It is a schematic diagram of the width adaptive pruning method for Transformer of the present invention;

[0060] Figure 3 is a schematic diagram of the focal and global distillation method for separating foreground and background of the present invention;

[0061] Figure 4 It is a structural diagram of GcBlock in the global distillation method of the present invention. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] The present invention proposes a DETR target detection model compression method based on a combination of pruning and knowledge distillation, which prunes and distills the Transformer structure in the target detection model to be compressed, and specifically includes the following steps:

[0064] Calculate the attention mechanism of each layer of the Transformer structure and use the attention mechanism to construct the mask;

[0065] The mask is used to adaptively prune the parameters of the linear layer and attention layer of the Transformer structure to obtain the first compression model of the original model;

[0066] The encoder of the Transformer structure to be compressed is used as the student network, and the original model is used as the teacher network for distillation learning;

[0067] The student network is distilled and updated according to the global distillation loss and the intersection distillation loss to complete the compression.

[0068] In this embodiment, a DETR target detection model compression method based on a combination of pruning and knowledge distillation mainly includes the following three steps:

[0069] S1. Build a DETR target detection model compression framework based on a combination of pruning and knowledge distillation. During the model compression process, distillation and pruning are carried out simultaneously without affecting each other.

[0070] S2. Use a set of learnable pruning-related parameters to adaptively adjust the Transformer width. Specifically, it includes the following steps:

[0071] S21. Add a binary mask variable and the significance score of this variable to the linear layer and attention layer of the Transformer encoder-decoder. The binary mask variable has the same size and shape as the weight tensor of the layer and determines which weights participate in the forward propagation.

[0072] S22. Zero the weights of the significance scores below the learnable threshold parameter until a predefined compression ratio is reached.

[0073] S3, using focal and global distillation, focal distillation separates the foreground and background, forcing the student to focus on the teacher's key pixels and channels, and global distillation reconstructs the relationship between different pixels and transfers it from the teacher to the student to make up for the missing global information in focal distillation. Specifically, it includes the following steps:

[0074] S31. In the focus distillation part, the feature map output by the Transformer encoder is compared with the true target to calculate the corresponding binary mask and scaling mask to separate the foreground and background.

[0075] S32, calculate the spatial and channel attention masks of the feature map;

[0076] S33, substituting binary mask, scaling mask, spatial and channel attention mask of feature map into focus distillation loss enables the student network to learn the focus information in the image that needs more attention from the teacher network;

[0077] S34, in the global distillation part, the feature map output by the Transformer encoder is input into GcBlock to capture the global information of the feature map;

[0078] S35. The global distillation loss is used to make up for the interaction information between the target and the overall image that is missing in the focal distillation.

[0079] The process of knowledge distillation and network pruning in the present invention is as follows: Figure 1 As shown in the figure, the teacher network processes the input feature map in turn through the backbone network, Transformer encoder and other parts of the DETR target detection model. For the Transformer encoder, the global distillation loss is calculated through the features of GcBlock between the teacher network and the student network, and the focal distillation loss is calculated through the binary mask, scaling mask, spatial attention mask and channel attention mask between the teacher network and the student network. The student network distills the encoder and decoder in the Transformer network based on the global distillation loss and focal distillation loss, and prunes the parameters of the linear layer and attention layer in the encoder and decoder in the Transformer network based on width adaptive pruning.

[0080] As an optional implementation, Figure 2 , the width adaptive pruning for Transformer mainly includes linear layer pruning and attention layer pruning, which are adaptively pruned according to their respective learnable significance scores and threshold parameters. Specifically, and Respectively represent the weight matrix and mask matrix of n input mapping m output. The mask matrix indicates whether the corresponding output in the linear matrix is ​​pruned, that is, the corresponding value is pruned to 0, which can be expressed as follows:

[0081]

[0082] Among them, M j represents the mask vector corresponding to the j-th output vector of a linear layer; S j represents the significance score corresponding to the jth output vector of a linear layer; Φ(S|m×r) represents the m×rth smallest value in the search matrix S; Represents the field of real numbers of dimension m×n.

[0083] In the forward pass of the Transformer, the j-th output of the linear layer can be expressed as follows:

[0084]

[0085] Among them, a j represents the jth output of the linear layer; M j,k Represents the element value of the jth row and column of the mask matrix. When M j,k = 0 means that the mapping weight between the kth input and the jth output of the linear layer is pruned. j,k=1 indicates that the mapping weight between the kth input and the jth output of the linear layer is retained; is the mapping weight between the kth input and the jth output of the linear layer; x k Represents the kth input value of the linear layer.

[0086] In the reverse loss propagation, the gradient of the binary step function of the mask matrix M is 0 everywhere. The network is learned by gradient descent. The learning process of the significance score in the linear layer can be expressed as follows:

[0087]

[0088] Wherein, L represents the loss function of the Transformer network. For example, in this embodiment, the cross entropy loss can be used, that is, the cross entropy between the input data and the output data of the Transformer network is used as the loss function. In the present invention, the Transformer network is used for image processing, including tasks such as image classification, target detection, and image generation. It can also be used for text processing, such as text classification, machine translation, named entity recognition, sentiment analysis and other tasks. In some scenarios where the Transformer network needs to be applied, such as when deployed on a mobile terminal, it is necessary to reduce the memory usage of the Transformer network and improve the reasoning efficiency of the Transformer network. Therefore, it is necessary to reduce the number of parameters and the amount of calculation of the Transformer network. The present invention is suitable for reducing the number of parameters of the linear layer and the attention layer in the Transformer network.

[0089] As with the linear layer, in the forward pass of the Transformer, the network is learned using gradient descent. The output of the masked multi-head self-attention can be expressed as follows:

[0090]

[0091] Among them, MSA(x) represents the output of the masked multi-head self-attention when the input value is x; x is the input data of the multi-head self-attention; W proj is the linear transformation matrix; H is the number of attention heads; M h Represents the mask vector corresponding to the h-th attention head of an attention layer; Attn h (x) represents the attention score output by the hth attention head when the input value of the masked multi-head self-attention is x, expressed as:

[0092]

[0093] Among them, W h,q is the weight matrix of the h-th head query matrix in the attention mechanism, W h,kis the weight matrix of the h-th head key matrix in the attention mechanism, is the weight matrix of the h-th head value matrix in the attention mechanism, and d is the dimension of the value matrix.

[0094] Mask M h Covering the entire attention head, the learning process of the saliency score in the attention layer can be expressed as:

[0095]

[0096] Among them, S h represents the significance score corresponding to the h-th attention head of an attention layer; MSA represents the output of the masked multi-head self-attention corresponding to the input data.

[0097] Based on the above two learning processes, the AdamW optimizer is used to update the significance score during model training. Units with larger significance scores can learn more important features and thus be retained during pruning.

[0098] The pruning ratio r of layer l l The learnable threshold parameter β of the lth layer is obtained by using the sigmoid function l It is obtained by limiting it to (0,1). In order to learn the pruning ratio and control the size of the pruning model, the augmented Lagrangian method is used to construct regularization on the threshold parameter. The penalty value can be expressed as follows:

[0099]

[0100] Among them, L p Represents the penalty value of regularization. This parameter is a penalty term constructed to learn the pruning ratio and control the size of the pruning model; are two Lagrange multipliers; R t R t is the target pruning ratio; R is the current pruning ratio, which can be expressed as follows:

[0101]

[0102] Among them, n l is the number of parameters in the lth layer; N is the total number of parameters in all layers. The learnable threshold parameter β l The learning process can be expressed as follows:

[0103]

[0104] Among them, σ(β l ) represents the threshold parameter β l The purpose of the standardization operation in this embodiment is to adjust the threshold parameter β in the derivation process. l The value is limited to between 0 and 1.

[0105] It can be seen that when n of a certain layer l When is larger, the larger the magnitude of the gradient, the more it tends to be pruned. Adding λ1 and λ2 during the training process can be jointly optimized with the network weights and can also promote the learning process of the threshold parameters.

[0106] After obtaining the above parameters, the weights of the significance scores below the learnable threshold parameters are reset to zero until the predefined compression ratio is achieved. The 6 encoder blocks in the Transformer structure and the linear layers and attention layers in the decoder blocks are adaptively pruned during the pruning training process to achieve the best performance in learning.

[0107] Figure 3 This is a schematic diagram of the focus and global distillation method for separating the foreground and background of the present invention. In the focus distillation part, the feature map output by the Transformer encoder is compared with the real target to calculate the corresponding binary mask and scaling mask to separate the foreground and background, and the spatial and channel attention masks of the feature map are calculated at the same time. Then, these four masks are substituted into the focus distillation loss to enable the student network to learn the focus information in the image that needs more attention from the teacher network. Specifically, the following steps are included:

[0108] For the imbalance of foreground and background, focal distillation is used to separate the image and guide the student network to focus on key pixels and channels. First, a binary mask M is set to separate the background and foreground, which is expressed as follows:

[0109]

[0110] Among them, M i,j represents the value of the i-th row and j-th column of the binary mask, (i, j) represents the pixel point in the i-th row and j-th column in the feature map output by the model in the teacher network, and r represents the detection box corresponding to the real target.

[0111] The larger the target, the more pixels it has, and the greater the loss, which will affect the extraction of small targets. In different images, the ratio of foreground to background varies greatly. Therefore, in order to treat different targets equally and balance the loss of foreground and background, a ratio mask S is set here, as shown in the following formula:

[0112]

[0113] Among them, (i, j) represents the pixel in the i-th row and j-th column of the feature map output by the model in the teacher network, r represents the detection box corresponding to the real target, and H r represents the height of the detection box r, W r Represents the width of the detection box r; H represents the height of the feature map output by the model in the teacher network, and W represents the width of the feature map output by the model in the teacher network.

[0114] The corresponding attention mask is obtained by selecting the focus pixel and channel, and then the absolute average values ​​of different pixels and channels are calculated respectively, which is expressed as follows:

[0115] A S (F) = H·W·softmax(G S (F) / T)

[0116]

[0117] A C (F) = C softmax(G C (F) / T)

[0118]

[0119] Among them, A S (F) represents the spatial attention mask of the feature map F output by the model in the teacher network, H represents the height of the feature map output by the model in the teacher network, W represents the width of the feature map output by the model in the teacher network, G S (F) represents the spatial attention map of the feature map F output by the model in the teacher network, T is the distillation temperature hyperparameter, C represents the number of channels of the feature map output by the model in the teacher network, and F c A represents the eigenvalue of the cth channel of the feature map, c∈{1,2,…,C}, |·| represents the absolute value; C (F) represents the channel attention mask of the feature map F output by the model in the teacher network, G C (F) represents the channel attention map of the feature map F output by the model in the teacher network, F i,j Represents the pixel value of the pixel in the i-th row and j-th column in the feature map.

[0120] Since the masks of the student and teacher models are significantly different, the teacher’s mask is used to guide the student during training. Using binary mask M, scaled mask S, spatial and channel attention mask A S and A C , thus obtaining the feature loss L fea It is expressed as the following formula:

[0121]

[0122] In addition, using the attention loss L fea The student detector is forced to imitate the spatial and channel attention masks of the teacher detector, which can be expressed as follows:

[0123]

[0124] Focus loss Lfocal is the feature loss L fea and attention loss L at The sum of is as follows:

[0125] L focal =L fea +L at

[0126] Among them, L focal The loss function learned for the focal distillation loss; L fea is the feature loss in focal distillation loss learning, L at is the attention loss in learning the focal distillation loss; γ represents the learnable parameter of the loss; Representing the spatial attention mask of the teacher network Spatial Attention Mask with Student Network The L1 loss between Represents the channel attention mask of the teacher network Channel attention mask with student network L1 loss between them; α and β represent learnable parameters; H, W, and C are the height, width, and number of channels of the feature map output by the model in the teacher network respectively; M i,j represents the value of the i-th row and j-th column in the binary mask matrix; S i,j Represents the value of the i-th row and j-th column in the scaling mask matrix; represents the value of the i-th row and j-th column in the spatial attention mask; Represents the value of the i-th row and j-th column in the channel attention mask; represents the value of the i-th row and j-th column in the c-th channel of the Transformer encoder output feature map in the teacher model; f(·) is the adaptive layer that reshapes the student network into the teacher network; Represents the value of the i-th row and j-th column in the c-th channel of the Transformer encoder output feature map in the student model.

[0127] In the global distillation part, the feature map output by the Transformer encoder is input into GcBlock to capture the global information of the feature map, and then the global distillation loss is used to make up for the interaction information between the target and the overall image that is missing in the focal distillation. Figure 4 , is a structural diagram of GcBlock in the global distillation method of the present invention, which specifically includes the following steps:

[0128] First, focal distillation is used to separate the image and force the student model to focus on the key parts. This distillation cuts off the relationship between the foreground and the background, so global distillation is needed to extract the global relationship between different pixels from the feature map and extract knowledge from the teacher model to the student model. GcBlock is used to capture global relationship information in a single image, so that the student detector learns the global relationship loss L from the teacher detector. global , see the following formula:

[0129] L global =λ·∑(R(F T )-R(F S )) 2

[0130]

[0131] Among them, L global represents the loss function of global distillation learning; λ represents the hyperparameter of the balance loss; R(F T ) represents the feature map output by the teacher network Transformer encoder and input into the feature map global information captured by GcBlock; R(F S ) represents the global information of the feature map output by the Transformer encoder of the student network input into the feature map captured by GcBlock; R(F) represents the global information of the feature map F output by the model in the teacher network; W v2 (·)、Wv1(·)、W K (·) represents a 1×1 convolution operation; ReLU(·) is the ReLU activation function; LN(·) represents the layer normalization function; F j represents the jth pixel in the feature map F output by the model in the teacher network, F m represents the mth pixel in the feature map F output by the model in the teacher network, N p Represents the number of pixels in the feature map F output by the model in the teacher network.

[0132] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A DETR target detection model compression method based on a combination of pruning and knowledge distillation, It is characterized in that Prune and distill the Transformer structure in the object detection model to be compressed, including the following steps: Calculate the attention mechanism of each layer of the Transformer structure and use the attention mechanism to construct the mask; The mask is used to adaptively prune the parameters of the linear layer and attention layer of the Transformer structure to obtain the first compression model of the original model; The encoder of the Transformer structure to be compressed is used as the student network, and the original model is used as the teacher network for distillation learning; The student network is distilled and updated according to the global distillation loss and the intersection distillation loss to complete the compression.

2. According to claim 1, a DETR target detection model compression method based on a combination of pruning and knowledge distillation, It is characterized in that Adaptive pruning involves the following steps:

101. In the forward pass of Transformer, n vectors are used as input to the linear layer, and the linear layer outputs m vectors. The j-th output of the linear layer is expressed as: Among them, a j represents the jth output of a linear layer; M j,k Represents the element value of the jth row and column of the mask matrix. When M j,k = 0 means that the mapping weight between the kth input and the jth output of the linear layer is pruned. j,k = 1, indicating that the mapping weight between the kth input and the jth output of the linear layer is retained; W j,k is the mapping weight between the kth input and the jth output of the linear layer; x k Represents the kth input value of the linear layer; 102. In the reverse loss propagation, the gradient of the binary step function of the mask matrix is ​​0 everywhere, and the learning process of the significance score of the linear layer can be expressed as: Among them, L represents the loss function of the Transformer network, M j Represents the mask vector corresponding to the j-th output vector of a linear layer. When M j = 1 means that the j-th output vector of the linear layer is retained. j = 0 means that the j-th output vector of the linear layer is pruned; S j represents the significance score corresponding to the j-th output vector of a linear layer; 201. In the forward pass of Transformer, the output of masked multi-head self-attention is expressed as: Among them, MSA represents the output of the masked multi-head self-attention corresponding to the input data; H is the number of self-attention heads; W proj is the linear transformation matrix; Attn h is the output of the hth attention head in the self-attention mechanism; M h Represents the mask vector corresponding to the h-th attention head of an attention layer; 202. The learning process corresponding to the saliency score of the masked multi-head self-attention includes: Among them, S h represents the significance score corresponding to the hth attention head of an attention layer; 301. Prune the Transformer network. During pruning, the augmented Lagrangian method is used to construct regularization on the threshold parameter. The penalty value of the regularization constructed on the threshold parameter is expressed as: Among them, L p represents the penalty value of regularization; 1 , 2 are two Lagrange multipliers; R t is the target pruning ratio; R is the current pruning ratio; 302. Threshold parameter β of layer l in Transformer network l The learning process is expressed as: Among them, n l is the number of parameters in the lth layer of the Transformer network; σ(β l ) represents the threshold parameter β l Normalize; N is the total number of parameters in all layers of the Transformer network.

3. According to claim 1, a DETR target detection model compression method based on a combination of pruning and knowledge distillation, It is characterized in that The process of distillation update based on global distillation loss and intersection distillation loss includes the following steps: Compare the feature map output by the model in the teacher network with the feature map of the real target, and calculate the binary mask for separating the foreground and the scaled mask for separating the background respectively; Compute spatial and channel attention masks of the model output feature maps; Using the mask of the teacher network to guide the student network to learn the focal distillation loss; The global information of the feature map output by the model in the teacher network is captured, and the interaction information between the target and the overall image that is missing in the focal distillation learning is compensated by global distillation learning.

4. According to claim 3, a DETR target detection model compression method based on a combination of pruning and knowledge distillation, It is characterized in that The binary mask used to separate the foreground is represented as: Among them, M i,j represents the value of the i-th row and j-th column of the binary mask, (i, j) represents the pixel point in the i-th row and j-th column in the feature map output by the model in the teacher network, and r represents the detection box corresponding to the real target.

5. According to claim 3, a DETR target detection model compression method based on a combination of pruning and knowledge distillation, It is characterized in that The scaling mask used to separate the background is expressed as: Among them, (i, j) represents the pixel in the i-th row and j-th column of the feature map output by the model in the teacher network, r represents the detection box corresponding to the real target, and H r represents the height of the detection box r, W r Represents the width of the detection box r; H represents the height of the feature map output by the model in the teacher network, and W represents the width of the feature map output by the model in the teacher network.

6. According to claim 3, a DETR target detection model compression method based on a combination of pruning and knowledge distillation, It is characterized in that Computing the spatial and channel attention masks of the model output feature maps includes: A S (F)=H·W·softmax(G S (F) / T) A C (F)=C·softmax(G C (F) / T) Among them, A S (F) represents the spatial attention mask of the feature map F output by the model in the teacher network, H represents the height of the feature map output by the model in the teacher network, W represents the width of the feature map output by the model in the teacher network, G S (F) represents the binary mask used to separate the foreground, T is the distillation temperature hyperparameter, C represents the number of channels of the feature map output by the model in the teacher network, and F c A represents the eigenvalue of the cth channel of the feature map, c∈{1,2,…,C}, |·| represents the absolute value; C (F) represents the channel attention mask of the feature map F output by the model in the teacher network, G C (F) represents the scaling mask used to separate the background, F i,j Represents the pixel value of the pixel in the i-th row and j-th column in the feature map.

7. According to claim 6, a DETR target detection model compression method based on a combination of pruning and knowledge distillation, It is characterized in that In the process of using the mask of the teacher network to guide the student network to learn the focal distillation loss, the loss function is expressed as: L focal =L fea +L at Among them, L focal The loss function learned for the focal distillation loss; L fea is the feature loss in focal distillation loss learning, L at is the attention loss in learning the focal distillation loss; γ represents the learnable parameter of the loss; Representing the spatial attention mask of the teacher network Spatial Attention Mask with Student Network The L1 loss between Represents the channel attention mask of the teacher network Channel attention mask with student network L1 loss between them; α and β represent learnable parameters; H, W, and C are the height, width, and number of channels of the feature map output by the model in the teacher network respectively; M i,j represents the value of the i-th row and j-th column in the binary mask matrix; S i,j Represents the value of the i-th row and j-th column in the scaling mask matrix; represents the value of the i-th row and j-th column in the spatial attention mask; Represents the value of the k-th channel attention mask; represents the value of the i-th row and j-th column in the k-th channel of the Transformer encoder output feature map in the teacher model; f(·) is the adaptive layer that reshapes the student network into the teacher network; Represents the value of the i-th row and j-th column in the k-th channel of the Transformer encoder output feature map in the student model.

8. According to claim 3, a DETR target detection model compression method based on a combination of pruning and knowledge distillation, It is characterized in that GcBlock is constructed to capture the global information of the feature map output by the model in the teacher network. The process of GcBlock capturing global information includes: Among them, R(F) represents the global information of the feature map F output by the model in the teacher network; W v2 (·), Wv 1 (·),W K (·) represents the convolution operation; ReLU(·) is the ReLU activation function; LN(·) represents the layer normalization function; F j represents the jth pixel in the feature map F output by the model in the teacher network, N p Represents the number of pixels in the feature map F output by the model in the teacher network.

9. A DETR target detection model compression method based on a combination of pruning and knowledge distillation according to claim 8, It is characterized in that When global distillation learning is used to make up for the missing interactive information between the target and the overall image in focal distillation learning, the loss function used is expressed as: L global =λ·∑(R(F T )-R(F S )) 2 Among them, L global represents the loss function of global distillation learning; λ represents the hyperparameter of the balance loss; R(F T ) represents the feature map output by the teacher network Transformer encoder and input into the feature map global information captured by GcBlock; R(F S ) indicates that the feature map output by the student network Transformer encoder is input into the global information of the feature map captured by GcBlock.

Images