A Compression Method for Swin Transformer Model Incorporating Weight Sharing, Pruning, and Distillation

Abstract

The present invention discloses a method for compressing the Swin Transformer model by integrating weight sharing, pruning, and distillation, which relates to the technical field of neural networks. The method includes the following steps: sharing weights among multiple layers of the Swin Transformer model, adding a transformation layer between the layers of the model to achieve weight transformation; constructing and analyzing the parameter dependence mapping graph of the transformation block to obtain the mutual dependence between parameters, and constructing a binary grouping matrix F to record the dependence relationship between all parameters, and realizing parameter grouping through a depth-first search algorithm, and the parameters belonging to the same group will be removed simultaneously; using the KL divergence distillation method to recover the performance of the model after weight sharing and structured pruning. The method can effectively reduce the complexity of the model and better balance the relationship between the computational complexity and performance of the model.

Description

Technical Field

[0001] The present invention relates to the field of neural network technology, and in particular to a SwinTransformer model compression method integrating weight sharing, pruning and distillation. Background Art

[0002] In recent years, in pursuit of improved model performance, deep neural network architectures have become increasingly complex, leading to a continuous increase in model computational complexity and posing significant challenges to computing and storage capabilities. The SwinTransformer (Shifted Window Layered Visual Transformer) model has gained widespread application due to its excellent performance. This model addresses the shortcomings of traditional Transformer models in computer vision tasks by introducing a "Shifted Window" mechanism, resulting in a model architecture that excels in these tasks. However, this model has high computational complexity. For example, the Swin-Large model trained on the ImageNet-1K dataset has a whopping 197 million parameters, making it unsuitable for applications with limited computing resources, such as edge computing, the Internet of Things, and real-time prediction. Research has shown that large-scale pre-trained models suffer from over-parameterization. Therefore, compressing and optimizing these pre-trained models to eliminate redundant parameters is necessary and feasible, while minimizing performance degradation. Compressed Swin Transformer models are more adaptable to various application scenarios while also reducing deployment costs, which is crucial for building efficient intelligent systems and ensuring their effective application in diverse environments.

[0003] While existing SwinTransformer model compression methods have achieved some success, they still face several challenges. First, current model compression methods have yet to strike a good balance between computational complexity and performance. Furthermore, existing research remains limited in its ability to coordinate the relationships between various compression techniques to improve model compression effectiveness. Therefore, how to integrate multiple model compression techniques to reduce computational complexity and achieve model lightweighting while minimizing performance degradation remains a pressing issue. Summary of the invention

[0004] The technical problem to be solved by the present invention is how to provide a Swin Transformer model compression method that can effectively reduce the complexity of the model and better balance the relationship between the model calculation complexity and model performance.

[0005] To solve the above technical problems, the technical solution adopted by the present invention is as follows: A Swin Transformer model compression method that integrates weight sharing, pruning, and distillation, comprising the following steps:

[0006] Weight sharing: Share weights between multiple layers of the Swin Transformer model, and add a transformation layer between the layers of the model to achieve weight transformation;

[0007] Structured pruning: Construct and analyze the parameter dependency mapping graph of the transformation block to obtain the mutual dependencies between parameters, and construct a binary grouping matrix F to record the dependency relationships between all parameters. Parameter grouping is achieved through a depth-first search algorithm, and parameters belonging to the same group will be removed simultaneously;

[0008] Distillation: Use the KL divergence distillation method to recover the performance of the model after weight sharing and structured pruning;

[0009] Use the compressed Swin Transformer model to classify images.

[0010] The beneficial effects of adopting the above technical solution are as follows: By integrating multiple compression techniques, the method successfully realizes the lightweight of the Swin Transformer model and effectively balances the relationship between computational complexity and performance. When the compressed Swin Transformer model is applied to the image classification task, it can reduce the computational complexity and improve the image classification accuracy. Brief Description of the Drawings

[0011] The present invention will be further described in detail below with reference to the drawings and specific embodiments.

[0012] Figure 1 is the flowchart of the method described in the embodiments of the present invention;

[0013] Figure 2 is the principle block diagram of classical weight sharing in the method described in the embodiments of the present invention;

[0014] Figure 3 is the principle block diagram of weight transformation sharing in the method described in the embodiments of the present invention;

[0015] Figure 4 is the MSA and MLP linear transformation block diagram in the method described in the embodiments of the present invention;

[0016] Figure 5 is the different dependency graphs of the neural network in the method described in the embodiments of the present invention;

[0017] Figure 6 is the training accuracy change curve graph in the embodiments of the present invention. Detailed implementation manners

[0018] Combined with the accompanying drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0019] Many specific details are set forth in the following description in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed below.

[0020] Generally, as Figure 1 shown, the embodiments of the present invention disclose a Swin Transformer model compression method that integrates weight sharing, pruning, and distillation. The method includes the following steps:

[0021] First, share weights between multiple layers of the Swin Transformer model to improve parameter utilization, and add a transformation layer to implement weight transformation to increase parameter diversity.

[0022] Secondly, construct and analyze the parameter dependence mapping graph of the transformation block to obtain the mutual dependence between parameters, and construct a binary grouping matrix F to record the dependence relationship between all parameters. Implement parameter grouping through the Depth-First Search (DFS) algorithm. Parameters belonging to the same group are removed simultaneously to reduce the model calculation complexity and accelerate model inference.

[0023] Finally, use the KL divergence distillation method to solve the problem of performance degradation caused by weight sharing and pruning.

[0024] Weight sharing

[0025] Weight sharing is an effective method to improve parameter utilization efficiency. Its core idea is to share parameters across layers. Classic weight sharing is as Figure 2 shown,

[0026] Weight sharing can be expressed as a recursive update of a Transformer block:

[0027] Z i+1 = f(Z i ; θ), i = 0,..., L - 1 (1)

[0028] In the formula, Z iDenote the feature embedding of the $i$-th layer sequence. $L$ is the total number of layers, and $\theta$ is the weight shared across layers by the Transformer block. Weight sharing can prevent the number of parameters from growing with the increase in network depth. This application proposes a weight sharing method. Specifically, first, multiple layer weights are merged into a single shared weight, and at the same time, transformations are involved to increase parameter diversity.

[0029] The key idea of weight transformation is to transform the shared weight so that different layers have slightly different weights, which can not only promote parameter diversity but also improve the model's expressive ability and training stability.

[0030] Weight transformation sharing is as Figure 3 shown. In this application, multiple layer weights are combined into a single weight on the shared part, and at the same time, a transformation layer is added to increase parameter diversity. $T(x)$ represents the transformation of the weight.

[0031] As Figure 4 shown, this application applies a linear transformation to the weights of the multi-head self-attention (MSA) and the multi-layer perceptron (MLP) for each weight-sharing transformation layer. Each layer includes a separate transformation matrix, so the weights of the MSA module and the MLP module are different between layers. The weights of the MSA and the MLP are shared at each stage, and two transformation layers are added to increase parameter diversity.

[0032] MSA transformation: To improve parameter diversity, we insert two linear transformations before and after the softmax self-attention module. The transformed attention is defined as:

[0033]

[0034] where $M$ is the number of attention heads, $d$ is the dimension of the query, key, and value matrices. At the $K$-th head, the query, key, and value are generated by linear projection, denoted by $Q$ k , $K$ k and $V$ k ∈ $\mathbb{R}$ N×d respectively. The weight is denoted by $A$ k , and softmax acts on each row of the input matrix. $F$ (1) , $F$ (2) ∈ $\mathbb{R}$ M×M are the linear transformations before and after softmax respectively. Such linear transformations can make each attention matrix $A$ n different, and at the same time, can combine the information between attention heads to increase parameter variance. Finally, a fully connected layer is applied to the concatenation of all head outputs.

[0035] MLP Transformation: Introduce a lightweight transformation in the MLP to improve parameter diversity. Specifically, let the input be Y = [y1,..., y d , where y d represents the embedding vector at the d-th position of all tokens. Then introduce a d-dimensional linear transformation to convert Y to Y' = [C (1) y1,..., C (d) y d , where C (1) ,..., C (d) ∈ R N×N is the independent weight matrix of the linear layer. Then the MLP output is re-expressed as:

[0036] H = σ(Y'W (1) + b (1) )W (2) + b (2) (4)

[0037] where σ represents the activation function, W (1) ∈ R d×d' , b (1) ∈ R d' , W (2) ∈ R d'×d and b (2) ∈ R d are the weights and biases of the first and second layers respectively. Usually, d' > d is set. To reduce the number of parameters and introduce locality in the transformation, depth convolution is used to sparsify and share the weights in each weight matrix, so that the number of parameters is K 2 d, instead of N 2 d (K << N), where K is the convolution kernel size. After the transformation, the output of the MLP is more diverse, improving the parameter utilization efficiency.

[0038] Structured Pruning:

[0039] The goal of network pruning is to remove redundant parameters from a given network to reduce its size and potentially speed up the inference speed. In this application, by constructing a parameter dependency mapping graph of the transformation block, the mutual dependencies between parameters are deeply analyzed, and the mutually dependent parameters are grouped together. The importance of each group of parameters is calculated by the L2 norm, and pruning is performed according to the set pruning ratio and the importance degree of each group of parameters.

[0040] As Figure 5 shown, starting from a linear neural network composed of three consecutive layers, this simple neural network can be structurally pruned by removing neurons to make it slender. In this case, it is easy to find that some dependencies appear between the parameters, which forces W L and W L+1Pruned simultaneously.

[0041] Specifically, to prune the connection W L and W L+1 of the k-th neuron, both W L [k, :] and W L+1 [:, k] will be deleted. When hoping to achieve acceleration by pruning a certain neuron (shown in black), multiple groups of parameters connected to this neuron need to be removed simultaneously. These mutually dependent parameters form the smallest unit of structured pruning, usually called a group.

[0042] Furthermore, model the parameter dependencies in the SwinTransformer model. In structured pruning, the parameters within the same group are pairwise dependent. When hoping to remove one of them, all the parameters belonging to this group need to be removed synchronously to ensure the structural correctness. By constructing a binary grouping matrix F to record the dependency relationships between all parameter pairs, if the parameters of the i-th layer and the j-th layer are mutually dependent, it is represented by F ij In this way, the grouping of parameters can be simply modeled as a query problem.

[0043] f(i) = {j | F ij = 1} (5)

[0044] However, whether parameters are mutually dependent is not only determined by themselves but also affected by the intermediate layers between them. The dependency relationships between adjacent layers can be deduced recursively. For example, if there is a dependency between adjacent layers A and B, and there is also a dependency between adjacent layers B and C, then it can be deduced that there is also a dependency between A and C, even though A and C are not directly connected. Furthermore, utilize the local dependency relationships between adjacent layers to recursively deduce the required grouping matrix F. And this local dependency relationship between adjacent layers is what we call the dependency mapping graph. It is a sparse and local relationship graph because it only models the dependencies for directly connected layers.

[0045] Thus, the grouping problem can be simplified into a path search problem. When there is a path between node i and node j in the dependency mapping graph, it can be obtained that F ij = 1, that is, i and j belong to the same group. Grouping is performed through the depth-first search (DFS) algorithm. Starting from a certain node i, find the new node j connected to it in the dependency mapping graph and merge it into the current group until there are no new connected nodes. Then, the parameters belonging to the same group will be removed simultaneously to achieve model compression.

[0046] Distillation:

[0047] To address the performance degradation issues caused by weight sharing and pruning, distillation is further employed to transfer knowledge from a large model to a small and compact model. The model obtained from pruning and weight sharing is retrained, and the resulting model serves as the student model in distillation.

[0048] The established distillation objective function is as follows:

[0049] L train = L pred + D KL (6)

[0050] where L pred is the prediction loss, and D KL is the KL divergence loss.

[0051] Considering prediction logit distillation. The prediction loss is as follows:

[0052]

[0053] where z s and z t are the logits predicted by the student model and the teacher model respectively, and T is the temperature value used to control the smoothness of the logits. In the experiments of this application, T is set to 1. CE represents the cross-entropy loss.

[0054] Meanwhile, the KL (Kullback-Leibler divergence) divergence method in knowledge distillation is adopted to improve the accuracy of the Swin Transformer student model. The KL divergence not only allows the student model to copy the direct output of the teacher model but also mimics the subtle changes in its probability distribution, thereby capturing richer inter-class relationships and probability structures. By minimizing the KL divergence between the output of the student model and the output of the teacher model, the student model can more accurately approximate the decision boundary of the teacher model, which is particularly crucial in complex image classification tasks. In addition, compared with the traditional classification loss, the KL divergence provides a more detailed error signal for the student model, helping it achieve more robust learning during the training process.

[0055]

[0056] where P and Q are the output probability distributions of the teacher and student models respectively. After softmax processing, the probability values for each class i are P(i) and Q(i).

[0057] Experimental results and analysis:

[0058] For the Swin Transformer model used in image classification tasks, the performance of the method described in this application was evaluated on the ImageNet-Tiny-200 dataset. First, a performance benchmark for an uncompressed Swin Transformer baseline model was established. Subsequently, the models compressed using the Torch-Pruning model compression technique and the models compressed using the method proposed in this application were independently tested, and a comparative analysis of the performance of these three different models was conducted.

[0059] The experimental results show that compared with the original uncompressed Swin Transformer model, after using the compression method proposed in this application, while the model computational complexity is reduced by 32%, the performance degradation is only about 3% at the lowest. This conclusion strongly proves that the compression method described in this application effectively realizes the reduction of the model size while ensuring high performance. In contrast, the model compression method proposed in this application surpasses the existing Torch-Pruning technology and shows a more excellent performance in maintaining the balance between model performance and compression rate.

[0060] Experiment settings and dataset:

[0061] In this embodiment, three models with different scales, namely Swin-Tiny, Swin-Small, and Swin-Base, were used as baseline models respectively. The ImageNet-Tiny-200 dataset was adopted, which contains 200 categories, and each category is composed of a subset from the large ImageNet dataset, providing 100,000 images. Each category has approximately 500 training images, 50 validation images, and 50 test images. During the training process, this application used the Adam optimizer with a learning rate of 10e-4 to train the model, and the learning rate decayed exponentially with a decay factor of 0.95 to ensure sufficient training. The experimental code for this experiment was written in the Python language, using the Pytorch deep learning framework, and was trained and tested on NVIDIA 3090. This experiment was compared with the currently popular model compression method Torch-Pruning, and the comparison metrics specifically included: the number of parameters, the amount of computation, the model recognition accuracy, etc.

[0062] Analysis of experimental results:

[0063] For Swin Transformer models of different scales, the batch_size was set to 64, and the models were trained for 50 epochs. The changes in accuracy during the training process of the original model, the model compressed by Torch-pruning, and the model compressed by the method proposed in this paper were compared. The changes in the training accuracy of the three models on the ImageNet-Tiny-200 dataset are as Figure 6 shown.

[0064] As Figure 6 shown, compared with the baseline model, the model pruned by the Torch-pruning method reduced the number of parameters by 30.69% on the Tiny-scale Swin Transformer model, but the Top-1 accuracy of the corresponding model decreased by 6.94%. Under the condition that the computational complexity of the model proposed in this application is lower, the average performance loss of the model is only about 4.5%, which is better than the comparative method.

[0065] It can be seen that the accuracy of the model compressed by the method proposed in this application remains at a relatively high level, indicating that while reducing the computational complexity of the model, this method shows a more excellent performance in maintaining the balance between model performance and compression rate. It should be noted that the reason why the blue curve in the figure has a relatively high recognition accuracy in the first round is that the model has been retrained before distillation, and the blue curve in the figure is the performance change curve in the final distillation training stage.

[0066] Table 1 Comparison of model performance and computational complexityTable 1Model performance computationalcomplexity

[0067]

[0068] In Table 1, the baseline Swin method is the method in "Hierarchical vision transformer using shiftedwindows[C] / / Proceedings of the IEEE / CVF international conference on computervision.2021:10012-10022.";

[0069] In Table 1, the Torch-Pruning method is the method in "Fang G, Ma X, Song M, et al. Depgraph: Towards any structural pruning [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2023: 16091-16101."

[0070] Table 1 shows the comparison of the performance and computational complexity of the original model, the Torch-pruning method, and the model compressed by the method described in this application. The calculation methods of performance loss and compression rate are the percentages of the decreased values of Top-1 and the number of parameters respectively accounting for the original values. The results in the table show that under high magnification compression, the performance of the traditional model compression algorithm drops significantly, resulting in a large loss of model performance. However, the method described in this application better balances the relationship among the number of parameters, computational amount, and classification accuracy. Compared with the Torch-Pruning model compression algorithm, the method described in this application achieves an average increase of 3.5% in image classification accuracy under the condition of lower model computational complexity. Compared with the original baseline model, the average loss of classification accuracy is about 4.5%.

[0071] The method realizes the lightweight of the SwinTransformer model and better balances the relationship between the model computational complexity and performance. Experiments on the ImageNet-Tiny-200 public dataset show that compared with the original uncompressed Swin Transformer model, after adopting the compression method described in this application, under the condition that the model computational complexity is reduced by 32%, the performance drops by only about 3% at the lowest, effectively reducing the model computational complexity.

Claims

1. A Swin Transformer model compression method integrating weight sharing, pruning and distillation, characterized by It includes the following steps: Weight sharing: Share weights among multiple layers of the Swin Transformer model, and add transformation layers between the model layers to achieve weight transformation; Structured pruning: Construct and analyze the parameter dependency mapping graph of the transformation block to obtain the mutual dependencies between parameters, and construct a binary grouping matrix F to record the dependency relationships between all parameters. Parameter grouping is achieved through the depth-first search algorithm, and parameters belonging to the same group will be removed simultaneously; Distillation: Use the KL divergence distillation method to recover the performance of the model after weight sharing and structured pruning; Use the compressed Swin Transformer model to classify images.

2. The Swin Transformer model compression method integrating weight sharing, pruning, and distillation according to claim 1, characterized in that: In the weight sharing: Combine the weights of multiple layers into a single weight on the shared part, and add transformation layers to increase parameter diversity.

3. A method for compressing a Swin Transformer model that integrates weight sharing, pruning, and distillation according to claim 2, wherein: Apply a linear transformation to the weights of the multi-head self-attention MSA module and the multi-layer perceptron MLP module in the model for each transformation layer of weight sharing; each layer includes a separate transformation matrix, and the weights of the MSA module and the MLP module are different between layers. Share the weights of MSA and MLP in each stage, and add two transformation layers to increase parameter diversity.

4. The Swin Transformer model compression method integrating weight sharing, pruning and distillation as claimed in claim 3, wherein Perform MSA transformation in the multi-head self-attention MSA module: Insert two linear transformations before and after the softmax self-attention module, and the transformed attention is defined as: where M is the number of attention heads, d is the dimension of the query, key, and value matrices. For the K-th head, the query, key, and value are generated by linear projection and denoted as Q k , K k , and V k ∈ R N×d , respectively. The weights are denoted as A k . The softmax function is applied to each row of the input matrix; F (1) and F (2) ∈ R M×M are the linear transformations before and after the softmax, respectively. The linear transformations make each attention matrix A n different and combine the information between attention heads to increase the parameter variance. Finally, a fully connected layer is applied to the concatenation of all head outputs.

5. A compression method for the Swin Transformer model that integrates weight sharing, pruning, and distillation, characterized in that Perform MLP transformation in the multi-layer perceptron MLP module: Introduce a lightweight transformation in the MLP, with the input being Y = [y1,..., y d , where y d represents the embedding vector at the d-th position of all tokens; then introduce a d-dimensional linear transformation to convert Y to Y' = [C (1) y1,..., C (d) y d , where C (1) ,..., C (d) ∈R N×N is the independent weight matrix of the linear layer, and then the MLP output is re-expressed as: H = σ(Y'W (1) + b (1) )W (2) + b (2) where σ represents the activation function, W (1) ∈R d×d' , b (1) ∈R d' , W (2) ∈R d'×d and b (2) ∈R d are the weights and biases of the first and second layers respectively, set d' > d; use depth convolution to sparsify and share the weights in each weight matrix, such that the number of parameters is K 2 d, instead of N 2 d, K << N, where K is the convolution kernel size.

6. The Swin Transformer model compression method integrating weight sharing, pruning and distillation according to claim 1, characterized in that: In the structured pruning: By constructing the parameter dependency mapping graph of the transformation block, deeply analyze the mutual dependencies between parameters, group the mutually dependent parameters together, calculate the importance of each group of parameters through the L2 norm, and perform pruning according to the set pruning ratio according to the importance degree of each group of parameters.

7. A method for compressing a Swin Transformer model that integrates weight sharing, pruning, and distillation according to claim 6, wherein: Pruning connection W L and W L+1 for the k-th neuron, remove W L [k, :] and W L+1 [:, k]. When acceleration is achieved by pruning a certain neuron, multiple groups of parameters connected to this neuron need to be removed simultaneously. These mutually dependent parameters form the smallest unit of structured pruning, called a group Group; Model the parameter dependency in the Swin Transformer model. In structured pruning, the parameters in the same group are dependent on each other. When you want to remove one of them, all the parameters in the group need to be removed simultaneously. A binary grouping matrix F is constructed to record the dependency between all parameter pairs. If the parameters of the i-th layer and the j-th layer are dependent on each other, F is used. ij To express it, the grouping of parameters is modeled as a query problem: f(i) = {j | F ij = 1} Utilize the local dependency relationship between adjacent layers to recursively deduce the required grouping matrix F. This local dependency relationship between adjacent layers is called the dependency mapping graph, which is a sparse and local relationship graph that only models the dependencies of directly connected layers; Thus, the grouping problem is simplified into a path search problem. When there is a path between nodes i and j in the dependency mapping graph, F ij = 1, that is, i and j belong to the same group, and grouping is performed through the depth-first search (DFS) algorithm; starting from a certain node i, find the new node j connected to it in the dependency mapping graph and merge it into the current group until there are no new connected nodes. Then, the parameters belonging to the same group will be removed simultaneously to achieve model compression.

8. A method for compressing a Swin Transformer model that integrates weight sharing, pruning, and distillation according to claim 1, wherein: Retrain the model obtained by pruning and weight sharing, and use the obtained model as the student model in distillation; The established distillation objective function is: L train =L pred +D KL where P and Q are the output probability distributions of the teacher and student models respectively, after softmax processing, the probability values for each category i are P(i) and Q(i); Consider prediction logit distillation, and the prediction loss is shown as follows: Among them, z s and z t are the logits predicted by the student model and the teacher model respectively, T is the temperature value used to control the smoothness of the logits, and CE represents the cross entropy loss; Meanwhile, the KL divergence method in knowledge distillation is adopted to improve the accuracy of the Swin Transformer student model; by minimizing the KL divergence between the outputs of the student model and the teacher model, the student model is made to approximate the decision boundary of the teacher model more closely. The KL divergence loss formula is as follows: where P and Q are the output probability distributions of the teacher and student models respectively. After softmax processing, the probability values for each class i are P(i) and Q(i).

Images