A compression method of a visual transformer model

Abstract

The application discloses a compression method of a visual Transformer model, determines a to-be-compressed block in a Transformer Block of a Transformer Encoder of the visual Transformer model according to actual needs, inputs an input image into an embedding layer of the visual Transformer model to generate image embedding, and then inputs the image embedding into the Transformer Encoder; when the Transformer Block is the to-be-compressed block, the important score of a patch token is calculated; the patch token processed by a multi-head self-attention module and a superposition normalization module is sampled and fused according to the important score, an image after compression of the patch token is obtained, and then the image is output to a subsequent module. The patch token of the Transformer Encoder is compressed, the calculation complexity of the visual Transformer model is reduced, the inference speed is improved, and meanwhile, the performance and universality of the model are maintained.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and more specifically relates to a compression method of a visual Transformer model. BACKGROUND

[0002] With the rapid development of computer vision, deep learning has achieved remarkable success in image classification, object detection, and semantic segmentation tasks. Among them, Convolutional Neural Networks (CNNs) have always been the mainstream method for traditional vision tasks due to their local receptive field and parameter sharing mechanism. However, the emergence of Vision Transformers (ViTs) marks a paradigm shift in the field of computer vision. ViTs utilize self-attention mechanisms and can capture global features in images, exhibiting strong feature modeling capabilities. Their performance even surpasses traditional CNN models in multiple tasks.

[0003] The advantage of ViTs lies in its flexible feature expression ability and global information modeling capability. ViTs divide the input image into fixed-size image patches, and then embed these patches. ViTs sequence the image data into a word sequence similar to natural language processing tasks. In this way, ViTs can capture the global correlation between all pixels in the image through self-attention mechanisms, thereby generating more rich context feature representations. This method breaks the limitations of CNNs relying on local receptive fields, and especially in complex scenes or tasks requiring global perception, it exhibits better performance.

[0004] However, this performance improvement comes at a huge computational cost. The computational complexity of ViTs models mainly comes from its self-attention mechanism, which needs to calculate the similarity between all input tokens. Specifically, for an input sequence containing N tokens, the computational complexity of the self-attention mechanism is O(N 2 ). When the image resolution is high, the number of tokens increases rapidly, resulting in a large consumption of computational resources during training and inference. In addition, ViTs models usually contain tens of millions to hundreds of millions of parameters, for example, a typical ViT-Base model contains 86M parameters, and requires large-scale datasets (such as ImageNet) to support training.

[0005] To address these challenges, researchers have proposed various optimization strategies to reduce the computational complexity of ViTs.

[0006] Firstly, efficient architecture design is one of the main means to reduce the computational demand of ViTs. This approach reduces the computational load while maximizing model performance by redesigning the basic structure of the model or introducing optimization mechanisms. The core idea is to allocate computational resources in a more economical way, thus achieving a good balance between performance and efficiency. SwinTransformer significantly reduces the computational complexity of high-resolution images through a hierarchical structure and shift window mechanism. Its method limits self-attention computation within a local window and achieves window-to-window interaction through window shifting, which enhances the modeling ability of local features and preserves global long-range dependency information. EfficientViT combines multi-scale attention mechanisms and lightweight convolution optimization to improve model efficiency. It limits the global attention computation range, reducing complexity from quadratic to linear, while embedding lightweight convolution to complement the ViTs' deficiencies in local feature extraction, thus capturing local context information at a lower cost and further improving the expressive ability. These methods optimize the architecture to maximize the balance between performance and efficiency, but they also have the problem of insufficient generality. The design is usually targeted at a specific model structure, making it difficult to be simply transferred to other models, thus limiting its applicability in diverse application scenarios.

[0007] Secondly, model quantization techniques are also widely used in the optimization of ViTs. Quantization effectively reduces computational complexity and memory consumption by reducing model parameters and calculations from high precision (such as 32-bit floating-point numbers) to low precision (such as 8-bit integers). However, the application of quantization techniques also faces many limitations. For example, Quantization-Aware Training (QAT) improves model adaptability by simulating the effects of quantization during training, but this often increases training time and complexity. In addition, Post-Training Quantization (PTQ) avoids additional training overhead, but it is often difficult to maintain model performance at very low precision (such as 4 bits or below), thus requiring higher quality and quantity of calibration data. These limitations also pose challenges to the practical deployment of quantization techniques. Due to the varying quantization precision and format supported by hardware platforms, models need to be optimized for specific hardware environments, which increases the difficulty of deployment.

[0008] Token compression techniques effectively reduce the computational complexity of ViTs models by reducing the number of input tokens processed by the model, which is an important direction for optimizing model efficiency. The core of token compression techniques is to retain key tokens according to the importance of input tokens and reduce unnecessary computational resource consumption while minimizing performance loss. For example, DynamicViT dynamically evaluates the importance score of each token by using a lightweight prediction module and discards low-score tokens, thereby adapting to different input characteristics while reducing complexity. However, the direct deletion of tokens may result in the loss of useful information, especially in fine-grained feature modeling. In addition, the introduction of a dynamic evaluation module also increases computational overhead. To solve these problems, the Tome method proposes to calculate the similarity between tokens and merge high-similarity tokens by weighting to retain more key information and reduce the negative impact of direct deletion. Although this approach is an effective means of reducing computational cost, it still needs to balance efficiency and performance and design more general compression strategies to adapt to diverse application scenarios. SUMMARY

[0009] The present application aims to overcome the shortcomings of the prior art and provide a compression method for visual Transformer models, which reduces the computational complexity of visual Transformer models and improves inference speed while maintaining the performance and versatility of the model by compressing patch tokens of the Transformer Encoder.

[0010] To achieve the above-mentioned application purposes, the compression method for visual Transformer models of the present application comprises the following steps:

[0011] S1: Let the number of Transformer Blocks in the Transformer Encoder of the visual Transformer model be M, and select K Transformer Blocks from the M Transformer Blocks as the to-be-compressed Blocks according to actual needs, and let the original serial number corresponding to the kth to-be-compressed Block be m k , k = 1, 2, …, K;

[0012] S2: input the input image into the embedding layer of the visual Transformer model, and the embedding layer generates image embedding x and inputs the Transformer Encoder, the generation method of the image embedding being:

[0013] Divide the input image into N patches, encode each patch to obtain an embedding vector n = 1, 2, …, N, and D represents the dimension of the embedding vector, and the position of the patch is encoded to obtain a position vector e n , and the patch representation vector z is obtained by superimposing n=f n +e n Then each patch represents the vector z. n The image embedding is constructed as a patch token. Where z cls Indicates a category token;

[0014] S3: The Transformer Encoder extracts features from the image embedding and then outputs them to subsequent modules for classification. During feature extraction, for the m-th Transformer Block, when m ≠ m k If m = m, then no model compression is performed. k The following method is used for model compression:

[0015] S3.1: Let the image embedding received by the current Transformer Block be... The 0th token Classification tokens 1 to N m The token is a patch token. j = 1, 2, ..., N m N m Indicates image embedding X m The number of patch tokens; denote the value matrix of the input to the multi-head self-attention module in the current Transformer Block. Where d v The dimension representing the values ​​is used to calculate the attention matrix. The importance score for each patch token is calculated using the following formula:

[0016]

[0017] Among them, A 0,n Represents category tokens With patch token Attention scores between Represents the value matrix V m Patch Token The corresponding value vector, where |||| denotes the norm;

[0018] S3.2: Let the image embedding obtained after processing by the multi-head self-attention module and the superposition normalization module be denoted as . Set the compression ratio R according to the actual situation, and calculate the number of patch token samples H. m =R×N m Based on the importance score of the patch token from N m patch token H was obtained from sampling mA patch token constitutes a patch token queue P, and the greater the importance score is, the higher the probability of being selected is;

[0019] S3.3: According to the actual situation, set the fusion ratio T, and calculate the number of fusion tokens G m =T*H m Then, the patch token queue P is divided into two queues C1 and C2, wherein the queue C1 contains G m patch tokens, and the queue C2 contains H m -G m patch tokens; each patch token in the queue C1 is traversed, the patch token with the greatest similarity is selected from the queue C2, and the patch token is deleted from the queue C2, and then the following formula is used to fuse to obtain a fused patch token:

[0020]

[0021] Wherein, represents the gth patch token in the queue C1, g=1, 2, …, G m , j g represents the original serial number of the patch token , w represents the fused patch token, w g represents the weight, represents the patch token with the greatest similarity to the patch token in the queue C2, represents the original serial number of the patch token .

[0022] The classification token, the G m patch tokens fused and the remaining patch tokens in the set C2 constitute an image embedding and are output to a subsequent module.

[0023] The compression method of the visual Transformer model has the following beneficial effects:

[0024] The compression method of the visual Transformer model has the following beneficial effects:

[0025] 1) The present application accurately identifies patch tokens with low importance degree in the model inference process by comprehensively considering the importance of patch tokens of the visual transformer model, effectively reduces redundant tokens without introducing additional trainable parameters, while maximizing the retention of important information, thereby maximizing the compression of the model without affecting the model accuracy, and significantly reducing the computing resources required for model running.

[0026] 2) Although the present application reduces the amount of patch information, it still maintains the high precision processing capability of the model by accurately controlling the selection and fusion of patch tokens, ensuring the accuracy of the image processing result.

[0027] 3) The compression method designed by the present application has high universality and can be applied to multiple models with minor modifications and good results. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 is a specific implementation flowchart of the visual Transformer model compression method of the present application;

[0029] Figure 2 is a flowchart of patch token compression in the present application;

[0030] Figure 3 is a structure diagram of the improved Transformer Encoder of the present application;

[0031] Figure 4 is a schematic diagram of the processing process of the compressed Block in the present embodiment. DETAILED DESCRIPTION

[0032] The specific embodiments of the present application will be described below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present application. It should be particularly noted that in the following description, when the detailed description of known functions and designs may obscure the main content of the present application, these descriptions will be omitted here.

[0033] EMBODIMENT

[0034] Figure 1 is a specific implementation flowchart of the visual Transformer model compression method of the present application. As shown in Figure 1 , the specific steps of the visual Transformer model compression method of the present application include:

[0035] S101: Determine the Transformer Block to be compressed:

[0036] The number of Transformer Blocks in the Transformer Encoder of the visual Transformer model is M. According to actual needs, K Transformer Blocks are selected from the M Transformer Blocks as compressed Transformer Blocks. The original serial number of the kth compressed Transformer block is denoted as m k , k = 1, 2, …, K.

[0037] In actual applications, the compressed Transformer blocks can be evenly distributed among all the Transformer blocks. Therefore, the original serial number m k of the compressed Transformer block can be calculated by the following formula:

[0038]

[0039] wherein, represents the floor function.

[0040] S102: Generating image embedding:

[0041] The input image is input into the embedding layer of the visual Transformer model, and the embedding layer generates the image embedding x and inputs the Transformer Encoder. The generation method of the image embedding is as follows:

[0042] The input image is evenly divided into N patches. These patches are regarded as the basic unit of model processing, i.e., tokens. Each patch is encoded to obtain an embedding vector n = 1, 2, …, N, and D represents the dimension of the embedding vector. The position of the patch is encoded to obtain a position vector e n , and the patch representation vector z n is obtained by superimposing f n + e n . Then, each patch representation vector z n is taken as a patch token to construct the image embedding x wherein z cls represents a classification token.

[0043] In this embodiment, the input image adopts image samples in the ImageNet dataset, and the image resolution is adjusted to 224x224 pixels to adapt to the input requirements of the model. Then, the image is normalized to scale the pixel value from [0, 255] to [0, 1], and the generalization ability of the model is improved through data enhancement techniques such as random cropping and flipping. These normalization processes ensure the quality and consistency of the input image, laying a solid foundation for the training and testing of the model.

[0044] Common patch size is 16x16 pixels or 32x32 pixels. In this embodiment, 16x16 is used. For example, if the input image size is 224x224 and the size of each patch is 16x16, the image will be divided into 16x16 patches, each of which will contain local information of the image.

[0045] In this embodiment, patch encoding uses linear layer implementation. First, each patch is flattened into a one-dimensional vector. When the patch size is 16x16 pixels and the image is an RGB image, the flattened vector of each patch is a 16x16x3=768-dimensional vector. Then the flattened patch is mapped through the linear layer to convert it into a D-dimensional vector. In this embodiment, D=768. Since the Transformer model cannot automatically process the order information of elements in the sequence, a position vector needs to be added to each patch. The position vector provides the relative or absolute position of the patch in the image. These position encodings can be pre-defined and generated using sine and cosine functions. Finally, the patch embedding vector and the position encoding are added to obtain the final patch representation vector. In order to enhance the aggregation ability of global information, a special classification token ([CLS] token) needs to be added at the beginning of the input sequence. The final image embedding is constructed by the classification token and the patch token (patch representation vector).

[0046] S103: Patch token compression:

[0047] The Transformer Encoder extracts features from the image embedding and then outputs to the subsequent module to complete the classification. During feature extraction, for the mth layer of the Transformer Block, when m≠m k , no patch token compression is performed, and when m=m k , patch token compression is performed. Figure 2 is the flowchart of patch token compression in this embodiment. As Figure 2 shown, the specific steps of the model compression of the present application include:

[0048] S201: Calculate patch token importance score:

[0049] To analyze the impact of each patch token on the final output and determine its priority for retention or discarding, it is necessary to evaluate the importance of each patch token. This invention utilizes the self-attention mechanism of the Transformer model to calculate the interrelationships between each patch token and other patch tokens, thereby achieving patch token importance evaluation. Each token input to the visual Transformer model generates an "attention weight matrix" through self-attention calculation with other tokens. These weights represent the dependency and correlation of each token with other tokens in the model. The self-attention mechanism can be expressed by the following formula:

[0050]

[0051] in, These represent the query matrix, key matrix, and value matrix, respectively. The superscript T indicates transpose, and d... k d represents the dimension of the key. v The dimension of the value is represented by N, which represents the number of input features.

[0052] The formula for calculating the attention weight matrix A is as follows:

[0053]

[0054] Based on the above analysis, the method for calculating the importance score of the patch token in this invention is as follows:

[0055] Let the image embedding received by the current Transformer Block be... The 0th token Classification tokens 1 to N m The token is a patch token. j = 1, 2, ..., N m N m Indicates image embedding X m The number of patch tokens; denote the value matrix of the attention module input in the current Transformer Block. Where d v The dimension representing the values ​​is used to calculate the attention matrix. The importance score for each patch token is calculated using the following formula:

[0056]

[0057] Among them, A 0,n Represents category tokens With patch token Attention scores between Represents the value matrix V m Patch Token The corresponding value vector (i.e., the jth row vector) is || ||, which represents the norm.

[0058] S202: Patch token sampling:

[0059] Next, sampling is performed according to the importance of the token, and the token to be retained is dynamically selected. The specific method is as follows: let the image embedding obtained after being processed by the multi-head self-attention module and the superimposed normalization module in the current Transformer Block be According to the actual situation, set the compression ratio R, and calculate the number of patch token samples H m = R x N m According to the importance score of the patch token, H m patch tokens are sampled from N m patch tokens to form a patch token queue P, and the greater the importance score, the higher the probability of being selected.

[0060] In this embodiment, the method of quota sampling is used for patch token sampling, and the specific method is as follows:

[0061] N m patch tokens are sorted in descending order of importance score, then the first H m patch tokens are divided into the first group, and the last N m -H m patch tokens are divided into the second group, h m,1 patch tokens are extracted from the first group to form a queue h m,2 patch tokens are extracted from the second group to form a queue wherein h m,1 = γ x H m , h m,2 = (1-γ)H m , and γ represents a preset quota ratio, γ > 0.5. Then, the queues are merged to obtain the patch token queue P.

[0062] In this embodiment, the set is merged using an insertion merging method, and the specific method is as follows: the patch tokens in the queue are inserted into the queue in the order of an arithmetic sequence, wherein the common difference d is calculated as follows:

[0063]

[0064] wherein q1 represents a preset insertion starting position, denotes rounding down.

[0065] S203: patch token fusion:

[0066] The fusion ratio T is set according to the actual situation, and the number of fused tokens G is calculated m = T x H m Then, the patch token queue P is divided into two queues C1 and C2, wherein the queue C1 contains G m patch tokens, and the queue C2 contains H m -G m patch tokens. Each patch token in the queue C1 is traversed, the patch token with the highest similarity is selected from the queue C2, and the patch token is deleted from the queue C2, and then the following formula is used to fuse to obtain the fused patch token:

[0067]

[0068] wherein, denotes the gth patch token in the queue C1, g = 1, 2, …, G m , j g denotes the original serial number of the patch token , denotes the fused patch token, w g denotes the weight, denotes the patch token with the highest similarity to the patch token in the queue C2, denotes the original serial number of the patch token .

[0069] The classification token, the G m patch tokens fused and the remaining patch tokens in the set C2 are used to construct an image embedding and output to a subsequent module.

[0070] Figure 3 is a structural diagram of the improved Transformer Encoder of the present application. As Figure 3 indicated, after the improvement of the present application, two modules of token sampling and token fusion are added to the original Transformer Encoder. Figure 4 is a schematic diagram of the processing process of the block to be compressed in the present embodiment. As Figure 4As shown, in a certain layer to be compressed Block, after patch token sampling and fusion, the size of the output image embedding is smaller than the size of the input image embedding. After K as to be compressed Block, the size of the image embedding can be greatly reduced, thereby realizing the compression of the image embedding.

[0071] In order to better illustrate the technical effect of the present application, the present application is experimentally verified by using specific examples.

[0072] Four existing model compression methods are selected for comparison in this embodiment, including:

[0073] DynamicViT, see document "Y. Rao, Z. Liu, W. Zhao, J. Zhou, and J. Lu, Dynamic spatial sparsification for efficient vision transformers and convolutional neural networks [C]. 2023";

[0074] Evo-vit, see document "Y. Xu, Z. Zhang, M. Zhang, K. Sheng, K. Li, W. Dong, L. Zhang, C. Xu, and X. Sun, Evo-vit: Slow-fast token evolution for dynamic vision transformer [C], 2022";

[0075] EViT, see document "Y. Liang, C. Ge, Z. Tong, Y. Song, J. Wang, and P. Xie, Not all patches are what you need: Expediting vision transformers via token reorganizations, [C]. 2022";

[0076] Ia-red^2, see document "B. Pan, R. Panda, Y. Jiang, Z. Wang, R. Feris, and A. Oliva, Ia-red^2: Interpretability-aware redundancy reduction for vision transformers [C]. 2021".

[0077] In this embodiment, experiments are performed from the aspects of Top-1 Acc (model classification accuracy), Params (model parameter quantity), Flops (model calculation quantity), Throughput (model throughput) and the like. Table 1 is a comparison table of evaluation indexes when the present application and other existing methods are applied to the DeiT benchmark model.

[0078]

[0079] Table 1

[0080] As shown in Table 1, the top-1 accuracy, FLOPs and throughput of each model are tested in this embodiment. As can be seen from Table 1, on the standard model DeiT-S, the calculation cost is reduced by 37% when the present application is used, and the top-1 accuracy is only reduced by 0.1%. It is worth noting that on the standard model DeiT-B, the present application can reduce the calculation cost by 35% while keeping the top-1 accuracy unchanged, and increase the throughput by 1.52 times. At the same time, compared with other methods, the present application achieves a good balance between model accuracy and model running speed.

[0081] In order to verify the effectiveness of the token sampling and token fusion in the present application, comparative experiments are also performed. The comparison methods include average fusion (Average fusing) and maximum fusion (Max fusing), and the evaluation indexes include Top-1 Acc (model classification accuracy) and Flops (model calculation quantity). Table 2 is a comparison table of evaluation indexes when the token sampling and token fusion method in the present application and the comparison method are applied to the DeiT benchmark model.

[0082]

[0083] Table 2

[0084] As shown in Table 2, compared with the original method, the method proposed by the present application has obviously improved the model classification accuracy index under the same calculation quantity.

[0085] Although the above describes the specific embodiments of the present application in order to facilitate those skilled in the art to understand the present application, it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

Claims

1. A method for compressing a visual Transformer model, characterized in that, Includes the following steps: S1: Let M be the number of Transformer Blocks in the Transformer Encoder of the visual Transformer model. Based on actual needs, select K Transformer Blocks from the M Transformer Blocks as Blocks to be compressed. Let m be the original index of the kth block to be compressed. k k = 1, 2, ..., K; S2: Input the input image into the embedding layer of the visual Transformer model. The embedding layer generates an image embedding x and inputs it into the Transformer Encoder. The method for generating the image embedding is as follows: The input image is uniformly divided into N patches, and each patch is encoded to obtain an embedding vector. n = 1, 2, ..., N, D represents the dimension of the embedding vector, and the position of the patch is encoded to obtain the position vector e. n The patch representation vector z is obtained by superposition. n =f n +e n Then each patch represents the vector z. n The image embedding is constructed as a patch token. Where z cls Indicates a category token; S3: The Transformer Encoder extracts features from the image embedding and then outputs them to subsequent modules for classification. During feature extraction, for the m-th Transformer Block, when m ≠ m k If m = m, then no model compression is performed. k The following method is used for model compression: S3.1: Let the image embedding received by the current Transformer Block be... The 0th token Classification tokens 1 to N m The token is a patch token. j = 1, 2, ..., N m N m Indicates image embedding X m The number of patch tokens; denote the value matrix of the input to the multi-head self-attention module in the current Transformer Block. Where d v The dimension representing the values ​​is used to calculate the attention matrix. The importance score for each patch token is calculated using the following formula: Among them, A 0,n Represents category tokens With patch token Attention scores between Represents the value matrix V m Patch Token The corresponding value vector, || || denotes taking the norm; S3.2: Let the image embedding obtained in the current Transformer Block after processing by the multi-head self-attention module and the stacking normalization module be denoted as . Set the compression ratio R according to the actual situation, and calculate the number of patch token samples H. m =R×N m Based on the importance score of the patch token from N m patch token H was obtained from sampling m The patch tokens form a patch token queue P, and the higher the importance score, the higher the probability of being selected. S3.3: Set the fusion ratio T according to the actual situation, and calculate the number of fusion tokens G. m =T×H m Then, the patch token queue P is divided into two queues C1 and C2, where queue C1 contains G. m A patch token, queue C2 contains H m -G m Each patch token is a patch token; iterate through each patch token in queue C1, select the patch token with the highest similarity from queue C2 and delete it from queue C2, then fuse them using the following formula to obtain the fused patch token: in, This represents the g-th patch token in queue C1, where g = 1, 2, ..., G. m j g Patch token The original serial number, The w represents the merged patch token. g Indicates weight, This indicates that the patch token is in queue C2. The patch token with the highest similarity Patch token The original serial number; Classification tokens, fused G m patch token The remaining patch tokens in set C2 form an image embedding and are output to subsequent modules.

2. The visual Transformer model compression method according to claim 1, characterized in that, The original sequence number m of the Block to be compressed k The following formula is used for calculation: in, This indicates rounding down to the nearest integer.

3. The visual Transformer model compression method according to claim 1, characterized in that, In step S3.2, a quota sampling method is used to sample patch tokens. The specific method is as follows: N m patch token Sort the scores from highest to lowest importance, then select the top H... m The patch tokens are divided into the first group, and then N... m -H m The patch tokens are divided into a second group, from which the first h are extracted. m,1 A queue of patch tokens Extract the first h from the second group m,2 A queue of patch tokens Where h m,1 =γ×H m h m,2 =(1-γ)H m γ represents the preset quota ratio, γ > 0.5; then the queue The merged tokens result in a patch token queue P.

4. The visual Transformer model compression method according to claim 3, characterized in that, The queue The merge operation uses an insertion merge method, specifically: the queue is... The patch tokens are inserted into the queue from the beginning in an arithmetic sequence order. In this context, the tolerance d is calculated using the following formula: Where q1 represents the preset insertion start position, This indicates rounding down to the nearest integer.

Images