Vision transformer model acceleration algorithm based on life cycle prediction

Abstract

The application discloses a visual Transformer model acceleration algorithm based on life cycle prediction, comprising the following steps: 1) making the Transformer model perform forward propagation of the first half part; 2) smoothing a life cycle function and constructing a visual Transformer weight conversion module; 3) constructing the visual Transformer weight conversion module; 4) model inference; and 5) training the model. The application uses a life cycle prediction module to predict the life cycle of each input picture block of the visual Transformer model. When performing model inference, the application discards the picture block in time according to the predicted life cycle, thereby reducing unnecessary inference calculation and achieving the effect of inference acceleration of the model. When training the model, the application converts the non-differentiable picture block life cycle into a differentiable weight function through the weight conversion module, and thus the end-to-end training of the model is facilitated.

Description

Technical Field

[0001] This invention relates to acceleration algorithms for computer vision models, and more particularly to acceleration algorithms for visual Transformers. Background Technology

[0002] With the continuous advancement and maturation of deep learning technology, numerous deep learning models have emerged in the field of computer vision. Since the introduction of ViT, the Visual Transformer model has achieved great success in multiple tasks, including image recognition, object detection, and semantic segmentation. Despite such widespread and significant success, the Visual Transformer model still faces some problems and challenges.

[0003] First, although the visual Transformer model has achieved significant breakthroughs in evaluation metrics across various computer vision tasks, it requires a huge amount of computation to support model inference, which severely limits the deployment and application of the Transformer model on energy-sensitive mobile and edge devices. Summary of the Invention

[0004] The present invention aims to overcome the above-mentioned defects of the prior art and provide an accelerated algorithm for visual Transformer models based on lifecycle prediction.

[0005] An accelerated algorithm for visual Transformer models based on lifecycle prediction includes the following steps:

[0006] (1) Make the Transformer model perform the first half of the forward propagation;

[0007] (11) Divide the input image into N image blocks of the same size {z i} i∈I And use linear mapping to map the image patch {z i} i∈I Mapped to the initial image patch feature vector {x i ′ (0)} i∈I .

[0008] (12) Using position embedding vectors The image patch feature vector is obtained by adding it to the image patch feature vector after position embedding.

[0009] (13) Perform LayerNorm regularization, and then feed the regularized feature vectors into the multi-head self-attention module:

[0010]

[0011] in It is the dot product of the key feature and the query feature.

[0012] (14) After passing the features through the multi-head self-attention module, the skip-layer connection method is used to add the input feature vector of LayerNorm to the output feature vector of the multi-head self-attention module.

[0013] (2) Smooth the lifecycle function and construct the visual Transformer weight transformation module. The steps are as follows:

[0014] (21) is the feature of each input image patch of the Visual Transformer. LayerNorm normalization is obtained

[0015]

[0016] Where i is the index of the image patch, cls is the index of the CLS identifier patch, I is the index set containing all ordinary image patches, x represents the image patch feature, and t base For the insertion layer index of the lifecycle prediction module, This represents the features of all image blocks in layer t.

[0017] (22) Use a bilinear function with trainable weights W to obtain the global importance score of each image patch, and use this global importance score as the image patch feature. Similarity s between and [CLS] identifier i :

[0018]

[0019] (23) Calculate the global importance score {s} i} i∈I∪{cls} The mean μ and variance σ of the global importance scores are calculated, and the lifetime τ of the image patch is obtained by re-standardizing the global importance scores. i :

[0020]

[0021] (3) Construct the visual Transformer weight transformation module, the steps are as follows:

[0022] (31) Define an indicator function to indicate whether the i-th image patch is discarded at layer t as follows: Where H(t) is the unit step function.

[0023] (32) will Insert each layer of the visual Transformer's self-attention module:

[0024]

[0025] where is the query-key inner product result between the i-th and j-th image patches, exp is the exponential function, and N is the total number of image patches.

[0026] (33) Fitting using the Sigmoid function after affine transformation where U is a parameter that adjusts the steepness of the Sigmoid function curve:

[0027]

[0028] (34) In the self-attention module, replace with β i (t):

[0029]

[0030] (4) Model inference;

[0031] (41) For different input images, fix different layers t of the vision Transformer, discard a fixed number n t of image patches, and ensure that for 1 ≤ t1 < t2 ≤ T, there is Here, T represents the total number of layers of the model.

[0032] (42) Calculate the target mean μ and target variance σ in the life cycle prediction module:

[0033]

[0034]

[0035] (43) Set the same image patch retention rate ρ for each layer of the model, such that for all layer indices t i ≤ t′ < t i+1 , all satisfy and

[0036] (44) In each layer of the vision Transformer, by sorting the life cycles of all image patches, discard those image patches with smaller life cycles, and artificially set the life cycle of the [CLS] image patch to infinity. By such settings, unnecessary image patches are deleted at appropriate positions to achieve the acceleration of the vision Transformer.

[0037] (5) Train the model;

[0038] (51) Load the pre-trained model weights into the visual Transformer backbone network and randomly initialize the lifecycle prediction module and the weight transformation module.

[0039] (52) Freeze the model parameters of the visual Transformer backbone network and train the remaining modules until the model converges.

[0040] (53) Unfreeze the model parameters of the visual Transformer backbone network and freeze the training of the remaining modules until convergence.

[0041] This invention accelerates model inference for the Visual Transformer model in the field of deep learning computer vision. The invention mainly comprises two modules: an image patch lifecycle prediction module and a weight transformation module. Through these two modules, the invention achieves accelerated model inference with minimal degradation of the Visual Transformer model's performance. The lifecycle prediction module predicts the lifecycle for each input image patch of the Visual Transformer model. During model inference, the invention discards image patches in a timely manner based on the predicted lifecycle, thereby reducing unnecessary inference computation and accelerating model inference. During model training, the invention uses the weight transformation module to convert the non-differentiable image patch lifecycle into a differentiable weight function, enabling convenient end-to-end model training.

[0042] The advantage of this invention is that it enables the visual Transformer model to significantly improve the inference speed with minimal loss of accuracy, thereby facilitating the deployment of the visual Transformer model in real-time sensitive scenarios. Attached Figure Description

[0043] Figure 1 This is a flowchart of the present invention.

[0044] Figure 2 This is a structural diagram of the present invention. Detailed Implementation

[0045] The following describes an image classification method that applies the lifecycle prediction-based visual Transformer model acceleration algorithm of this invention:

[0046] (1) Input the image to be classified into the model for the first part of forward propagation.

[0047] (11) Divide the image to be classified into N image blocks of the same size {z i} i∈I And use linear mapping to map the image patch {z i} i∈IMapped to the initial image patch feature vector {x i ′ (0)} i∈I .

[0048] (12) Using image patch position embedding vectors The image patch feature vector is obtained by adding it to the image patch feature vector after position embedding.

[0049] (13) Perform LayerNorm regularization, and then feed the regularized feature vectors into the multi-head self-attention module:

[0050]

[0051] (14) of which It is the dot product of the key feature and the query feature.

[0052] (15) After passing the features through the multi-head self-attention module, the skip connection method is used to add the input feature vector of LayerNorm to the output feature vector of the multi-head self-attention module.

[0053] (2) Smooth the lifecycle function and construct the visual Transformer weight transformation module for image classification tasks. The steps are as follows:

[0054] (21) is the feature of each input image patch of the Visual Transformer. LayerNorm normalization is obtained

[0055]

[0056] Where i is the index of the image patch, cls is the index of the CLS identifier patch, I is the index set containing all ordinary image patches, x represents the image patch feature, and t base For the insertion layer index of the lifecycle prediction module, This represents the features of all image blocks in layer t.

[0057] (22) Use a bilinear function with trainable weights W to obtain the global importance score of each image patch, and use this global importance score as the image patch feature. Similarity s between and [CLS] identifier i :

[0058]

[0059] (23) Calculate the global importance score {s} i} i∈I∪{cls}The mean μ and variance σ, and re-normalize the global importance score to obtain the life cycle τ of the image patch i :

[0060]

[0061] (3) Construct a visual Transformer weight conversion module for the image classification task, the steps are as follows:

[0062] (31) Define the indicator function for representing whether the i-th image patch is discarded at the t-th layer as where H(t) is the unit step function.

[0063] (32) Insert into each self-attention module of the visual Transformer:

[0064]

[0065] where is the query-key inner product result between the i-th image patch and the j-th image patch,

[0066] exp is the exponential function, and N is the total number of image patches.

[0067] (33) Use the affine-transformed Sigmoid function to fit where U is the parameter that adjusts the steepness of the Sigmoid function curve:

[0068]

[0069] (34) In the self-attention module, replace with β i (t):

[0070]

[0071] (4) Image classification model inference

[0072] (41) For different input images, fix different layers t of the visual Transformer, discard a fixed number n t of image patches, and ensure that for 1 ≤ t1 < t2 ≤ T, there is Here T represents the total number of layers of the model.

[0073] (42) Calculate the target mean μ and target variance σ in the life cycle prediction module:

[0074]

[0075]

[0076] (43) Set the same image patch retention rate ρ for each layer of the model, so that all layer indices t i ≤t′ <t i+1 All are satisfied as well as

[0077] (44) In each layer of the visual Transformer, by sorting the lifecycles of all image blocks, discarding those with shorter lifecycles, and artificially setting the lifecycle of the [CLS] image block to infinity, unnecessary image blocks are deleted at appropriate locations, thus accelerating the visual Transformer.

[0078] (5) Training the image classification model

[0079] (51) Load the pre-trained image classification model weights into the visual Transformer backbone network, and randomly initialize the lifecycle prediction module and the weight transformation module.

[0080] (52) Freeze the model parameters of the visual Transformer backbone network for image classification and train the remaining modules until the model converges.

[0081] (53) Unfreeze the model parameters of the visual Transformer backbone network for image classification and freeze the training of the remaining modules until convergence.

[0082] (6) Obtain image classification results

[0083] (61) Input the feature vector corresponding to the CLS identifier of the last layer into the linear layer to obtain the classification vector.

[0084] (62) Sort the components of the classification vector to obtain the largest component, and the corresponding category is the category of the image to be classified.

[0085] The table below shows the image classification results of the image classification method based on the lifecycle prediction-accelerated visual Transformer model on DeiT-{S,B}:

[0086]

[0087] The data in the table show that the image classification method based on the lifecycle prediction-accelerated visual Transformer model can effectively improve model speed and reduce the loss of model accuracy.

Claims

1. A visual Transformer model acceleration algorithm based on life cycle prediction, comprising the following steps: (1) Make the Transformer model forward propagate the first half; (11) dividing the input picture into picture blocks of the same size and mapping the picture blocks into the first picture block feature vector using a linear mapping; (12) using a position embedding vector adding the picture block feature vector and the position embedding vector to obtain a picture block feature vector after position embedding ; (13) to perform LayerNorm regularization and send the regularized feature vector into the multi-head self-attention module: wherein is the dot product of the key and query features; (14) After passing through the multi-head self-attention module, use the skip-layer connection method to add the input feature vector of LayerNorm to the output feature vector of the multi-head self-attention module; (2) Smooth the life cycle function and build a visual Transformer weight conversion module, the steps are as follows: (21) for each input picture block feature of the visual Transformer LayerNorm normalization is performed to obtain : wherein is an index of a picture block, is an index of a CLS identifying block, is a set of indices containing all normal picture blocks, represents a picture block feature, is an insertion layer index of a life cycle prediction module, represents a first layer of all picture block features; (22) using a bilinear function with trainable weights to obtain a global importance score for each picture block and using this global importance score as a picture block feature and between the similarity : (23) calculating a global importance score of the mean and variance and re-normalizing the global importance score to obtain a life span of the picture block : (3) Build a visual Transformer weight conversion module, the steps are as follows: (31) defining an indicator function for indicating whether a picture block at a layer is discarded or not as where is a unit step function; and ​ (32) to inserting a self-attention module into each layer of a vision transformer: wherein is the query-key inner product result between the th picture block and the th picture block, exp is the exponential function, is the total number of picture blocks; (33) Fitting using an affine transformed Sigmoid function where is a parameter that regulates the steepness of the Sigmoid function curve: (34) in the self-attention module, replace with : (4) Model inference; (41) For different input images, fix the different layers of the visual Transformer. Discard a fixed number The image blocks, and ensure that for ,have ;here Indicates the total number of layers in the model; (42) Target mean in the compute life cycle prediction module and target variance : (43) setting the same picture block reservation rate for each layer of the model so that all layer indices satisfy and ; (44) In each layer of the visual Transformer, by sorting the life cycle of all image blocks, discard those image blocks with small life cycle, and artificially set the life cycle of the [CLS] image block to infinity; (5) Train the model; (51) Load the pre-trained model weight to the visual Transformer backbone network, and randomly initialize the life cycle prediction module and the weight conversion module; (52) Freeze the model parameters of the visual Transformer backbone network, train the remaining modules until the model converges; (53) Unfreeze the model parameters of the visual Transformer backbone network, and freeze the remaining modules for training until convergence.

Images