Visual model pruning optimization method and system based on visual cue word fine tuning

Abstract

The invention discloses a visual model pruning optimization method and system based on visual cue word fine tuning. The method comprises the steps that a to-be-pruned model is provided, the to-be-pruned model is a visual Transform model, the model comprises blocks of multiple levels, and the blocks of the previous level transmit image features to the blocks of the next level; the blocks are pruned and optimized in the transmission process, visual cue words are inserted into image features, the visual cue words reflect comprehensive features of feature marks which are cut off currently, and the blocks generate next-level image features based on previous-level image features containing the visual cue words. According to the technical scheme provided by the invention, a plug-and-play solution is provided for model lightweight, visual cue words are constructed to represent part of cut features to be transmitted backwards, obvious loss of model performance caused by model pruning is avoided, the model fine tuning process is accelerated, the performance loss in the model compression process is reduced, and the model compression efficiency is improved. Therefore, both high compression efficiency and low training overhead are realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a visual model pruning optimization method and system based on visual cue word fine-tuning. Background Art

[0002] The visual transformer model is a large model that performs well in the field of computer vision. In recent years, deep neural networks based on the transformer architecture have shown strong capabilities in various fields of computer vision. However, this comes with higher computing power requirements than traditional convolutional neural networks, which is obviously not friendly to edge devices. Therefore, how to compress the visual transformer model to obtain a model with high precision and low computing power requirements has become a key issue in the current academic community.

[0003] Model pruning is an important sub-direction of model compression. Current pruning methods for visual transformer models generally suffer from serious performance degradation or excessive computing power required in the model fine-tuning phase. Therefore, it is necessary to develop a new visual transformer pruning optimization method that is compatible with existing visual transformer pruning methods while achieving both high compression efficiency and low training overhead. Summary of the invention

[0004] In view of the deficiencies in the prior art, the object of the present invention is to provide a visual model pruning optimization method and system based on visual cue word fine-tuning.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention includes:

[0006] In a first aspect, the present invention provides a visual model pruning optimization method based on visual cue word fine-tuning, wherein the visual model is used for image recognition, and the visual model pruning optimization method comprises:

[0007] Providing a model to be pruned, wherein the model to be pruned is a visual Transformer model and is pre-trained, wherein the visual Transformer model includes multiple levels of blocks, wherein the blocks of the previous level transfer image features to the blocks of the next level, and the blocks of the first level extract initial image features based on training graphics;

[0008] Performing pruning optimization on the block during the transfer of the image features;

[0009] A visual cue word is inserted into the image feature, the visual cue word reflects the comprehensive features of the feature mark currently pruned, and the block generates a subsequent level of image features based on the previous level of image features containing the visual cue word.

[0010] In a second aspect, the present invention further provides a visual model pruning optimization system based on visual cue word fine-tuning, wherein the visual model is used for image recognition, and the visual model pruning optimization system comprises:

[0011] A model initialization module is used to provide a model to be pruned, wherein the model to be pruned is a visual Transformer model and is pre-trained, wherein the visual Transformer model includes multiple levels of blocks, wherein the blocks of the previous level transfer image features to the blocks of the next level, and the blocks of the first level extract initial image features based on training graphics;

[0012] A pruning optimization module, used for pruning the block during the transmission of the image feature;

[0013] The pruning optimization module also includes a prompt insertion unit, which is used to insert a visual prompt word into the image feature, wherein the visual prompt word reflects the comprehensive characteristics of the feature mark currently pruned, and the block generates a next-level image feature based on the previous-level image feature containing the visual prompt word.

[0014] Based on the above technical solution, compared with the prior art, the beneficial effects of the present invention at least include:

[0015] The technical solution provided by the present invention provides a plug-and-play solution for lightweighting the visual Transformer model. By constructing visual cue words to represent the pruned features and passing them backward, the obvious loss of model performance caused by model pruning is avoided, the model fine-tuning process is accelerated, and the performance loss during model compression is reduced, thereby achieving both high compression efficiency and low training overhead.

[0016] The above description is only an overview of the technical solution of the present invention. In order to enable those skilled in the art to more clearly understand the technical means of the present application and implement them according to the contents of the specification, the following is a description of the preferred embodiments of the present invention in conjunction with detailed drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a visual model pruning optimization method provided by a typical implementation case of the present invention. DETAILED DESCRIPTION

[0018] In view of the deficiencies in the prior art, the inventor of this case has proposed the technical solution of the present invention after long-term research and extensive practice. The technical solution, its implementation process and principle will be further explained as follows.

[0019] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0020] See also Figure 1 As shown, an embodiment of the present invention provides a visual model pruning optimization method based on visual cue word fine-tuning, wherein the visual model is used for image recognition, and the visual model pruning optimization method comprises the following steps:

[0021] Providing a model to be pruned, wherein the model to be pruned is a visual Transformer model and is pre-trained, wherein the visual Transformer model includes multiple levels of blocks, wherein the blocks of the previous level transfer image features to the blocks of the next level, and the blocks of the first level extract initial image features based on training graphics;

[0022] Performing pruning optimization on the block during the transfer of the image features;

[0023] Wherein, a visual cue word is inserted into the image feature, the visual cue word reflects the comprehensive features of the feature mark currently pruned, and the block generates a next-level image feature based on the previous-level image feature containing the visual cue word.

[0024] Regarding image features, in some embodiments, the image features include a class label, the visual cue word, and multiple image block features; the image block feature corresponds to any image block of the input image input into the visual Transformer model, and the class label is used to extract the global features of the input image.

[0025] In the prior art, the application of the visual Transformer model is divided into a pre-training stage, which is usually defined as Stage I, and subsequent training for specific needs, which is defined as Stage II, i.e. the traditional model fine-tuning stage. Currently, all visual transformer pruning methods that require fine-tuning are trained in the Stage II stage, and the training process can be expressed as follows:

[0026] Let the sliced ​​image input to the visual transformer be I = [I1, I2, ..., I m ], where m is the total number of blocks in the image. After mapping and position embedding, the image feature can be expressed as E = [E CT , E j |1≤j≤m], where E j =Emb(I j ), E CT is a learnable class tag (token), which is connected to E in a spliced ​​manner jPreviously, it was used to extract the global features of the image to obtain the final classification result. Now suppose the visual transformer has N blocks, then its feature extraction process can be expressed as,

[0027] Z 1 =L1(E;ρ1)

[0028] Z i =L i (Z i-1 ρ i-1 )i=2,3,…,N

[0029] Where L i Represents the i-th visual transformer block. The visual transformer block usually consists of a multi-head self-attention layer MSA(·) and a feed-forward neural network layer FFN(·), so the existing model pruning work can be simply expressed as:

[0030] L i (·)=P(FFN(MSA(·));ρi) or L i (·) = FFN(P(MSA(·); ρi))

[0031] Where P is the existing pruning method, and ρ is the pruning rate corresponding to this stage, which will be adjusted with the number of training rounds.

[0032] The key point of the present invention is that in the class token E CT After E j A learnable visual cue word VPT is inserted before the pruning to adaptively store the comprehensive features of the pruned tokens to alleviate the information loss caused by pruning.

[0033] At this point, the image feature E carrying the visual cue word is passed into the transformer module, and then the final E CT After the classification network, the final classification result is obtained. Specifically in some implementation schemes, the image features can be expressed as:

[0034] INSERT(E,VPT)=[E CT ,VPT,E j |1≤j≤m]

[0035] Among them, INSERT represents the image feature, E CT represents the class label, VPT represents the visual prompt word, E j represents the image block feature, j represents the sequence number of the image block, and m represents the total number of the image blocks.

[0036] Further in some embodiments, the image feature transfer process can be expressed as:

[0037] Z 1 =L1(INSERT(E,VPT);ρ1)

[0038] Z i =L i (INSERT(Z i-1 , VPT); ρ i-1 ))i=2,3,…,N

[0039] Z 1 represents the initial image features, Z i represents the rest of the image features, L represents the corresponding block, and ρ represents the pruning rate corresponding to the block. At this time, the image feature E carrying the visual cue word is passed into the transformer module, and then the final E CT After the classification network, the final classification result is obtained.

[0040] In some embodiments, the hyperparameter of the pruning optimization is the number of the visual cue words, and the dimension of the visual cue words is the same as the dimension of the class label.

[0041] In some embodiments, the block is consistent with the existing technical solutions, both of which include a feedforward neural network layer and a multi-head self-attention layer, and the pruning optimization is also expressed as:

[0042] L i (·)=P(FFN(MSA(·));ρi) or L i (·) = FFN(P(MSA(·); ρi))

[0043] Among them, FFN represents the feedforward neural network layer, MSA represents the multi-head self-attention layer, and P represents the pruning optimization strategy, including any one or more combinations of class label pruning, attention head pruning, and feedforward network pruning. Pruning technology is divided into two categories according to the pruning target: convolutional neural network pruning and transformer pruning. The technical solution provided by the present invention is only effective for transformer-based pruning methods, including the above-mentioned token pruning, attention head pruning, and feedforward network pruning.

[0044] In some embodiments, the generation process of the visual cue word is expressed as:

[0045] Perform random initialization to form the initial prompt word;

[0046] During the training process, the parameters of the initial prompt word are updated through a back-propagation algorithm.

[0047] Specifically, in practical applications, the generation of visual cue words can be randomly initialized and sampled from a normal distribution, which can be expressed as VPT~N(0,1).

[0048] Visual cues are not generated based on features, but during the training process, the parameters of VPT are updated through the back-propagation algorithm to maximize the performance of the model.

[0049]

[0050] Where η is the learning rate, is the gradient of the loss function with respect to L with respect to the visual cue word VPT.

[0051] Adaptive Adjustment: Visual cues can adaptively store and represent the comprehensive features of tokens that are removed during pruning, thereby reducing model complexity while maintaining model performance as much as possible.

[0052] In some embodiments, during the pruning optimization process, only the training of parameters related to the visual cue words is opened, and the remaining parameters are frozen. It is worth mentioning that it is different from the existing visual transformer pruning method. The method provided by the present invention only opens the training of model parameters related to visual cue words, and freezes other parameters to reduce the training overhead brought by the fine-tuning stage. The specific approach is that only the number of manually set cue words is a hyperparameter, and the dimension of the visual cue words is kept consistent with the dimension of the original token.

[0053] In specific applications, visual cues are used to compensate for information lost during pruning. Therefore, parameters directly related to visual cues are considered the most important in the optimization process and need to be opened for training, while other parameters, such as patch embedding, may have less impact on performance during pruning and can be frozen to reduce video memory usage.

[0054] As for how to evaluate the pruning effect, in some embodiments, the loss evaluation method of pruning optimization includes evaluating the relationship between the additional overhead caused by inserting the visual cue word and the performance loss alleviated by inserting the visual cue word.

[0055] In the specific technical solution, the calculation process of the additional overhead is expressed as:

[0056] FLOPs original =O(L 2 ·d+2·L·d·d ff );

[0057] FLOPs with VPT =O((L+n) 2·(d+n·d vpt )+2·(L+n)·(d+n·d vpt )·d ff );

[0058] FLOPs extra =FLOPs with VPT -FLOPs original ;

[0059] Among them, FLOPs extra represents the additional overhead, FLOPs original represents the raw expected cost, FLOPs with VPT represents the current expected cost, O(·) represents the time complexity or space complexity of the algorithm, L represents the length of the input sequence, that is, the number of tokens input to the Transformer model, d represents the embedding dimension of each token, and d ff It represents the hidden layer dimension in the feedforward neural network, usually a multiple of d, used for nonlinear transformation in each layer of the Transformer. n represents the number of tokens of the visual prompt word (VPT), ​​and d vpt Represents the feature dimension of each token in the visual prompt word (VPT);

[0060] The calculation process of the performance loss is expressed as:

[0061] Accuracy margin =Accuracy after -Accuracy before ;

[0062] Among them, Accuracy margin Indicates the performance loss, Accuracy after Indicates the current prediction accuracy, Accuracy before Represents the original prediction accuracy without pruning.

[0063] As a specific application example, the original pruning model will reduce the model operation cost. The larger the pruning rate, the smaller the model cost, but at the same time it will reduce the model performance, that is, the classification accuracy. The VPT we introduced will cause additional operation cost, so we will increase the pruning rate to balance the additional cost brought by VPT, making it the same as the cost of the original pruning model.

[0064] “Overhead” refers to the additional overhead of the original model in the forward process caused by adding n VPTs, which can be expressed in floating-point operations FLOPs. The FLOPs of the original Transformer block can be expressed as, and the FLOPs of the self-attention mechanism can be approximated as O(L 2 ·d), the FLOPs of the FFN layer can be approximated as O(2·L·d·d ff ), so the total FLOPs of the original Transformer block can be approximated as: FLOPs original =O(L 2 ·d+2·L·d·d ff ), and after adding n VPTs, the input feature dimension becomes d+n·d vpt , so after adding n VPTs, the total FLOPs can be approximated to FLOPs with VPT =O((L+n) 2 ·(d+n·d vpt )+2·(L+n)·(d+n·d vpt )·d ff ), so the additional overhead of introducing VPT can be defined as FLOPs extra =FLOPs with VPT -FLOPs original Performance loss refers to the difference between the classification accuracy of the original model and the classification accuracy of the pruned model, so it is expressed as: Accuracy margin =Accuracy after -Accuracy before .

[0065] In the technical solution provided by the present invention, the addition of visual cues is to enhance the model's tolerance to token loss, but it will also introduce additional reasoning overhead, which is contrary to the original intention of the pruning method to reduce the reasoning overhead. Therefore, it is necessary to maintain a balance between the additional overhead brought by the cues and the performance loss they alleviate.

[0066] Based on the above technical solution, the present invention achieves the goal of low-cost fine-tuning by only training inserted visual cue words, and uses learnable visual cue words to reduce the performance loss caused by visual transformer pruning, and is compatible with existing visual transformer pruning methods, so it can be plug-and-play.

[0067] In the above technical solution, the specific principle of the visual prompt word VPT is:

[0068] VPT is a token that is input into the transformer block at the same time as the feature token. It can interact with other tokens to optimize the feature extraction and fusion process during training.

[0069] The specific steps in actual operation are as follows:

[0070] Random Initialization: VPTs are initially randomly initialized in the model. This means that each cue word is randomly drawn from a probability distribution (such as a uniform distribution or a normal distribution). This initialization provides a starting point for the model to learn and adjust from.

[0071] Forward propagation: During training, input data (such as images) is fed into the model, including VPTs. These prompt words participate in the forward propagation along with the rest of the model, affecting the model’s output.

[0072] VPTs may be designed to interact with other features or tokens in the model to help the model capture and encode important visual information.

[0073] Calculate loss: The output of the model (such as the classification result) is compared with the true label and the loss function (such as cross entropy loss) is calculated. This loss value measures the difference between the model prediction and the actual result.

[0074] Back propagation: Use the back propagation algorithm to calculate the gradient of the loss function with respect to VPTs, excluding the gradients of the remaining tokens to reduce video memory overhead. This process starts from the output layer and goes backwards through each layer of the model until the gradients of all parameters are calculated.

[0075] The gradient represents the rate of change of the loss function with respect to each parameter, indicating how to adjust the parameters to minimize the loss.

[0076] Parameter update: Use an optimization algorithm (such as Adam) to update the parameters of the VPTs according to the calculated gradients.

[0077] Iterative training: Repeat the above process until the performance of the model on the training set reaches a satisfactory level, or the training reaches a predetermined number of iterations.

[0078] Correspondingly, an embodiment of the present invention further provides a visual model pruning optimization system based on visual cue word fine-tuning, wherein the visual model is used for image recognition, and the visual model pruning optimization system comprises:

[0079] A model initialization module is used to provide a model to be pruned, wherein the model to be pruned is a visual Transformer model and is pre-trained, wherein the visual Transformer model includes multiple levels of blocks, wherein the blocks of the previous level transfer image features to the blocks of the next level, and the blocks of the first level extract initial image features based on training graphics;

[0080] A pruning optimization module, used for pruning the block during the transmission of the image feature;

[0081] Specifically, the pruning optimization module also includes a prompt insertion unit, which is used to insert a visual prompt word into the image feature, wherein the visual prompt word reflects the comprehensive characteristics of the feature mark currently pruned, and the block generates a next-level image feature based on the previous-level image feature containing the visual prompt word.

[0082] Furthermore, an embodiment of the present invention also provides a readable storage medium, in which a computer program is stored. When the computer program is executed, the steps of the above-mentioned visual model pruning optimization method are executed.

[0083] The technical solution of the present invention is further described in detail below through several embodiments and in conjunction with the accompanying drawings. However, the selected embodiments are only used to illustrate the present invention, and do not limit the scope of the present invention.

[0084] Example 1

[0085] This embodiment illustrates the pruning optimization process of the visual Transformer model, which is specifically as follows:

[0086] Training library: PyTorch, a popular open source machine learning library widely used in computer vision and natural language processing tasks. Training parameter settings: Learning rate: 2e, using learning rate decay strategy. Batch size: 256, eight cards. Epoch number: 50, depending on the convergence speed of the model. Optimizer: Adam or SGD, usually Adam performs better in deep learning tasks. Loss function: cross entropy loss, for classification tasks. Pruning condition: calculate the importance of tokens, and arrange them from large to small in importance, and prune ρ% tokens.

[0087] The model is pruned and optimized during the feature transfer process. The image feature transfer process is expressed as:

[0088] Z 1 =L1(INSERT(E,VPT);ρ1)

[0089] Z i =L i (INSERT(Z i-1, VPT); ρ i-1 ))i=2,3,…,N

[0090] Z 1 represents the initial image features, Z i represents the rest of the image features, L represents the corresponding block, and ρ represents the pruning rate corresponding to the block. At this time, the image feature E carrying the visual cue word is passed into the transformer module, and then the final E CT After the classification network, the final classification result is obtained.

[0091] Based on the above implementation cases, it can be clearly seen that the technical solution provided by the embodiments of the present invention provides a plug-and-play solution for lightweighting the visual Transformer model. By constructing visual cue words to represent the pruned features and passing them backward, it avoids the obvious loss of model performance caused by model pruning, accelerates the model fine-tuning process and reduces performance loss during model compression, thereby achieving both high compression efficiency and low training overhead.

[0092] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A visual model pruning optimization method based on visual cue word fine-tuning, wherein the visual model is used for image recognition, and the visual model pruning optimization method comprises: Providing a model to be pruned, wherein the model to be pruned is a visual Transformer model and is pre-trained, wherein the visual Transformer model includes multiple levels of blocks, wherein the blocks of the previous level transfer image features to the blocks of the next level, and the blocks of the first level extract initial image features based on training graphics; Performing pruning optimization on the block during the transfer of the image features; The method is characterized in that a visual cue word is inserted into the image feature, the visual cue word reflects the comprehensive features of the feature mark currently being pruned, and the block generates a next-level image feature based on the previous-level image feature containing the visual cue word.

2. The visual model pruning optimization method according to claim 1, characterized in that: The image features include a class tag, the visual cue word, and a plurality of image block features; The image block feature corresponds to any image block of the input image input to the visual Transformer model, and the class label is used to extract the global feature of the input image.

3. The visual model pruning optimization method according to claim 2, characterized in that: The image features are expressed as: INSERT(E, VPT)=[E CT ,VPT,E i |1≤j≤m] Among them, INSERT represents the image feature, E CT represents the class label, VPT represents the visual prompt word, E j represents the image block feature, j represents the sequence number of the image block, and m represents the total number of the image blocks.

4. The visual model pruning optimization method according to claim 3, characterized in that: The transfer process of the image features is expressed as: With 1 =L1(INSERT(E, VPT);ρ1) Z i =L i (INSERT(Z i-1 ,VPT);ρ i-1 ))i=2,3,…,N Z 1 represents the initial image features, Z i Represents the remaining image features, L represents the corresponding block, and ρ represents the pruning rate corresponding to the block.

5. The visual model pruning optimization method according to claim 2, characterized in that: The hyperparameter of the pruning optimization is the number of the visual cue words, and the dimension of the visual cue words is the same as the dimension of the class label.

6. The visual model pruning optimization method according to claim 1, characterized in that: The block includes a feedforward neural network layer and a multi-head self-attention layer, and the pruning optimization is expressed as: L i (·) = P(FFN(MSA(·)); ρ i ) or L i (·) = FFN(P(MSA(·); ρ i )) Among them, FFN represents the feedforward neural network layer, MSA represents the multi-head self-attention layer, and P represents the pruning optimization strategy, including any one or more combinations of class label pruning, attention head pruning, and feedforward network pruning.

7. The visual model pruning optimization method according to claim 1, characterized in that: The generation process of the visual prompt words is expressed as: Perform random initialization to form the initial prompt word; During the training process, the parameters of the initial prompt word are updated through a back-propagation algorithm.

8. The visual model pruning optimization method according to claim 1, characterized in that: During the pruning optimization process, only the parameters directly related to the visual cue words are opened for training, and the remaining parameters are frozen.

9. The visual model pruning optimization method according to claim 1, characterized in that: The loss evaluation method for pruning optimization includes evaluating the relationship between the additional overhead caused by inserting the visual prompt word and the performance loss alleviated by inserting the visual prompt word; The calculation process of the additional overhead is expressed as: FLOPs original =O(L 2 ·d+2·L·d·d ff ); FLOPs withVPT =0((L+n) 2 ·(d+n·d vpt )+2·(L+n)·(d+n·d vpt )·d ff ); FLOPs extra =FLOPs with VPT -FLOPs original ; Among them, FLOPs extra represents the additional overhead, FLOPs original represents the raw expected cost, FLOPs with VPT represents the current expected overhead, O(·) represents the time complexity or space complexity of the algorithm, L represents the length of the input sequence, d represents the embedding dimension of each tag, and d ff represents the hidden layer dimension in the feedforward neural network, n represents the number of tags of the visual cue word, d vpt Representing the feature dimension of each token in the visual cue word; The calculation process of the performance loss is expressed as: Accuracy margin =Accuracy after -Accuracy before ; Among them, Accuracy margin Indicates the performance loss, Accuracy after Indicates the current prediction accuracy, Accuracy before Represents the original prediction accuracy without pruning.

10. A visual model pruning optimization system based on visual cue word fine-tuning, wherein the visual model is used for image recognition, and the visual model pruning optimization system comprises: A model initialization module is used to provide a model to be pruned, wherein the model to be pruned is a visual Transformer model and is pre-trained, wherein the visual Transformer model includes multiple levels of blocks, wherein the blocks of the previous level transfer image features to the blocks of the next level, and the blocks of the first level extract initial image features based on training graphics; A pruning optimization module, used for pruning the block during the transmission of the image feature; It is characterized in that the pruning optimization module also includes a prompt insertion unit, which is used to insert a visual prompt word into the image feature, and the visual prompt word reflects the comprehensive characteristics of the feature mark currently pruned, and the block generates a next-level image feature based on the previous-level image feature containing the visual prompt word.

Images