Pruning method and system for CLIP model

Abstract

The invention discloses a pruning method and system for a CLIP model. The pruning method for the CLIP model comprises the following steps: acquiring a target data set; a sliding window is preset, the sliding window comprises a pruning unit and an updating unit, the pruning unit corresponds to the ith layer of the target CLIP model, the updating unit corresponds to the (i + 1) th layer of the target CLIP model, the number of the layers of the target CLIP model is n, and i is larger than or equal to 1 and smaller than n; and inputting the target data set into a target CLIP model, pruning the ith layer and updating the (i + 1) th layer in sequence according to the sliding window until the nth layer is pruned, and obtaining a pruned CLIP model. According to the method, the next layer is updated and compensated in real time while the pruning operation is carried out, so that the precision loss is kept at the minimum level in the pruning process, and the accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of pre-training model compression, and in particular to a pruning method and system for a CLIP model. Background Art

[0002] The CLIP (Contrastive Language-Image Pre-Training) model is a pre-training model that integrates image and text modalities. Its core is to achieve the alignment of image and text modalities in a unified space. The CLIP model trains the image encoder and text encoder by applying the contrastive loss function on a large-scale image-text pairing dataset (data volume ranges from 400 million to 2 billion pairs), thereby achieving excellent image-text alignment performance. The CLIP model is not only suitable for classification and recall tasks, but also because it is trained on massive data, the features extracted by its encoder can be widely used in various downstream tasks. Recently, the CLIP model has been applied in the medical field, for example, in the classification and detection tasks of echocardiography, showing its potential in disease classification and detection.

[0003] Model pruning technology aims to reduce the model size and speed up the model reasoning by reducing model parameters. Since researchers pursue better convergence or higher performance in the training phase when building models, most current models are over-parameterized. Based on this premise, the core strategy of model pruning is to identify and remove unnecessary parameters in the model to achieve the goal of reducing the number of model parameters and the amount of reasoning calculations. When performing pruning operations, researchers strive to retain the original performance of the model as much as possible so that the model can be deployed on edge computing devices or save computing resources in the cloud.

[0004] TinyCLIP is a technique for structured compression of CLIP, which improves the Attention and FFN modules of CLIP through a model distillation method. This method uses a learnable mask to identify and retain key parts, and combines sparsity and distillation alignment losses. After that, the encoder output is fused using a teacher model and multiplication operations, and a progressive distillation strategy is used to complete the distillation. However, this method has a large demand for GPU resources during the distillation process and the distillation time is long.

[0005] Mope-CLIP is a structured pruning method that does not require a learnable mask. It evaluates the importance of modules through the module-wise pruning error (MOPE), which is achieved by setting module parameters to zero and evaluating performance fluctuations. The method is divided into two stages: pre-training and fine-tuning. The pre-training stage performs one-time pruning and knowledge distillation, while the fine-tuning stage performs pruning in width and depth dimensions and uses a variety of loss functions for knowledge distillation. Mope-CLIP relies on global distillation, which increases GPU resources and distillation time.

[0006] In the process of implementing the prior art, the inventor discovered that:

[0007] Existing technologies have huge demands on GPU resources. Given the need for large batch sizes during the distillation process, for example, TinyCLIP requires 32 A100 GPUs during the distillation process, while MopeCLIP requires 8 V100 GPUs. In addition, existing technologies have high demands on time. Specifically in the pre-training stage, the MopeCLIP model takes about 320 GPU hours, while the TinyCLIP model takes more than a thousand GPU hours. As for downstream tasks, TinyCLIP is not involved, and its work focuses on the compression of general CLIP models. When MopeCLIP processes image and text recall tasks, if its standard compression strategy is adopted, it will take about 320 GPU hours; if the width-first and then depth-first pruning method is adopted, it will take 640 GPU hours. Summary of the invention

[0008] The main purpose of the present invention is to provide a pruning method and system for a CLIP model, thereby overcoming the deficiencies in the prior art.

[0009] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention includes:

[0010] A first aspect of the present invention provides a pruning method for a CLIP model, comprising the following steps:

[0011] Get the target dataset;

[0012] A sliding window is preset, the sliding window includes a pruning unit and an updating unit, the pruning unit corresponds to the i-th layer of the target CLIP model, the updating unit corresponds to the i+1-th layer of the target CLIP model, the target CLIP model has n layers, 1≤i<n;

[0013] The target data set is input into the target CLIP model, and according to the sliding window, the i-th layer is pruned in order from 1 to n-1, and the i+1-th layer is updated until the n-th layer is pruned, thereby obtaining a pruned CLIP model.

[0014] In some more specific solutions, the obtaining of the target data set specifically includes:

[0015] Collecting image data;

[0016] Extracting label information corresponding to the image data, and mapping the label information to obtain a class name list;

[0017] According to the class name list, construct text to obtain text data;

[0018] Construct a mapping relationship between image data and text data to form several pairs of data as the target data set.

[0019] Further, the text data is obtained by using the text encoder of the target CLIP model, and the i-th layer of the text encoder is pruned in sequence according to the sliding window, and the i+1-th layer of the text encoder is updated until the n-th layer of the text encoder is pruned, thereby obtaining a pruned text encoding model;

[0020] Acquire image data using the image encoder of the target CLIP model, prune the i-th layer of the image encoder in sequence according to the sliding window, and update the i+1-th layer of the image encoder until the n-th layer of the image encoder is pruned, thereby obtaining a pruned image coding model;

[0021] The pruned text encoding model is integrated with the image encoding model to obtain the pruned CLIP model.

[0022] Furthermore, the pruning method for the CLIP model further includes:

[0023] The plurality of pairs of data are sequentially input into the target CLIP model.

[0024] In some more specific solutions, the pruning method is performed in multiple rounds of iterations, specifically including:

[0025] For the first iteration,

[0026] Get the standard initial value, which includes at least the standard input Z of the first layer and the standard output Z of the second layer of the target CLIP model. 2 ;

[0027] Prune the first layer of the target CLIP model to obtain the first pruned layer;

[0028] Input the standard input Z into the first pruning layer to obtain the pruned biased output Z 1 ', prune the biased output Z 1 'Input the second layer to get the fine-tuned biased output Z 2 ”;

[0029] Compare standard output Z 2 and fine-tuning the biased output Z 2 The difference between ” forms the loss value, and the second layer is updated according to the loss value to obtain the second fine-tuning layer;

[0030] For the i-th iteration, where i is incremented from 2 to n-1,

[0031] Set standard output Z i First pass through the i+1th layer and get the standard output Z i+1 ,

[0032] Prune the i-th fine-tuning layer to obtain the i-th pruned layer;

[0033] Prune the biased output Z i-1 'Input the i-th pruning layer to get the pruned biased output Z i ';

[0034] Prune the biased output Z i 'Input the i+1th layer to obtain the fine-tuned biased output Z i+1 ”;

[0035] Compare the standard output Z of layer i+1 i+1 With fine-tuning biased output Z i+1 The difference between the two forms the loss value, and the i+1th layer is updated to obtain the i+1th fine-tuning layer, where the standard output Z i+1 The standard output from layer i+1 of the original model.

[0036] Furthermore, the process of pruning the i-th fine-tuning layer specifically includes:

[0037] The i-th fine-tuning layer is traversed to determine the attention layer and the feedforward layer of the i-th fine-tuning layer; the attention layer and the feedforward layer of the i-th fine-tuning layer are pruned using Wanda or SparseGPT to obtain the i-th pruned layer.

[0038] Furthermore, the hook function is used to obtain the standard output Z of the second layer. 2 .

[0039] Furthermore, the process of updating the i+1th layer specifically includes:

[0040] Based on the MSE loss function, calculate the standard output Z i+1 With fine-tuning biased output Z i+1"The deviation loss value;

[0041] According to the deviation loss value, the parameters of the i+1th layer are optimized to minimize the standard output Z i+1 With fine-tuning biased output Z i+1 ” Deviation between .

[0042] Furthermore, the pruning method for the CLIP model also includes:

[0043] Setting hyperparameters, wherein the hyperparameters include at least an initial learning rate and a number of iterations;

[0044] Set the training termination threshold;

[0045] Based on the hyperparameters and the training termination threshold, the Adam optimizer is used to optimize the loss function.

[0046] A second aspect of the present invention provides a pruning system for a CLIP model, comprising:

[0047] An acquisition module is used to acquire a target data set;

[0048] A setting module, used for presetting a sliding window, wherein the sliding window includes a pruning unit and an updating unit, wherein the pruning unit corresponds to the i-th layer of the target CLIP model, and the updating unit corresponds to the i+1-th layer of the target CLIP model, wherein the target CLIP model has n layers, 1≤i<n;

[0049] The pruning module is used to input the target data set into the target CLIP model, and prune the i-th layer in the order of i from 1 to n-1 according to the sliding window, and update the i+1-th layer until the n-th layer is pruned, thereby obtaining a pruned CLIP model.

[0050] Compared with the prior art, the advantages of the present invention include at least:

[0051] First, in a pruning method for a CLIP model provided in the present invention, the target data set is input into the target CLIP model, and the i-th layer is pruned and the i+1-th layer is updated in turn according to the sliding window, so as to compensate for the loss in the CLIP model pruning process in real time, effectively avoiding the heavy burden brought by the full model distillation, and preventing the model performance from dropping significantly after pruning, thereby ensuring that the accuracy loss after pruning is small.

[0052] Second, the pruning method for the CLIP model provided in the present invention utilizes the advantages of Wanda or SparseGPT fast pruning and migrates it to the target CLIP model for application, which can significantly improve the speed of the pruning process and further improve the overall work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flowchart of a pruning method for a CLIP model provided by an embodiment of the present invention;

[0054] Figure 2 is a flowchart of a pruning method for a CLIP model provided by an embodiment of the present invention;

[0055] Figure 3 It is an expanded diagram of a pruning process of a pruning method for a CLIP model provided by an embodiment of the present invention;

[0056] Figure 4 is a line graph of the average previous accuracy and unstructured pruning sparsity of the CLIP-B-32 model provided by an embodiment of the present invention using four pruning methods on six data sets;

[0057] Figure 5 is a line graph of the average top five accuracy and unstructured pruning sparsity of the CLIP-B-32 model provided by an embodiment of the present invention using four pruning methods on six data sets;

[0058] Figure 6 It is a line graph of the previous accuracy and unstructured pruning sparsity of the CLIP-L-14 model fine-tuned on Cifar100 provided by an embodiment of the present invention using four pruning methods on the Cifar100 dataset;

[0059] Figure 7 This is a line graph of the top five accuracy of the CLIP-L-14 model fine-tuned on Cifar100 provided by an embodiment of the present invention using four pruning methods on the Cifar100 dataset and the unstructured pruning sparsity. DETAILED DESCRIPTION

[0060] In view of the deficiencies in the prior art, the inventor of this case has proposed the technical solution of the present invention after long-term research and extensive practice. The technical solution, its implementation process and principle will be further explained as follows.

[0061] See also Figure 1-Figure 2 ,A pruning method for the CLIP model disclosed in this application mainly includes two stages.

[0062] The first stage: data preprocessing process, see Figure 2The left border of the data. By preprocessing the data, that is, cleaning and formatting the raw data to meet specific requirements and standards, it can be directly input into the CLIP model. The second stage: pruning the CLIP model. The whole process runs in a sliding window manner. See Figure 2 Specifically, the first layer of the model is pruned, and then the second layer is fine-tuned using the mean square error (MSE) loss function to restore its performance. This process continues until the last layer of the model to obtain the final pruned model. Figure 2 , explains the pruning process, including pruning and parameter updating, where the loss function involved in parameter updating.

[0063] Please refer to Figure 1 , a pruning method for a CLIP model disclosed in this embodiment includes the following steps:

[0064] S100: Obtain a target data set.

[0065] Taking the classification task as an example, the dataset used is an open source dataset, which consists of image data and label information. When pruning the CLIP model, the target dataset consists of several data pairs, each of which contains text data and image data. The specific acquisition process is as follows:

[0066] First, by loading the open source dataset, the RandomSampler is used to randomly sample the dataset and select 500 images as image data. Then, the Dataloader is used to read the data and its collate_fn function is customized. In this function, the built-in Processor of the CLIP model is used to preprocess the image data to obtain a processed image calibration dataset.

[0067] In the classification task, since there is no text sentence directly available, only label information exists. Collect the label information corresponding to the 500 images in the first step and map these label information to a list of class names (if the label information is directly expressed as a class name, this step can be omitted).

[0068] After obtaining the class name list, the text is constructed according to the prompt template provided by CLIP. For each class name, a prompt template is randomly selected to construct the text, thereby generating 500 sentences. Subsequently, these 500 sentences are processed by the CLIP built-in Tokenizer to obtain text calibration data.

[0069] Finally, a mapping relationship between the image calibration data set and the text calibration data is constructed to form several pairs of data as the target data set.

[0070] In some more specific scenarios, the target dataset uses six open source datasets. These open source datasets can be directly obtained on the Huggingface platform. Specifically, these six open source datasets include:

[0071] CIFAR100: A widely used natural dataset with 100 classes and a test set size of 10,000.

[0072] FLOWER102: A flower dataset with 102 classes and a test set size of 6149.

[0073] FOOD101: A food dataset with 101 classes and a test set size of 25250.

[0074] EuroSAT: A remote sensing image classification dataset with 10 categories and a test set size of 2700.

[0075] SUN397: A large-scale scene classification dataset with 397 categories and a test set size of 21750.

[0076] Resisc45: A remote sensing image classification dataset with 45 categories and a test set size of 6300.

[0077] Before pruning, you need to complete the following preparations:

[0078] First, register a hook function for the target CLIP model and obtain the standard initial value through the function. The standard initial value includes the standard input Z of the first layer of the target CLIP model, that is, the pre-pruning input of the first layer, the standard output Z of the second layer 2 , as the initial value for subsequent pruning process iterations.

[0079] The video memory capacity occupied by these two standard initial values ​​is approximately: (500*77*512+50*500*768)*2Byte≈0.075GB. For a larger CLIP model, this part of the video memory will not exceed 0.15GB.

[0080] S200: Preset a sliding window. The sliding window includes a pruning unit and an updating unit, wherein the pruning unit corresponds to the i-th layer of the target CLIP model, and the updating unit corresponds to the i+1-th layer of the target CLIP model, wherein the target CLIP model has n layers, 1≤i<n.

[0081] Specifically, refer to Figure 2, the target CLIP model includes a text encoder and an image encoder. This target CLIP model has n layers, where the text encoder and the image encoder each have n layers. X is used to represent the output and output value of the text encoder. Y is used to represent the output and output value of the image encoder. Therefore, the standard input Z of the first layer includes X and Y respectively, and the standard output Z of the second layer 2 Include X 2 , Y 2 In addition, for further processing and analysis, two sliding windows can be set, which correspond to the text encoder and the image encoder respectively.

[0082] S300: Input the target data set into the target CLIP model, prune the i-th layer in order from 1 to n-1 according to the sliding window, and update the i+1-th layer until the n-th layer is pruned to obtain a pruned CLIP model. The specific process includes:

[0083] Input the 500 pairs of calibration data collected previously into the target CLIP model. It should be noted that the pair-by-pair input method is used here to minimize the demand for GPU video memory;

[0084] Acquire text calibration data using the text encoder of the target CLIP model, prune the i-th layer of the text encoder in sequence according to the sliding window, and update the i+1-th layer of the text encoder until the n-th layer of the text encoder is pruned, thereby obtaining a pruned text encoding model;

[0085] Acquire image calibration data using the image encoder of the target CLIP model, prune the i-th layer of the image encoder in sequence according to the sliding window, and update the i+1-th layer of the image encoder until the n-th layer of the image encoder is pruned, thereby obtaining a pruned image coding model;

[0086] The pruned text encoding model is integrated with the image encoding model to obtain the pruned CLIP model.

[0087] In this solution, the pruning method is performed in multiple rounds of iterations, specifically including:

[0088] For the first iteration:

[0089] Get the standard initial value, which includes at least the standard input Z of the first layer and the standard output Z of the second layer of the target CLIP model. 2 ;

[0090] Prune the first layer of the target CLIP model to obtain the first pruned layer;

[0091] Input the standard input Z into the first pruning layer to obtain the pruned biased output Z 1 ', prune the biased output Z 1 'Input the second layer to get the fine-tuned biased output Z 2 ”;

[0092] Compare standard output Z 2 and fine-tuning the biased output Z 2 The difference between ” forms the loss value, and the second layer is updated according to the loss value to obtain the second fine-tuning layer.

[0093] For the second iteration:

[0094] Set standard output Z 2 First pass through the third layer and get the standard output Z 3 ;

[0095] Prune the second fine-tuning layer to obtain a second pruned layer;

[0096] Prune the biased output Z 1 'Enter the second pruning layer to get the pruned biased output Z 2 ';

[0097] Prune the biased output Z 2 'Enter the third layer to obtain the fine-tuned biased output Z 3 ”;

[0098] Compare the standard output Z of the third layer 3 With fine-tuning biased output Z 3 The difference between ” forms the loss value, and the third layer is updated to obtain the third fine-tuning layer.

[0099] In general, for the i-th iteration, where i is incremented from 2 to n-1,

[0100] Set standard output Z i First pass through the i+1th layer and get the standard output Z i+1 ;

[0101] Prune the i-th fine-tuning layer to obtain the i-th pruned layer;

[0102] Prune the biased output Z i-1 'Input the i-th pruning layer to get the pruned biased output Z i ';

[0103] Prune the biased output Z i 'Input the i+1th layer to obtain the fine-tuned biased output Z i+1 ”;

[0104] Compare the standard output Z of layer i+1 i+1 With fine-tuning biased output Z i+1The difference between the two forms the loss value, and the i+1th layer is updated to obtain the i+1th fine-tuning layer, where the standard output Z i+1 The standard output from layer i+1 of the original model.

[0105] The process of pruning the i-th fine-tuning layer specifically includes:

[0106] The i-th fine-tuning layer is traversed to determine the attention layer and the feedforward layer of the i-th fine-tuning layer; the attention layer and the feedforward layer of the i-th fine-tuning layer are pruned using Wanda or SparseGPT to obtain the i-th pruned layer.

[0107] The process of updating the i+1th layer specifically includes:

[0108] Based on the MSE loss function, calculate the standard output Z i+1 With fine-tuning biased output Z i+1 "The deviation loss value;

[0109] According to the deviation loss value, the parameters of the i+1th layer are optimized to minimize the standard output Z i+1 With fine-tuning biased output Z i+1 ” Deviation between .

[0110] In this scheme, pruning is performed on the attention layer and feedforward layer of the i-th layer, which is defined as inter-layer pruning. This pruning strategy belongs to the conventional technique in the category of post-training pruning. Compared with global-level pruning, inter-layer pruning can more accurately screen pruning parameters, flexibly set thresholds, and effectively maintain the performance of the pruned model in the post-training pruning paradigm. Therefore, inter-layer pruning is usually the preferred method.

[0111] To achieve optimal performance, only SparseGPT can be used, because Wanda is actually a simplified version of SparseGPT and performs poorly in CLIP pruning compared to SparseGPT. Wanda is used for pruning here to illustrate the effectiveness of the SLD technology of the present invention combined with these fast pruning methods in CLIP pruning, while achieving fast pruning and maintaining a certain post-pruning performance.

[0112] In deep learning models, the fast pruning technology used by SparseGPT belongs to the post-training inter-layer pruning method. This method prunes layer by layer, but does not fully consider the resulting errors and their cumulative effects during the pruning process. Since the errors are amplified in layer-by-layer reasoning until the last layer of the model, this leads to a significant decrease in model performance.

[0113] In contrast, the two-layer pruning strategy fine-tunes the second layer after pruning the first layer to reduce the accumulation of pruning errors in the first layer. Subsequently, while pruning the second layer, the third layer is fine-tuned to further reduce the error accumulation of the second layer. Through this iterative optimization until the last layer, the strategy can effectively reduce the error caused by pruning, thereby alleviating the negative impact of pruning on model performance.

[0114] The core of SparseGPT is that it uses the second-order gradient information of Hessian as an indicator and realizes the screening of fast pruning parameters through approximate calculation. Although this process is efficient, its recovery ability is limited due to the use of fast approximation methods. Therefore, it is difficult to achieve ideal results when pruning models such as CLIP using SparseGPT.

[0115] The performance is restored by the SLD fast distillation method proposed in this application, which not only maintains the original advantages of SparseGPT's fast processing and low resource consumption, but also achieves excellent performance recovery effects, thereby developing an efficient pruning algorithm for CLIP.

[0116] In the field of model pruning, common pruning standards can be summarized as follows:

[0117] First, based on the absolute size (magnitude) of the parameter values, remove parameters whose magnitude is less than S% of the target sparsity. This criterion is simple to operate, but is usually accompanied by a decrease in performance.

[0118] Second, based on the contribution of the weight to the gradient of the loss function, the weights whose gradient contribution is lower than the target sparsity S% are removed. This criterion performs better in performance, however, it requires obtaining gradient information during training, which increases the consumption of computing resources.

[0119] Third, the parameter values ​​induced by the regularization term (such as L1 regularization) during the training process are removed, and those parameters that are less than the target sparsity S% are removed. This criterion can also achieve good performance, but it is only applicable to the model during the training process, and may make it difficult for the training to converge, requiring certain parameter adjustment skills, and increasing the consumption of computing resources.

[0120] Fourth, the second-order gradient information brought by the Hessian matrix. Pruning the model after training. Since the entire model has been trained, it can be considered that the entire model is optimal before pruning. At this time, pruning a model parameter matrix W0 can be regarded as Taylor expansion near W0, and pruning all parameters can be regarded as the size of the parameter matrix W0 fluctuating the parameter. In this way, the error caused by pruning the parameter can be calculated through Taylor expansion, ignoring the number of Taylor 3rd order and above. Since the model is considered to be trained, the derivative of the 1st order term is 0, and 0 multiplied by any number is also 0. At this time, only the 2nd order term, that is, the Hessian matrix, remains. This standard has good performance, can be applied to pruning after training, and is fast.

[0121] This application performs pruning based on the second-order gradient information brought by the Hessian matrix. While maintaining a relatively fast speed, it retains important parameters as much as possible, making the subsequent distillation process easier to carry out and being able to accurately and quickly complete the pruning loss compensation process.

[0122] Taking text data as an example, see Figure 3 , the specific pruning process includes:

[0123] Step 1: Get the input before pruning of the first layer, i.e., the standard input X, and the output before pruning of the second layer, i.e., the standard output X 2 ;

[0124] Step 2: Traverse the first layer of the text encoder of the target CLIP model, determine the attention layer and feedforward layer of the first layer, and use Wanda or SparseGPT to prune the attention layer and feedforward layer of the first layer, input the pre-pruning input of the first layer, that is, the standard input X, into the pruned first layer to obtain the pruned biased output X 1 ';

[0125] Step 3: Prune the biased output of the first layer X 1 'Input the second layer and get the fine-tuned biased output X 2 "; Based on the standard output X of the second layer 2 and fine-tuning the biased output X 2 ", construct the loss function Loss: MSE(X 2 , X 2 ”), optimize the loss function to minimize the second layer X 2 " and standard output X 2 The deviation between

[0126] Step 4: Input the second layer's standard input to X 2 Pass it to the third layer to get the standard output X of the third layer 3Subsequently, the attention layer and feedforward layer of the second layer are pruned using Wanda or SparseGPT technology to obtain the second pruned layer; the pruned biased output X 1 'Input the pruned second layer to get the pruned biased output X 2 '; Prune the biased output X 2 'Enter the third layer and obtain the fine-tuned biased output X 3 ". Then, based on the third layer's fine-tuned biased output X 3 " and standard output X 3 , construct the loss function Loss: MSE(X 3 , X 3 ”) and optimizes it to minimize the biased output X of the third layer 3 " and standard output X 3 The deviation between .

[0127] For the i-th iteration, where i is incremented from 3 to n-1,

[0128] Pass the standard output Xi through the i+1th layer first, and get the standard output X i+1 ;

[0129] Prune the i-th fine-tuning layer to obtain the i-th pruned layer;

[0130] Prune the biased output X i-1 'Input the i-th pruning layer to get the pruned biased output X i ';

[0131] Prune the biased output X i 'Enter the i+1th layer to obtain the fine-tuned biased output X i+1 ”;

[0132] Compare the standard output X of layer i+1 i+1 With fine-tuning biased output X i+1 The difference between the values ​​of " forms the loss value, and the i+1th layer is updated to obtain the i+1th fine-tuning layer, where the standard output X i+1 The standard output from layer i+1 of the original model.

[0133] Step 5: Prune the last layer and do not perform any fine-tuning or updating. The process ends.

[0134] The pruning process of image data is similar to that of text data. First, the first layer of the image encoder is pruned, and then the loss function Loss is constructed: MSE(Y 2 , Y 2 ”) updates the second layer of the image encoder, and then prunes the second layer of the image encoder to construct the loss function Loss: MSE(Y 3, Y 3 ”) updates the third layer of the image encoder… prunes the i-th layer of the image encoder and constructs the loss function Loss: MSE[Y (i++1) , Y (i+1) ”] Update the i+1th layer of the image encoder, and repeat this cycle until the pruning of the nth layer is completed.

[0135] In this scheme, the specific optimization process of the loss function includes:

[0136] Setting hyperparameters, wherein the hyperparameters include at least an initial learning rate and a number of iterations;

[0137] Set the training termination threshold;

[0138] Based on the hyperparameters and the training termination threshold, the Adam optimizer is used to optimize the loss function.

[0139] In this scheme, the parameter configuration of the update process is specified in detail. Based on the experimental results of three different CLIP models on multiple data sets, the hyperparameters are uniformly set as follows: the initial learning rate is set to 1e -5 The upper limit of the number of training iterations is 15 epochs, the loss function adopts the mean square error (MSE), and the condition for early termination of training is set, that is, training is stopped when the average loss value drops to 0.01.

[0140] The updating process is achieved by calculating the loss value obtained by the loss function and executing the back-propagation algorithm to adjust the parameter values.

[0141] The specific steps of the update are as follows: Adopt the Adam optimization algorithm and set the learning rate to 1e -5 , the loss value between the output of the second layer of the input data and the standard output set by the parameters of this layer is calculated through the mean square error (MSE) loss function. This loss value is used for back propagation to update the parameters of the second layer. The upper limit of the iteration number of the update process is set to 15 epochs, and a condition for early termination of training is set, that is, when the average loss value is lower than 0.01, the fine-tuning process is stopped. At the end of each epoch, the average loss value is calculated by MSE. If the value is less than the set threshold, the training is terminated.

[0142] The following introduces the experimental results:

[0143] In the scheme, the settings for pruning sparsity include different proportions such as 50%, 60%, 70%, and 80% for unstructured pruning. Specifically, x% sparsity means that x% of the parameters are 0. In addition, two modes of semi-structured pruning, 2:4 and 4:8, are also included. Both modes have a sparsity of 50% and have certain structural characteristics, which are more friendly to hardware devices. The pruning model used is based on the officially released CLIP model, and its image encoder part has been fine-tuned on six different data sets to obtain six fine-tuned CLIP models. The pruning operation is implemented on these fine-tuned CLIP models.

[0144] See also Figure 4-Figure 5 , the Dense model line represents the performance of the original model, and the other lines with the same mark represent the same set of comparative experiments. Among them, the dotted line represents the pruning performance of the baseline method, and the solid line represents the performance of the present application. In the figure, the horizontal axis represents the sparsity of the model, and the vertical axis represents the accuracy of the model. Specifically, the dotted solid line corresponds to the SparseGPT+SLD model, and the dotted line corresponds to the SparseGPT model; the triangular solid line corresponds to the Wanda+SLD model, and the triangular dotted line corresponds to the Wanda model. Observing the graphical data, it can be seen that under the premise of keeping the model sparsity unchanged, the method proposed in the present invention can significantly maintain a higher model accuracy than the baseline method. For example, by comparing the difference between the dotted solid line and the dotted line, and the triangular solid line and the triangular dotted line, the advantages of the present method can be clearly seen. The SLD (Sliding-distillation) distillation of the present invention can compensate for the CLIP pruning process in real time, avoiding the huge burden brought by the full model distillation.

[0145] For the Huggingface official CLIP-B-32 model, the classification accuracy was evaluated on 6 open source datasets under different sparsity configurations. The test results and average classification accuracy are shown in the following table. Each data point is presented in the format of Accl / Acc5. The vertical axis of the chart represents the pruning method used, the pruning sparsity, the six test datasets, and the average of the six test datasets. The Dense row specifically refers to the original model accuracy. In the Method column, the row marked with SLD represents the method proposed in this application, and its performance will be compared and analyzed with the baseline method without SLD marking.

[0146] Table 1 shows the top-one accuracy and top-five accuracy of the CLIP-B-32 model on eight datasets using four different pruning methods at 50% sparsity, as well as the corresponding average top-one accuracy and average top-five accuracy.

[0147] Table 1 clip-vit-base-patch32

[0148]

[0149] From Table 1, it can be concluded that at 50% sparsity, SparseGPT+SLD has significantly improved accuracy compared to SparseGPT. Similarly, Wanda+SLD has improved accuracy compared to Wanda. When comparing SparseGPT+SLD with Wanda+SLD, it can be clearly found that SparseGPT+SLD has better accuracy.

[0150] Table 2 shows the top-one accuracy and top-five accuracy of the CLIP-B-32 model on eight datasets using four different pruning methods at 60% sparsity, as well as the corresponding average top-one accuracy and average top-five accuracy.

[0151] Table 2 clip-vit-base-patch32

[0152]

[0153]

[0154] From Table 2, it can be concluded that at 60% sparsity, SparseGPT+SLD has significantly improved accuracy compared to SparseGPT. Similarly, Wanda+SLD has improved accuracy compared to Wanda. When comparing SparseGPT+SLD with Wanda+SLD, it can be clearly found that SparseGPT+SLD has better accuracy.

[0155] Table 3 shows the top-one accuracy and top-five accuracy of the CLIP-B-32 model on eight datasets using four different pruning methods at 70% sparsity, as well as the corresponding average top-one accuracy and average top-five accuracy.

[0156] Table 3 clip-vit-base-patch32

[0157]

[0158] From Table 3, it can be concluded that at 70% sparsity, SparseGPT+SLD has significantly improved accuracy compared to SparseGPT. Similarly, Wanda+SLD has improved accuracy compared to Wanda. When comparing SparseGPT+SLD with Wanda+SLD, it can be clearly found that SparseGPT+SLD has better accuracy.

[0159] Table 4 shows the top-one accuracy and top-five accuracy of the CLIP-B-32 model on eight datasets using four different pruning methods at 80% sparsity, as well as the corresponding average top-one accuracy and average top-five accuracy.

[0160] Table 4 clip-vit-base-patch32

[0161]

[0162] From Table 4, it can be concluded that at 80% sparsity, SparseGPT+SLD has significantly improved accuracy compared to SparseGPT. Similarly, Wanda+SLD has improved accuracy compared to Wanda. When comparing SparseGPT+SLD with Wanda+SLD, it can be clearly found that SparseGPT+SLD has better accuracy.

[0163] Table 5 shows the top-one accuracy and top-five accuracy of the CLIP-B-32 model on eight datasets using four different pruning methods under 2:4 sparsity, as well as the corresponding average top-one accuracy and average top-five accuracy.

[0164] Table 5 clip-vit-base-patch32

[0165]

[0166]

[0167] From Table 5, we can conclude that at 2:4 sparsity, SparseGPT+SLD has significantly improved accuracy compared to SparseGPT. Similarly, Wanda+SLD has improved accuracy compared to Wanda. When comparing SparseGPT+SLD with Wanda+SLD, it is obvious that SparseGPT+SLD has better accuracy.

[0168] Table 6 shows the top-one accuracy and top-five accuracy of the CLIP-B-32 model on eight datasets using four different pruning methods under 4:8 sparsity, as well as the corresponding average top-one accuracy and average top-five accuracy.

[0169] Table 6 clip-vit-base-patch32

[0170]

[0171] From Table 6, it can be concluded that at 4:8 sparsity, SparseGPT+SLD has significantly improved accuracy compared to SparseGPT. Similarly, Wanda+SLD has improved accuracy compared to Wanda. When comparing SparseGPT+SLD with Wanda+SLD, it can be clearly found that SparseGPT+SLD has better accuracy.

[0172] From Tables 1 to 6, it can be concluded that under the condition of 50% sparsity, SparseGPT+SLD has the highest accuracy.

[0173] In summary, there are the following advantages. First, the resource consumption is small. An NVIDIA GeForce RTX 4090 graphics card is used, which is equipped with 24GB of video memory. However, in actual applications, the usage of video memory does not exceed 10GB, which shows that the model has relatively low demand for hardware resources during operation. Second, the pruning speed is fast. When using an RTX 4090 graphics card, the entire pruning process can be completed in less than 0.5 GPU hours, which can significantly improve efficiency and reduce waiting time. In addition, the classification accuracy loss is small. Even if some redundant parameters are removed during the pruning process, the model can still maintain a high accuracy.

[0174] More specifically, we take a CLIP-L-14 model fine-tuned on the CIFAR-100 dataset as an example. CIFAR-100 is a benchmark dataset widely used in the field of computer vision, mainly for image classification tasks. The dataset covers 100 categories, each category contains 600 32x32 pixel color images. The dataset contains a total of 60,000 images, of which 50,000 images are used for training and the remaining 10,000 images are used for testing.

[0175] The 10,000 official test images of CIFAR-100 were selected as the performance evaluation indicators. For the selection of pruning input data, the 50,000 training images provided by the Huggingface official website were used, and 500 data were randomly selected as samples. The specific pruning process is as follows:

[0176] 1. Load the Cifar100 dataset.

[0177] 2. Use RandomSampler to randomly sample the data set and select 500 pictures as image data.

[0178] 3. Use Dataloader to read data and customize its collate_fn function. In this function, use the built-in Processor of the CLIP model to preprocess the image data to obtain a processed image calibration dataset.

[0179] 4. Collect the label information corresponding to the 500 images in the first step, and map these label information to a list of class names (if the label information is directly expressed as a class name, this step can be omitted).

[0180] 5. After obtaining the list of category names, the text was constructed according to the Prompt templates officially provided by CLIP. For each category name, a Prompt template was randomly selected to construct the text, thus generating 500 sentences.

[0181] The prompt template on Cifar100 provided by CLIP is as follows:

[0182] cifar100_templates=[

[0183] 1ambda c: f'a photo of a{c}.',

[0184] lambda c: f'a blurry photo of a{c}.',

[0185] lambda c: f'a black and white photo of a{c}.',

[0186] lambda c: f'a low contrast photo of a{c}.,

[0187] lambda c: f'a high contrast photo of a{c}.,

[0188] lambda c: f'a bad photo of a{c}.', ]

[0190] 6. Then, these 500 sentences are processed through the CLIP built-in Tokenizer to obtain text calibration data.

[0191] 7. Register hooks (callback functions) for the first and second layers of the target CLIP model to collect the standard input of the first layer and the standard output of the second layer.

[0192] 8. Use a pair-by-pair input method to input the 500 pairs of calibration data obtained previously into the target CLIP model.

[0193] 9. Use the previously registered hooks to obtain the standard input of the first layer and the standard output of the second layer.

[0194] 10. Traverse the first layer of the target CLIP model and determine the attention layer and feedforward layer of the first layer.

[0195] 11. Input the pre-pruning input of the first layer, i.e., the initially collected standard input, to the first layer, and use Wanda or SparseGPT to prune the attention layer and feedforward layer of the first layer to obtain the biased output of the first layer;

[0196] 12. Use the biased output of the first layer as the input of the second layer of the target CLIP model, and the previously collected standard output of the second layer of the target CLIP model as the target, and use MSE to fine-tune the second layer of the target CLIP model.

[0197] 13. For fine-tuning, you need to set the fine-tuning configuration, and the hyperparameters are uniformly set to: Use the Adam optimizer, and the initial value of the learning rate is 1e -5 The upper limit of training epochs is 15, MSE is used as loss, and the epoch average loss threshold for terminating training early is set to 0.01.

[0198] 14. Use the previously collected second-layer standard output as the input of the third layer for inference to obtain the standard output of the third layer. Use the first-layer biased output obtained by inference after pruning the first layer as the input before pruning the second layer. Prune the second layer and construct a loss function to update the third layer.

[0199] 15. Repeat the above process until the n-1th layer of the target CLIP model is pruned and the nth layer is fine-tuned.

[0200] 16. The last layer of the target CLIP model is specially processed and ends after pruning. No fine-tuning is required and the model pruning process ends.

[0201] Here are the experimental results for this example:

[0202] See also Figure 6-Figure 7, the Dense model line represents the performance of the original model, and the other lines with the same mark represent the same set of comparative experiments. Among them, the dotted line represents the pruning performance of the baseline method, and the solid line represents the performance of the present application. In the figure, the horizontal axis represents the sparsity of the model, and the vertical axis represents the accuracy of the model. Specifically, the dotted solid line corresponds to the SparseGPT+SLD model, and the dotted line corresponds to the SparseGPT model; the triangular solid line corresponds to the Wanda+SLD model, and the triangular dotted line corresponds to the Wanda model. Observing the graphical data, it can be seen that under the premise of keeping the model sparsity unchanged, the method proposed in the present invention can significantly maintain a higher model accuracy than the baseline method. For example, by comparing the differences between the dotted solid line and the dotted line, and the triangular solid line and the triangular dotted line, the advantages of this method can be clearly seen.

[0203] For the Huggingface official CLIP-L-14 model, the classification accuracy test results on Cifar100 under the semi-structured sparsity setting are shown in the following table. Each data point is presented in the format of Accl / Acc5. The vertical axis of the chart represents the pruning method used, pruning sparsity, and Cifar100 test data set. The Dense row specifically refers to the original model accuracy. In the Method column, the row marked with SLD represents the method proposed in this application, and its performance will be compared and analyzed with the baseline method without SLD marking. The SLD here is the pruning method of this application.

[0204] Table 7 shows the top-one accuracy and top-five accuracy of the CLIP-L-14 model on the Cifar100 dataset using four different pruning methods under 2:4 sparsity.

[0205] Table 7 finetuned clip-vit-large-patch14

[0206]

[0207]

[0208] From Table 7, it can be concluded that at 2:4 sparsity, SparseGPT+SLD has significantly improved accuracy compared to SparseGPT. Similarly, Wanda+SLD has improved accuracy compared to Wanda. When comparing SparseGPT+SLD with Wanda+SLD, it can be clearly found that SparseGPT+SLD has better accuracy.

[0209] Table 8 shows the top-one accuracy and top-five accuracy of the CLIP-L-14 model on the Cifar100 dataset using four different pruning methods at 4:8 sparsity.

[0210] Table 8 finetuned clip-vit-large-patch14

[0211] Method Sparsity Cifar100 Dense 0％ 0.930 / 0.993 SparseGPT 4∶8 0.877 / 0.982 SparseGPT+SLD 4∶8 0.906 / 0.989 Wanda 4∶8 0.767 / 0.943 Wanda+SLD 4∶8 0.882 / 0.984

[0212] From Table 8, it can be concluded that at 4:8 sparsity, SparseGPT+SLD has significantly improved accuracy compared to SparseGPT. Similarly, Wanda+SLD has improved accuracy compared to Wanda. When comparing SparseGPT+SLD with Wanda+SLD, it can be clearly found that SparseGPT+SLD has better accuracy.

[0213] From Tables 7 and 8, it can be observed that under the sparsity condition of 2:4, SparseGPT+SLD has the highest accuracy.

[0214] The present application also provides a pruning system for the CLIP model, including:

[0215] An acquisition module is used to acquire a target data set;

[0216] A setting module, used for presetting a sliding window, wherein the sliding window includes a pruning unit and an updating unit, wherein the pruning unit corresponds to the i-th layer of the target CLIP model, and the updating unit corresponds to the i+1-th layer of the target CLIP model, wherein the target CLIP model has n layers, 1≤i<n;

[0217] The pruning module is used to input the target data set into the target CLIP model, prune the i-th layer in turn according to the sliding window, and update the i+1-th layer until the n-th layer is pruned to obtain the pruned CLIP model.

[0218] It should be understood that the above embodiments are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with the technology to understand the content of the present invention and implement it accordingly, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A pruning method for a CLIP model, characterized in that: The following steps are involved: Get the target dataset; A sliding window is preset, the sliding window includes a pruning unit and an updating unit, the pruning unit corresponds to the i-th layer of the target CLIP model, the updating unit corresponds to the i+1-th layer of the target CLIP model, the target CLIP model has n layers, 1≤i <n; The target data set is input into the target CLIP model, and according to the sliding window, the i-th layer is pruned in order from 1 to n-1, and the i+1-th layer is updated until the n-th layer is pruned, thereby obtaining a pruned CLIP model.

2. The pruning method for the CLIP model according to claim 1, characterized in that: The obtaining of the target data set specifically includes: Collecting image data; Extracting label information corresponding to the image data, and mapping the label information to obtain a class name list; According to the class name list, construct text to obtain text data; Construct a mapping relationship between image data and text data to form several pairs of data as the target data set.

3. The pruning method for the CLIP model according to claim 2, characterized in that: Specifically include: Acquire text data with the text encoder of the target CLIP model, prune the i-th layer of the text encoder in sequence according to the sliding window, and update the i+1-th layer of the text encoder until the n-th layer of the text encoder is pruned, thereby obtaining a pruned text encoding model; Acquire image data using the image encoder of the target CLIP model, prune the i-th layer of the image encoder in sequence according to the sliding window, and update the i+1-th layer of the image encoder until the n-th layer of the image encoder is pruned, thereby obtaining a pruned image coding model; The pruned text encoding model is integrated with the image encoding model to obtain the pruned CLIP model.

4. The pruning method for the CLIP model according to claim 2, characterized in that: Also includes: The plurality of pairs of data are sequentially input into the target CLIP model.

5. The pruning method for the CLIP model according to claim 1, characterized in that: The pruning method is performed in multiple rounds of iterations, specifically including: For the first iteration, Obtain standard initial values, which at least include the standard input Z of the first layer and the standard output Z2 of the second layer of the target CLIP model; Prune the first layer of the target CLIP model to obtain the first pruned layer; Input the standard input Z into the first pruned layer to obtain the pruned biased output Z1', and input the pruned biased output Z1' into the second layer to obtain the fine-tuned biased output Z2"; Compare the difference between the standard output Z2 and the fine-tuned biased output Z2" to form a loss value, and update the second layer according to the loss value to obtain the second fine-tuning layer; For the i-th iteration, where i is incremented from 2 to n-1, Set standard output Z i First pass through the i+1th layer and get the standard output Z i+1 , Prune the i-th fine-tuning layer to obtain the i-th pruned layer; Prune the biased output Z i-1 'Input the i-th pruning layer to get the pruned biased output Z i '; Prune the biased output Z i 'Input the i+1th layer to obtain the fine-tuned biased output Z i+1 ”; Compare the standard output Z of layer i+1 i+1 With fine-tuning biased output Z i+1 The difference between the two forms the loss value, and the i+1th layer is updated to obtain the i+1th fine-tuning layer, where the standard output Z i+1 The standard output from layer i+1 of the original model.

6. The pruning method for the CLIP model according to claim 5, characterized in that: The process of pruning the i-th fine-tuning layer specifically includes: The i-th fine-tuning layer is traversed to determine the attention layer and the feedforward layer of the i-th fine-tuning layer; the attention layer and the feedforward layer of the i-th fine-tuning layer are pruned using Wanda or SparseGPT to obtain the i-th pruned layer.

7. The pruning method for the CLIP model according to claim 5, characterized in that: Also includes: Use the hook function to get the standard output Z2 of the second layer.

8. The pruning method for the CLIP model according to claim 5, characterized in that: The process of updating the i+1th layer specifically includes: Based on the MSE loss function, calculate the standard output Z i+1 With fine-tuning biased output Z i+1 "The deviation loss value; According to the deviation loss value, the parameters of the i+1th layer are optimized to minimize the standard output Z i+1 With fine-tuning biased output Z i+1 ” Deviation between .

9. The pruning method for the CLIP model according to claim 7, characterized in that: Also includes: Setting hyperparameters, wherein the hyperparameters include at least an initial learning rate and a number of iterations; Set the training termination threshold; Based on the hyperparameters and the training termination threshold, the Adam optimizer is used to optimize the loss function.

10. A pruning system for a CLIP model, characterized in that: include: An acquisition module is used to acquire a target data set; A setting module is used to preset a sliding window, wherein the sliding window includes a pruning unit and an updating unit, wherein the pruning unit corresponds to the i-th layer of the target CLIP model, and the updating unit corresponds to the i+1-th layer of the target CLIP model, wherein the target CLIP model has n layers, 1≤i <n; The pruning module is used to input the target data set into the target CLIP model, and prune the i-th layer in the order of i from 1 to n-1 according to the sliding window, and update the i+1-th layer until the n-th layer is pruned, thereby obtaining a pruned CLIP model.

Images