Model cutting method and device, equipment, storage medium and program product
By calculating the target pruning rate and bias correction method, the neural network model is pruned, which solves the deployment difficulties and resource consumption problems caused by the large number of parameters, and achieves efficient model pruning and performance recovery.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-29
- Publication Date
- 2026-04-14
AI Technical Summary
Existing neural network models have a large number of parameters, making them difficult to deploy on mobile devices. Furthermore, existing pruning methods require retraining, which consumes a lot of resources and cannot be performed in some cases.
By calculating the target pruning rate, the original network model is pruned and then biased to obtain the target network model. The objective function includes an output error subfunction, a loss subfunction, and a nuclear norm subfunction, in order to minimize the impact of pruning and restore performance.
Reduce resource consumption without retraining, minimize the impact of pruning on model parameters, restore network performance, and achieve efficient deployment.
Smart Images
Figure CN121859970A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of data processing technology, and in particular relates to a model trimming method, apparatus, device, storage medium and program product. Background Technology
[0002] A current trend in neural networks is that models are getting larger and have more and more parameters. Early convolutional neural networks already had tens of millions of parameters, and today's mainstream large language models have reached tens of billions of parameters. Such a massive number of parameters makes neural networks extremely resource-intensive. The deployment problem is even more pronounced on mobile devices because mobile devices have very limited memory and computing resources, making it very difficult to deploy large models on these devices.
[0003] Existing pruning methods typically require retraining the network model after pruning. This retraining step is extremely time-consuming and resource-intensive, essentially equivalent to the cost of training the original model. Furthermore, in many cases, the training dataset is unavailable; some companies, while open-sourcing their network models, do not provide the training data. In such situations, retraining is not only costly but also impossible. Summary of the Invention
[0004] This application provides a model trimming method, apparatus, device, storage medium, and program product that can trim a model without retraining.
[0005] In a first aspect, embodiments of this application provide a model trimming method, the method comprising: Obtain the model parameters of the original network model; Based on the model parameters, and with the constraint that the size of the pruned original network model meets a preset constraint, the target pruning rate of the original network model is calculated with the objective of minimizing a preset objective function. The objective function includes an output error sub-function, a loss sub-function, and a kernel norm sub-function. The output error sub-function is used to calculate the error of the output of the last layer after pruning the original network model. The loss sub-function is used to calculate the loss of attention scores between visual tokens in each attention layer after pruning the original network model. The kernel norm sub-function is used to calculate the kernel norm of each self-attention layer after pruning the original network model. The original network model is pruned according to the target pruning rate to obtain the first network model; Based on the difference in mean and variance of network parameters between the original network model and the first network model, as well as the preset target correction vector, the first network model is biased to obtain a target network model. The target network model has the same mean and variance in each channel as the original network model.
[0006] Secondly, embodiments of this application provide a model trimming device, the device comprising: The acquisition module is used to obtain the model parameters of the original network model; The calculation module is used to calculate the target pruning rate of the original network model based on the model parameters, with the constraint that the size of the pruned model meets the preset constraints, and with the objective of minimizing the preset objective function. The objective function includes an output error sub-function, a loss sub-function, and a kernel norm sub-function. The output error sub-function is used to calculate the error of the output of the last layer after pruning the original network model. The loss sub-function is used to calculate the loss of attention scores between visual tokens in each attention layer after pruning the original network model. The kernel norm sub-function is used to calculate the kernel norm of each self-attention layer after pruning the original network model. The pruning module is used to prune the original network model according to the target pruning rate to obtain a first network model; The correction module is used to correct the deviation of the first network model based on the difference between the mean and variance of the network parameters of the original network model and the first network model, as well as the preset target correction vector, to obtain a target network model. The target network model has the same mean and variance in each channel as the original network model.
[0007] Thirdly, embodiments of this application provide an electronic device, the device including: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the model trimming method as described above.
[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the model trimming method described in any of the above claims.
[0009] Fifthly, embodiments of this application provide a computer program product, wherein instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the model trimming method as described in any of the above claims.
[0010] The model pruning method, apparatus, device, storage medium, and program product of this application embodiment can calculate the target pruning rate of the original network model based on the model parameters of the original network model, with the pruned model size satisfying preset constraints as a constraint, and minimizing a preset objective function as the objective. The objective function includes an output error sub-function, a loss sub-function, and a kernel norm sub-function. The original network model is then pruned according to the target pruning rate to obtain a first network model. Then, based on the difference in the mean and variance ratio of the network parameters between the original network model and the first network model, and a preset target correction vector, the first network model is biased to obtain a target network model, ensuring that the mean and variance of the target network model are consistent with those of the original network model in each channel. Thus, in this application embodiment, pruning the model using the optimal target pruning rate ensures that the parameters of the original network model are minimally affected by pruning. Furthermore, the bias correction processing after pruning not only avoids retraining, saving resources and improving efficiency, but also recovers the performance loss caused by pruning. Attached Figure Description
[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a schematic flowchart of a model trimming method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating another model trimming method provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the changes in parameter distribution before and after trimming, provided in an embodiment of this application. Figure 4 This is a flowchart illustrating yet another model trimming method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a model cutting device provided in another embodiment of this application; Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0013] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.
[0014] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.
[0015] A current trend in neural networks is that models are getting larger and have more and more parameters. Early convolutional neural networks already had tens of millions of parameters, and today's mainstream large language models have reached tens of billions of parameters. Such a massive number of parameters makes neural networks extremely resource-intensive. The deployment problem is even more pronounced on mobile devices because mobile devices have very limited memory and computing resources, making it very difficult to deploy large models on these devices.
[0016] Existing pruning methods typically require retraining the network model after pruning. This retraining step is extremely time-consuming and resource-intensive, essentially equivalent to the cost of training the original model. Furthermore, in many cases, the training dataset is unavailable; some companies, while open-sourcing their network models, do not provide the training data. In such situations, retraining is not only costly but also impossible.
[0017] The acquisition, storage, use, and processing of data in this application comply with relevant national laws and regulations. It should be noted that certain software, components, models, and other existing industry solutions may be mentioned in the embodiments of this application. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.
[0018] To address the problems of the prior art, embodiments of this application provide a model trimming method, apparatus, device, storage medium, and program product. The model trimming method provided in this application embodiment will be described first below.
[0019] Figure 1 A schematic flowchart of a model trimming method provided in one embodiment of this application is shown. Figure 1 As shown, a model trimming method may include the following steps S101 to S104: S101. Obtain the model parameters of the original network model; S102. Based on the model parameters, with the constraint that the size of the pruned model meets the preset constraints, and with the objective of minimizing the preset objective function, calculate the target pruning rate of the original network model. The objective function includes an output error sub-function, a loss sub-function, and a nuclear norm sub-function. The output error sub-function is used to calculate the error of the last layer output after pruning the original network model. The loss sub-function is used to calculate the loss of attention scores between visual tokens in each attention layer after pruning the original network model. The nuclear norm sub-function is used to calculate the nuclear norm of each attention layer after pruning the original network model. S103. Prune the original network model according to the target pruning rate to obtain the first network model; S104. Based on the difference in mean and variance of network parameters between the original network model and the first network model, as well as the preset target correction vector, the first network model is biased to obtain the target network model. The target network model is consistent with the mean and variance of the original network model in each channel.
[0020] The model pruning method of this application embodiment can calculate the target pruning rate of the original network model based on the model parameters of the original network model, with the pruned model size satisfying preset constraints as a constraint, and minimizing a preset objective function as the objective function. The objective function includes an output error sub-function, a loss sub-function, and a kernel norm sub-function. The original network model is then pruned according to the target pruning rate to obtain a first network model. Then, based on the difference in the mean and variance ratio of the network parameters between the original and first network models, and a preset target correction vector, the first network model is biased to obtain a target network model, ensuring that the mean and variance of the target network model are consistent with those of the original network model in each channel. Thus, in this application embodiment, pruning the model using the optimal target pruning rate ensures that the parameters of the original network model are minimally affected by pruning. Furthermore, the bias correction processing after pruning not only avoids retraining, saving resources and improving efficiency, but also recovers the performance loss caused by pruning.
[0021] In S101, the original network model described above can, for example, be a convolutional neural network model, a transformer neural network model, or a vision and language multimodal model (Large Language and Vision Assistant, LLaVA).
[0022] In S102, the objective function mentioned above may include an output error subfunction, a loss subfunction, and a nuclear norm subfunction. The output error subfunction calculates the error of the last layer's output after pruning the original network model; the loss subfunction calculates the loss of attention scores between visual tokens in each attention layer after pruning the original network model; and the nuclear norm subfunction calculates the nuclear norm of each attention layer after pruning the original network model.
[0023] The aforementioned preset constraints can be set by developers based on actual needs regarding the size of the trimmed model.
[0024] In some embodiments of this application, the target pruning rate of the original network model is calculated based on model parameters, with the constraint that the size of the pruned model after pruning the original network model meets preset constraints, and with the objective of minimizing a preset objective function. For example, the target pruning rate of the original network model can be directly calculated based on model parameters, with the constraint that the size of the pruned model after pruning the original network model meets preset constraints, and with the objective of minimizing a preset objective function. Alternatively, pruning rate sampling can be performed within a preset numerical range according to a preset pruning rate interval to construct a pruning rate dataset, which includes multiple pruning rates. The weights of each layer of the network are pruned using the pruning rates in the pruning rate dataset, with the constraint that the size of the pruned model after pruning the original network model meets preset constraints, and with the objective of minimizing a preset objective function, to calculate the target pruning rate of each layer of the network. Each target pruning rate is the pruning rate value on the rate-distortion output error curve of the weights and activations of each layer, where the slope satisfies a preset Lagrange operator. The target pruning rate of the original network model includes the target pruning rates of multiple layers of the network.
[0025] In S103, in some embodiments of this application, the original network model is pruned according to a target pruning rate to obtain a first network model. For example, this can be achieved by uniformly pruning the weights of each layer in the original network model according to the target pruning rate. Alternatively, the weights of each layer in the original network model can be pruned differently according to the target pruning rate of each layer to obtain the first network model.
[0026] In S104, the target network model described above can be consistent with the mean and variance of the original network model in each channel.
[0027] The aforementioned target correction vector can be a correction vector that minimizes the output deviation between the first network model and the original network model.
[0028] In some embodiments of this application, the first network model is biased and corrected based on the difference between the mean and variance of the network parameters of the original network model and the first network model, as well as a preset target correction vector, to obtain a target network model. For example, this can be achieved by calculating the difference between the mean and variance of the network parameters of the original network model and the first network model; performing signal recovery on the first network model based on the difference between the mean and variance of the network parameters to obtain a second network model; and correcting the bias of the weights of each layer in the second network model based on the preset target correction vector to obtain the target network model. The target correction vector is used to correct the bias between the second network model and the original network model.
[0029] The target network model obtained by the model pruning method in this application can be applied to text processing, image processing, speech recognition, etc. Text processing includes sentiment analysis, spam classification, etc. Image processing includes object detection, face recognition, etc. Speech recognition includes speech-to-text, voice command control, etc.
[0030] The data distribution characteristics, hardware deployment conditions, and core business requirements of the embodiments of this application differ in different application scenarios. The actual application effect of this application may vary due to factors such as scenario adaptability, data quality, and compliance requirements. When implementing the application in specific scenarios, users should make targeted adjustments and optimizations to the technical solution based on their own business realities, and must strictly comply with the relevant laws, regulations, and industry standards on data security, privacy protection, and compliance management in the corresponding industry to ensure that the application process is legal and compliant.
[0031] The local dataset for the target network model can include image data, text data, and audio data. Image data can be medical images or surveillance footage; text data can be user comments or news texts; and audio data can be voice commands or recorded files. For example, in image classification tasks, important testing dimensions can be texture, edges, color distribution, etc.; in text classification tasks, important testing dimensions can be word frequency, word vector similarity, and syntactic structure, etc.
[0032] The performance information of the target network model can include metrics such as accuracy, recall, and the harmonic mean of precision and recall (F1 score). It can also monitor the computational resource consumption during the inference process after the target network model is deployed on the user terminal, such as the utilization of the central processing unit (CPU), the utilization of the graphics processing unit (GPU), inference latency, and storage resource consumption. The performance information reflects the actual inference effect of the target network model on local data in the current round after the original network model has been pruned and deployed.
[0033] In some embodiments, the model parameters may include the weights of a multilayer network, and S102 may specifically include: According to the preset number of cut rate intervals, the cut rate is sampled within the preset numerical range to construct a cut rate dataset, which includes multiple cut rates; The weights of each network layer are pruned using the pruning rates in the dataset. The pruning process is constrained by ensuring the pruned model size meets preset constraints, and the objective is to minimize a preset objective function. The target pruning rate for each network layer is calculated as the pruning rate value whose slope on the rate-distortion output error curve of the weights and activations satisfies a preset Lagrange operator. The target pruning rate of the original network model includes the target pruning rate of multi-layer networks.
[0034] The aforementioned cropping rate dataset may include multiple cropping rates.
[0035] In some embodiments of this application, the cutting rate is sampled within a preset numerical range according to a preset number of cutting rate intervals to construct a cutting rate dataset. For example, the number of cutting rate intervals can be set. Then, uniformly sample the clipping rate within the numerical range of 0 to 1. , This allows us to construct a cropping rate dataset. In practice... It can be set to a positive integer such as 100, depending on the number of model parameters and the size of the dataset.
[0036] The target pruning rate of the original network model described above can include the target pruning rate of a multi-layer network. Each target pruning rate is the pruning rate value on the rate-distortion output error curve of the weights and activations of each layer, which satisfies the slope of a preset Lagrange operator. The target pruning rates can be different for each layer.
[0037] In some embodiments of this application, the weights of each network layer are pruned using the pruning rate dataset, with the constraint that the size of the pruned network model meets a preset constraint, and the objective being to minimize a preset objective function. The target pruning rate of each network layer is then calculated. For example, a Lagrange operator can be used. Find the slope of the rate-distortion output error curve for each layer's weights and activations that is equal to... The clipping ratio at a given point is the optimal clipping ratio for that layer. Within a certain range, traversal methods such as binary search can be used to find the optimal solution that satisfies the constraints of the optimization problem, i.e., the model size. In practice, additional variables can be set. , and then ,right Solve by binary search .
[0038] In this embodiment, a pruning rate dataset is first constructed by sampling the pruning rate within a preset numerical range according to a preset pruning rate interval. The pruning rate dataset includes multiple pruning rates. Then, the weights of each network layer are pruned using the pruning rates in the pruning rate dataset. The target pruning rate of each network layer is calculated with the constraint that the size of the pruned network model meets a preset constraint and the objective function being minimized. Each target pruning rate is the pruning rate value on the rate-distortion output error curve of the weights and activations of each layer, which satisfies the preset Lagrange operator. In this way, target pruning rates are differentially assigned to multiple networks in the original network model, thereby ensuring that the performance of the target network model after pruning is optimal.
[0039] In some embodiments, the above-described S104 may specifically include: Calculate the difference in mean network parameters and the ratio of network parameter variances between the original network model and the first network model; Based on the mean difference and variance ratio of the network parameters, the signal is recovered from the first network model to obtain the second network model; Based on the preset target correction vector, the weights of each layer in the second network model are corrected for deviation to obtain the target network model. The target correction vector is used to correct the deviation between the second network model and the original network model.
[0040] In some embodiments of this application, the difference between the mean of network parameters and the ratio of the variance of network parameters between the original network model and the first network model are calculated. For example, the difference between the mean of network parameters u between the original network model and the first network model can be calculated, and then (WE(W))^2 represents the variance of the original network model before pruning, and (W*-E(W*))^2 represents the variance of the first network model after pruning. Finally, the ratio of the variance of network parameters before and after pruning is calculated: v = (WE(W))^2 / (W* - E(W*))^2.
[0041] In some embodiments of this application, a second network model is obtained by signal recovery of a first network model based on the ratio of the mean difference of network parameters to the variance of network parameters. For example, the signal recovery of the pruned network can be performed based on the ratio of the mean difference u of network parameters to the variance v, and the pruned parameters can be replaced with the recovered parameters. The specific calculation formula is as follows: W* = v * (W* + u).
[0042] The aforementioned target correction vector can be used to correct the deviation between the second network model and the original network model.
[0043] In this embodiment, since the parameters of each layer of the original network model are pruned, which affects the output of the current layer, a target correction vector is superimposed on the second network model to correct the bias of the weights of each layer in the second network model. After superposition, the output error of the current layer can be reduced, so that the target network model can achieve the effect of restoring the performance of the original network model.
[0044] As one implementation of this application, in order to obtain the optimal target correction vector, before correcting the bias of the weights of each layer of the second network model according to the preset target correction vector to obtain the target network model, the above method may further include: In the second network model, variables are set, and the output deviation function between the second network model and the original network model is constructed; The variable corresponding to minimizing the output deviation function is determined as the target correction vector.
[0045] In some embodiments of this application, a second network model is defined as follows: i The layer's Bias correction vector is Δ bi Construct a second network model y ^ i =Layer( xi ; W ^ i , bi +Δ bi ) and the original network model yi =Layer( xi ; Wi , bi Output deviation function Δyi(k) =y^ i(k) yi(k) .
[0046] In some embodiments of this application, the variable corresponding to minimizing the output deviation function is determined as the target correction vector. For example, the target correction vector Δ can be obtained by solving for minimizing the output deviation function. bi .
[0047] In this embodiment, by setting variables in the second network model and constructing the output deviation function between the second network model and the original network model, the optimal target correction vector is obtained by minimizing the output deviation function, which is then used for deviation correction after pruning, thereby restoring the performance of the target network model.
[0048] In some embodiments, the expression for the objective function described above is as follows:
[0049] in, Here, S represents the tradeoff coefficient, and S represents the constraint condition. For the output error sub-function, For loss subfunction, For nuclear norm subfunctions.
[0050] The above trade-off coefficients The constraint S can be set according to the user's actual needs. This application is not limited to a fixed value, and no specific limitation is made here.
[0051] In this embodiment, since the objective function includes an output error subfunction, a loss subfunction, and a kernel norm subfunction, by minimizing the error of the last layer output after network pruning, while maximizing the preservation of the ranking of the attention scores of the visual token by the Visual Transformer (ViT) model, the feature error of the attention layer on subsequent network inference is minimized by parameter pruning, and the addition of the kernel norm facilitates more efficient pruning rate allocation.
[0052] In some embodiments, the formula for the above output error subfunction is as follows:
[0053] in, Indicates the cutting of the first i The parameters of each layer introduce errors into the output of the original network model. For the number of network layers, This is the output of the last layer of the original network model. This is the output of the final layer of the pruned network model.
[0054] In some embodiments, the expression for the above loss function is as follows:
[0055] in, i For the first i Multi-head self-attention layer, h This represents all attention heads in the multi-head self-attention layer. p Indicates the position of the visual token. q To and p All subsequent visual tokens that are paired The first element in the cropped matrix i Layer k The first attention head and the first p The attention score corresponding to each visual token. For the th element in the trimmed matrix i Layer k The first attention head and the first q The attention score corresponding to each visual token. The first element in the original matrix i Layer k The first attention head, the first p The attention score corresponding to each token. The first element in the original matrix i Layer k The first attention head, the first q The attention score corresponding to each token.
[0056] In some embodiments, the expression for the above nuclear norm subfunction is as follows:
[0057] in, For the first in the original network model i The attention score matrix of the multi-head self-attention (MHSA) layer is the nth layer in the pruned network model. i Multi-head self-attention MHSA Attention score matrix of the layer.
[0058] To facilitate understanding of the model trimming method in the embodiments of this application, the actual application process of this model trimming method is described as follows: This application proposes a model pruning method, a novel technique for pruning parameters of artificial intelligence neural networks, such as... Figure 2 As shown, by using bias correction, the model's performance is restored after pruning, so that the distribution of the model parameters is close to the original model before pruning. Therefore, it can be deployed without retraining after pruning, which can reduce the computational burden of neural networks and the demand for computing resources. It is a lightweight pruning method that avoids retraining.
[0059] This application enables significant pruning of mainstream Transformer neural network models, including LLaVA, without requiring retraining. Specifically, by minimizing the error of the final layer output after pruning while maximizing the preservation of the ViT model's attention scores for visual tokens, the impact of parameter pruning on the feature error of the attention layer for subsequent network inference is minimized. Furthermore, a distribution regularization is applied to the pruned network to restore performance. Figure 3 It shows the changes in parameter distribution before and after cropping.
[0060] like Figure 4 As shown, a model trimming method may include the following steps S401 to S405: S401: Minimize the output error of the last layer after network pruning.
[0061] A pruning rate is assigned to all parameter layers of the network (including fully connected and convolutional layers). This pruning rate is a variable, and the Euclidean distance between the output of the last layer before and after pruning each layer is summed. The sum represents the overall impact of pruning all layers on the network. The transformed output error not only takes into account the joint effect of network parameter pruning, but also ensures that the pruning rate is distributed layer by layer to better maintain network task performance, while also facilitating subsequent optimization solutions.
[0062] set up Define the number of network layers. This is the output of the last layer of the original network. The change in the network output is measured using the following formula:
[0063] That is, the Euclidean distance between the two, where , This represents the dimension of the output tensor of the last layer of the Transformer model after flattening. In practice, a small sample can be randomly selected from the training dataset as a calibration dataset. The calibration dataset should ensure a uniform distribution of classes. Alternatively, random noise with a distribution consistent with the target dataset can be used instead of the calibration dataset without affecting the pruning assignment. Model inference is then performed on this dataset to obtain the actual network output error.
[0064] make This indicates that the parameters of the i-th layer are pruned before being fed into the neural network output. The resulting error This indicates that the activations of the i-th layer are pruned and fed into the neural network output. The resulting error. Parameters refer to the weights of all parameterized network layers in the Transformer, including Projection layer, QKV, FC1, FC2, etc. Activations refer to the outputs of any intermediate layers in the network, including token features and Query, Key, Value, etc., from each intermediate layer of the Transformer model. Other non-parameterized layers, such as Attention, Softmax, Layernorm, etc., are not considered. The formula for calculating the network output error using additivity (equivalent to the above output error subfunction) is as follows:
[0065] S402: Minimize the ranking loss of attention scores for visual tokens. The perturbation caused by pruning the ranking of attention scores among visual tokens in the self-attention layers of the ViT network can significantly impact the performance of the ViT task. Therefore, an attention score ranking loss is introduced into the final objective function.
[0066] The pruning scheme should, as far as possible, maintain the order of the pre-self-attention weights among the key tokens for each query token in the MSHA layer. That is, if the attention score matrix of the i-th layer MHSA is... The specific form of the ranking loss of the attention score at this layer (equivalent to the loss sub-function mentioned above) is as follows:
[0067] S403: Combine the kernel norm of the attention map with layer-by-layer clipping rate search. Calculate the kernel norm of the attention layer output after clipping at a specific clipping rate for each layer. The addition of the kernel norm facilitates more efficient clipping rate allocation.
[0068] For the i-th level MSHA, the nuclear norm specification (equivalent to the above nuclear norm subfunction) takes the form:
[0069] S404: Combining the losses from steps S401, S402, and S403, the objective function of the optimization problem is obtained. Existing layer-by-layer pruning methods generally require complex search methods to obtain accurate pruning ratio allocation, or improve solution efficiency by finding suboptimal solutions. Therefore, this application can also use the Lagrange method to solve for the overall optimal pruning ratio allocation.
[0070] The objective function of the optimization problem is:
[0071] in, Here, S represents the tradeoff coefficient, and S is the constraint on model size. Since both the objective function and the constraint function are linear combinations of the pruning rates of the optimization variables at each level, they can be solved using efficient linear programming algorithms, such as the Lagrange method.
[0072] First, on a calibration dataset, based on all possible pruning rates... The objective function value at the specified pruning rate is obtained by performing network inference on the weights of each layer using that pruning rate. In practice, the pruning rate interval can be set. Then, uniformly sample the cutting rate from 0 to 1. , In practice It can be set to a positive integer such as 100, depending on the number of model parameters and the size of the dataset. Next, the collected coefficients are used to solve the problem using the Lagrange method: a Lagrange operator is defined. Find the slope of the rate-distortion output error curve where the weights and activations at each layer are equal to... The clipping ratio value at a given point is the optimal clipping ratio for that layer (equivalent to the target clipping ratio mentioned above), and the slope can be obtained through enumeration search. Within a certain range, traversal methods such as binary search can be used to find the optimal solution that satisfies the constraints of the optimization problem, i.e., the model size. In practice, additional variables can be set. , and then ,right Solve by binary search .
[0073] S405: Distribution regularization and bias correction. A correction vector is superimposed on the bias parameter to ensure that the distribution of the weight after clipping is consistent with that before clipping. Specifically, the correction aims to make the mean and variance of each channel of the weight consistent before and after clipping.
[0074] A) Calculate the variance of the parameters of each layer in the original network before pruning, and also calculate the variance of the parameters of each layer after pruning. Let (WE(W))^2 represent the variance of the original network before pruning, and (W*-E(W*))^2 represent the variance of the network after pruning. Then calculate the ratio of the variances of the network parameters before and after pruning, using the following formula: v = (WE(W))^2 / (W* - E(W*))^2.
[0075] B) Recover the signal from the pruned network based on the difference in mean u and the ratio of variance v, and replace the pruned parameters with the recovered parameters. The specific calculation formula is as follows: W* = v * (W* + u).
[0076] C) A correction operation is also performed on the bias in the parameters of each layer. Specifically, for the output of each layer, the impact of pruning the parameters of the current layer on the output of that layer is statistically analyzed. Next, a variable is set, namely a correction vector to be superimposed on the bias, and an optimal superposition vector is defined such that the superposition reduces the output error of the current layer. Then, the optimal superposition vector (equivalent to the target correction vector mentioned above) is solved through optimization. Finally, the bias value is updated using the solved optimal superposition vector.
[0077] In summary, pruning a network is equivalent to shifting the distribution of parameters from a signal perspective. Correcting the pruned parameters based on this shift restores the network's original performance while avoiding retraining. Since this correction operation only involves basic parameter calculations, it is far more efficient and faster than retraining.
[0078] The experimental results are shown in Table 1 below: Table 1: Pruning results on mainstream ViT network models
[0079] Table 1 above illustrates the pruning results of this application on mainstream ViT models. The results show that, without retraining the compressed network, this application can prune the model by more than 70% while maintaining the original accuracy, i.e., the accuracy reduction is less than two percentage points. The results are also compared with the current best method, X-Pruner. The table shows that, achieving the same pruning rate, the accuracy of this application is higher than the compared methods.
[0080] Based on the model trimming method provided in the above embodiments, this application also provides specific implementation methods of the model trimming device. Please refer to the following embodiments.
[0081] like Figure 5 As shown, the model trimming device 500 provided in this application embodiment may include the following modules: acquisition module 501, calculation module 502, trimming module 503 and correction module 504.
[0082] The acquisition module 501 is used to acquire the model parameters of the original network model; The calculation module 502 is used to calculate the target pruning rate of the original network model based on the model parameters, with the constraint that the size of the pruned model meets the preset constraints, and with the objective function being minimized. The objective function includes an output error sub-function, a loss sub-function, and a nuclear norm sub-function. The output error sub-function is used to calculate the error of the last layer output after pruning the original network model. The loss sub-function is used to calculate the loss of attention scores between visual tokens in each attention layer after pruning the original network model. The nuclear norm sub-function is used to calculate the nuclear norm of each attention layer after pruning the original network model. The pruning module 503 is used to prune the original network model according to the target pruning rate to obtain the first network model; The correction module 504 is used to correct the deviation of the first network model based on the difference between the mean and variance of the network parameters of the original network model and the first network model, as well as the preset target correction vector, to obtain the target network model. The mean and variance of the target network model are consistent with those of the original network model in each channel.
[0083] The model pruning device in this embodiment can calculate a target pruning rate for the original network model based on its parameters, with the pruned model size meeting preset constraints, and aiming to minimize a preset objective function. The objective function includes an output error sub-function, a loss sub-function, and a kernel norm sub-function. The original network model is then pruned according to the target pruning rate to obtain a first network model. Next, the first network model is corrected for deviation based on the difference in mean and variance of network parameters between the original and first network models, and a preset target correction vector, resulting in a target network model. This target network model ensures that the mean and variance of each channel are consistent with the original network model. Thus, in this embodiment, pruning the model using the optimal target pruning rate ensures that the parameters of the original network model are minimally affected by pruning. Furthermore, the deviation correction after pruning not only avoids retraining, saving resources and improving efficiency, but also recovers the performance loss caused by pruning.
[0084] In some embodiments, the model parameters may include the weights of a multilayer network, and the calculation module 502 may specifically include: The building unit is used to sample the cropping rate within a preset numerical range according to a preset cropping rate interval, and build a cropping rate dataset, which includes multiple cropping rates; The first computational unit is used to prune the weights of each network layer using the pruning rates in the dataset. The pruning process is constrained by the original network model size satisfying preset constraints, and the objective is to minimize a preset objective function. The target pruning rate for each network layer is calculated as the pruning rate value whose slope on the rate-distortion output error curve of the weights and activations satisfies a preset Lagrange operator. The target pruning rate of the original network model can include the target pruning rate of multi-layer networks.
[0085] In some embodiments, the correction module 504 described above may specifically include: The second calculation unit is used to calculate the difference in the mean of network parameters and the ratio of the variance of network parameters between the original network model and the first network model. The recovery unit is used to recover the signal of the first network model based on the mean difference and variance ratio of the network parameters to obtain the second network model; The correction unit is used to correct the deviation of the weights of each layer of the second network model according to the preset target correction vector, so as to obtain the target network model. The target correction vector is used to correct the deviation between the second network model and the original network model.
[0086] As one implementation of this application, in order to obtain the optimal target correction vector, the above-mentioned device 500 may further include: The building module is used to set variables in the second network model and construct the output deviation function between the second network model and the original network model; The determination module is used to determine the variable corresponding to the minimized output deviation function as the target correction vector.
[0087] In some embodiments, the expression for the objective function described above is as follows:
[0088] in, Here, S represents the tradeoff coefficient, and S represents the constraint condition. For the output error sub-function, For loss subfunction, For nuclear norm subfunctions.
[0089] In some embodiments, the formula for the above output error subfunction is as follows:
[0090] in, Indicates the cutting of the first i The parameters of each layer introduce errors into the output of the original network model. For the number of network layers, This is the output of the last layer of the original network model. This is the output of the final layer of the pruned network model.
[0091] In some embodiments, the expression for the above loss function is as follows:
[0092] in, i For the first i Multi-head self-attention layer, h This represents all attention heads in the multi-head self-attention layer. p Indicates the position of the visual token. q To and p All subsequent visual tokens that are paired The first element in the cropped matrix i Layer k The first attention head and the first p The attention score corresponding to each visual token. For the th element in the trimmed matrix i Layer k The first attention head and the first q The attention score corresponding to each visual token. The first element in the original matrix i Layer k The first attention head, the first p The attention score corresponding to each token. The first element in the original matrix i Layer k The first attention head, the first q The attention score corresponding to each token.
[0093] In some embodiments, the expression for the above nuclear norm subfunction is as follows:
[0094] in, For the first in the original network model i Multi-head self-attention MHSA The attention score matrix of the layer is the nth layer in the pruned network model. i Multi-head self-attention MHSA Attention score matrix of the layer.
[0095] Figure 6 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.
[0096] An electronic device may include a processor 601 and a memory 602 storing computer program instructions.
[0097] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.
[0098] Memory 602 may include mass storage for data or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 602 is non-volatile solid-state memory.
[0099] In a particular embodiment, memory 602 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Thus, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.
[0100] The processor 601 reads and executes computer program instructions stored in the memory 602 to implement any of the model trimming methods in the above embodiments.
[0101] In one example, the electronic device may also include a communication interface 603 and a bus 610. For example, Figure 6 As shown, the processor 601, memory 602, and communication interface 603 are connected through bus 610 and complete communication with each other.
[0102] The communication interface 603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.
[0103] Bus 610 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.
[0104] The electronic device can execute the model trimming method in the embodiments of this application, thereby achieving the combination Figure 1 and Figure 5 The described model cutting method and apparatus.
[0105] Furthermore, in conjunction with the model trimming methods in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the model trimming methods in the above embodiments.
[0106] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the model trimming methods described in the above embodiments.
[0107] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.
[0108] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0109] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0110] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.
[0111] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.
Claims
1. A model trimming method, characterized in that, include: Obtain the model parameters of the original network model; Based on the model parameters, and with the constraint that the size of the pruned original network model meets a preset constraint, the target pruning rate of the original network model is calculated with the objective of minimizing a preset objective function. The objective function includes an output error sub-function, a loss sub-function, and a kernel norm sub-function. The output error sub-function is used to calculate the error of the output of the last layer after pruning the original network model. The loss sub-function is used to calculate the loss of attention scores between visual tokens in each attention layer after pruning the original network model. The kernel norm sub-function is used to calculate the kernel norm of each self-attention layer after pruning the original network model. The original network model is pruned according to the target pruning rate to obtain the first network model; Based on the difference in mean and variance of network parameters between the original network model and the first network model, as well as the preset target correction vector, the first network model is biased to obtain a target network model. The target network model has the same mean and variance in each channel as the original network model.
2. The method according to claim 1, characterized in that, The model parameters include the weights of the multilayer network. The step of calculating the target pruning rate of the original network model based on the model parameters, with the constraint that the size of the pruned original network model meets a preset constraint, and with the objective of minimizing a preset objective function, includes: According to the preset number of cut rate intervals, the cut rate is sampled within the preset numerical range to construct a cut rate dataset, which includes multiple cut rates; The weights of each layer of the network are pruned using the pruning rates in the pruning rate dataset. The pruned model size satisfies a preset constraint, and the objective is to minimize a preset objective function. The target pruning rate for each layer is calculated as the pruning rate value on the rate-distortion output error curve of the weights and activations of that layer, where the slope satisfies a preset Lagrange operator. The target pruning rate of the original network model includes the target pruning rate of the multi-layer network.
3. The method according to claim 1, characterized in that, The step of correcting the deviation of the first network model based on the difference in mean and variance ratio of network parameters between the original network model and the first network model, and a preset target correction vector, to obtain the target network model includes: Calculate the difference in mean network parameters and the ratio of variance of network parameters between the original network model and the first network model; Based on the mean difference of the network parameters and the variance ratio of the network parameters, signal recovery is performed on the first network model to obtain the second network model; According to the preset target correction vector, the weights of each layer of the second network model are corrected for deviation to obtain the target network model. The target correction vector is used to correct the deviation between the second network model and the original network model.
4. The method according to claim 3, characterized in that, Before correcting the bias of the weights of each layer in the second network model according to the preset target correction vector to obtain the target network model, the method further includes: In the second network model, variables are set, and the output deviation function between the second network model and the original network model is constructed; The variable corresponding to minimizing the output deviation function is determined as the target correction vector.
5. The method according to claim 1, characterized in that, The expression for the objective function is as follows: in, Here, S is the weighting factor, and S is the constraint condition. The output error sub-function is... For the loss sub-function, Let be the nuclear norm subfunction.
6. The method according to claim 5, characterized in that, The formula for the output error sub-function is as follows: in, Indicates the cutting of the first i The parameters of the layers then introduce errors into the output of the original network model. For network layers, This is the output of the last layer of the original network model. This is the output of the final layer of the pruned network model.
7. The method according to claim 5, characterized in that, The expression for the loss function is as follows: Among them, the i For the first i Multi-head self-attention layer, the h This represents all attention heads in a multi-head self-attention layer. p Indicates the location of the visual token, the q To and p All subsequent visual tokens that are paired The first element in the cropped matrix i Layer k The first attention head and the first p The attention score corresponding to each visual token. For the th element in the clipped matrix i Layer k The first attention head and the first q The attention score corresponding to each visual token. The first element in the original matrix i Layer k The first attention head, the first p The attention score corresponding to each token. The first element in the original matrix i Layer k The first attention head, the first q The attention score corresponding to each token.
8. The method according to claim 5, characterized in that, The expression for the nuclear norm subfunction is as follows: in, For the first in the original network model i Multi-head self-attention MHSA The attention score matrix of the layer is the nth layer in the pruned network model. i Multi-head self-attention MHSA Attention score matrix of the layer.
9. A model cutting device, characterized in that, The device includes: The acquisition module is used to obtain the model parameters of the original network model; The calculation module is used to calculate the target pruning rate of the original network model based on the model parameters, with the constraint that the size of the pruned model meets the preset constraints, and with the objective of minimizing the preset objective function. The objective function includes an output error sub-function, a loss sub-function, and a kernel norm sub-function. The output error sub-function is used to calculate the error of the output of the last layer after pruning the original network model. The loss sub-function is used to calculate the loss of attention scores between visual tokens in each attention layer after pruning the original network model. The kernel norm sub-function is used to calculate the kernel norm of each self-attention layer after pruning the original network model. The pruning module is used to prune the original network model according to the target pruning rate to obtain a first network model; The correction module is used to correct the deviation of the first network model based on the difference between the mean and variance of the network parameters of the original network model and the first network model, as well as the preset target correction vector, to obtain a target network model. The target network model has the same mean and variance in each channel as the original network model.
10. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the model trimming method as described in any one of claims 1-8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the model trimming method as described in any one of claims 1-8.
12. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the model trimming method as described in any one of claims 1-8.