Model compression method and related device

Abstract

The embodiment of the present invention provides a model compression method and related devices, which belong to the field of artificial intelligence technology. The method includes: using training data to perform model training to obtain a first deep learning model, and obtaining a first importance score corresponding to the self-attention layer of the first deep learning model; pruning the self-attention layer of the first deep learning model according to the first importance score to obtain a second deep learning model; retraining the second deep learning model using the training data to obtain a third deep learning model, and obtaining a second importance score corresponding to the self-attention layer in the third deep learning model; quantizing the accuracy of the third deep learning model according to the second importance score to obtain a fourth deep learning model; using the first deep learning model to perform distillation processing on the fourth deep learning model to obtain a target deep learning model. Through layer-by-layer adaptive quantization and mixed precision calculation strategies, the computing efficiency is improved and the model reasoning process is accelerated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model compression method and related devices. Background Art

[0002] In recent years, with the improvement of computing power and the optimization of algorithms, deep neural networks have achieved remarkable results in image classification, natural language processing, speech recognition and other fields.

[0003] However, the use of deep learning models often faces huge parameters and high computing requirements, resulting in a significant increase in video memory usage and computing latency, especially in resource-constrained environments (such as mobile devices and edge computing), where deployment becomes difficult. In related technologies, model lightweighting technology mainly reduces the scale of neural network models and reduces computational complexity, so that models can be efficiently inferred and deployed with limited resources. Current technologies mainly use methods such as network pruning and knowledge distillation to lightweight models, but these methods often require a lot of computing resources and time. In addition, under high compression ratios, the accuracy of the pruned model has the problem of poor recovery effect, which seriously affects the overall performance of the model.

[0004] Therefore, it is urgent to design a technical solution to solve at least one of the above technical problems. Summary of the invention

[0005] The main purpose of the embodiments of the present invention is to provide a model compression method and related devices, aiming to solve the problem of reducing model accuracy due to model lightweighting in related technologies.

[0006] In a first aspect, an embodiment of the present invention provides a model compression method, comprising:

[0007] Performing model training using the training data to obtain a first deep learning model, and obtaining a first importance score corresponding to a self-attention layer of the first deep learning model;

[0008] Pruning the self-attention layer of the deep learning model according to the first importance score to obtain a second deep learning model;

[0009] Retraining the second deep learning model using the training data to obtain a third deep learning model, and obtaining a second importance score corresponding to the self-attention layer in the third deep learning model;

[0010] quantifying the accuracy of the third deep learning model according to the second importance score to obtain a fourth deep learning model;

[0011] The fourth deep learning model is distilled using the first deep learning model to obtain a target deep learning model.

[0012] In a second aspect, an embodiment of the present invention provides a model compression device, comprising:

[0013] A first training module is used to perform model training using the training data to obtain a first deep learning model, and obtain a first importance score corresponding to a self-attention layer of the first deep learning model;

[0014] A pruning processing module, used for pruning the self-attention layer of the deep learning model according to the first importance score to obtain a second deep learning model;

[0015] A second training module is used to retrain the second deep learning model using the training data to obtain a third deep learning model, and obtain a second importance score corresponding to the self-attention layer in the third deep learning model;

[0016] A precision quantization module, configured to perform precision quantization on the third deep learning model according to the second importance score to obtain a fourth deep learning model;

[0017] A distillation processing module is used to use the first deep learning model to perform distillation processing on the fourth deep learning model to obtain a target deep learning model.

[0018] In a third aspect, an embodiment of the present invention further provides a terminal device, comprising a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of any model compression method provided in the specification of the present invention are implemented.

[0019] In a fourth aspect, an embodiment of the present invention further provides a storage medium for computer-readable storage, characterized in that the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any model compression method provided in the specification of the present invention.

[0020] The embodiment of the present invention provides a model compression method and a related device, the method comprising: using training data to perform model training to obtain a first deep learning model, and obtaining a first importance score corresponding to the self-attention layer of the first deep learning model; pruning the self-attention layer of the deep learning model according to the first importance score to obtain a second deep learning model; retraining the second deep learning model using the training data to obtain a third deep learning model, and obtaining a second importance score corresponding to the self-attention layer in the third deep learning model; performing precision quantization on the third deep learning model according to the second importance score to obtain a fourth deep learning model; using the first deep learning model to perform distillation processing on the fourth deep learning model to obtain a target deep learning model. The method combines layer-by-layer adaptive quantization and mixed precision calculation strategies to improve the calculation efficiency and significantly accelerate the model reasoning process. Moreover, the method adopts a pruning and quantization joint optimization strategy to have high flexibility and can be applied to different types of deep learning models. Finally, the method can effectively maintain the model accuracy through intelligent pruning and quantization-aware training based on importance scores, avoid the accuracy loss caused by excessive pruning and quantization, and thus solve the problem that the model accuracy cannot be maintained when the model is lightweighted in the related technology, thereby reducing the model accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of a flow chart of a model compression method based on a deep learning model provided in an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of the principle of a model compression method based on a deep learning model provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of a distillation processing algorithm provided in a model compression method based on a deep learning model provided in an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the module structure of a model compression device based on a deep learning model provided in an embodiment of the present invention;

[0026] Figure 5 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0029] It should be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0030] The embodiment of the present invention provides a model compression method and related devices. The model compression method can be applied to a terminal device, which can be an electronic device such as a tablet computer, a laptop computer, a desktop computer, a personal digital assistant, and a wearable device. The terminal device can be a server or a server cluster.

[0031] In recent years, with the improvement of computing power and the optimization of algorithms, deep neural networks have achieved remarkable results in many fields such as image classification, natural language processing, and speech recognition. For example, the bidirectional encoder representation large model, as a pre-trained language model based on the self-attention architecture, performs well in a variety of natural language processing tasks. However, due to its large number of parameters and computing requirements, it leads to significant problems of video memory usage and computing latency, making it difficult to deploy in resource-constrained environments (such as mobile devices and edge computing).

[0032] To solve this problem, model lightweight technology came into being. By reducing the scale of the neural network model and reducing the computational complexity, the model can be efficiently inferred and deployed on resource-limited edge devices. Currently, the widely used model lightweight methods include model compression, model quantization, knowledge distillation, network pruning, etc. From the perspective of whether the model is pre-trained, it can be divided into complex model compression methods and lightweight model design methods. In the prior art, a hybrid solution combines pruning and quantization techniques. Pruning is used to reduce redundant connections in the model, while quantization is used to convert model parameters from high-precision representation to low-precision representation. The combination can achieve a significant reduction in model size and a significant reduction in computational complexity at the same time. However, it lacks a dynamic adjustment mechanism, and it is difficult to restore the accuracy of the model after pruning under high compression ratios, which has a significant impact on the performance and performance of the model.

[0033] In summary, the existing model compression methods have the following problems:

[0034] (1) It is difficult to restore the accuracy of the model after pruning: Under high compression ratio conditions, the existing pruning technology makes it difficult to restore the accuracy of the model after pruning, which affects the model performance. When fine-tuning the pruned model, a large amount of computing resources and time are required, and the accuracy recovery effect is not good.

[0035] (2) It is difficult to find the best balance point during the quantization process: Existing quantization methods usually use a fixed quantization level, which is difficult to flexibly adjust according to the importance of the model layer, resulting in low accuracy or insufficient compression ratio of some layers. The fixed quantization level strategy is difficult to adapt to the computational complexity and importance of different model layers, affecting the overall performance of the model.

[0036] (3) Lack of intelligent and adaptive adjustment capabilities: The pruning and quantization strategies in existing technologies are usually static and lack the ability to dynamically adjust according to the actual runtime model and data. Models exhibit different performance requirements in different application scenarios and data sets, and static strategies are difficult to meet actual needs.

[0037] (4) It is difficult to integrate multi-stage optimization strategies: Existing methods usually treat pruning, quantization and other optimization techniques separately, lacking a unified, multi-stage optimization strategy, resulting in unsatisfactory overall results. The lack of a systematic method to integrate multiple optimization techniques results in insufficient optimization of the model during compression and acceleration.

[0038] (5) The existence of special structures in large-scale deep models: For example, the nonlinear multi-head attention layer and feedforward neural network in the bidirectional encoder representation model make many structured pruning strategies that can be used in convolutional neural networks useless. When designing compact networks, it is also difficult to efficiently prune these special structures while maintaining the original accuracy.

[0039] Therefore, a model compression method is urgently needed to solve the above problems.

[0040] Some embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0041] Please refer to Figure 1 , Figure 1 A flowchart of a model compression method provided by an embodiment of the present invention.

[0042] like Figure 1 As shown, the model compression method includes steps S101 to S105.

[0043] Step S101: Use training data to perform model training to obtain a first deep learning model, and obtain a first importance score corresponding to the self-attention layer of the first deep learning model.

[0044] For example, choose a deep learning framework that is suitable for the task (such as TensorFlow or PyTorch), and then choose a suitable deep learning framework based on the task. For example, if it is an image classification task, you can use PyTorch and common neural network structures such as ResNet or Transformer. Define the model structure in the selected framework, making sure to include self-attention layers or other structures that need to be analyzed for importance.

[0045] Exemplarily, data preprocessing and enhancement techniques, such as normalization, data enhancement, etc., are used to prepare training data, and then model training is performed on the selected deep learning framework based on the training data to obtain a first deep learning model.

[0046] Exemplarily, the API of the selected framework, such as the named_parameters() method of PyTorch, is used to obtain the weight parameters of the self-attention layer in the first deep learning model, and then the importance scoring method, such as gradient-based sensitivity analysis, summation of absolute values ​​of weights, etc., is used to obtain the first importance score, wherein the first importance score can provide a basis and guidance for subsequent model optimization (such as pruning).

[0047] In some embodiments, obtaining a first importance score corresponding to the self-attention layer of the first deep learning model includes: obtaining a first weight matrix corresponding to each attention head of the self-attention layer of the first deep learning model, and calculating the weight amplitude corresponding to the first weight matrix; obtaining the weight gradient of the output of the feedforward neural network of the self-attention layer of the first deep learning model relative to each attention head after forward propagation; obtaining the activation value of each attention head of the self-attention layer of the first deep learning model; and determining the first importance score by performing data fusion according to the weight amplitude, the weight gradient, and the activation value.

[0048] Exemplarily, the named_parameters() method of the first deep learning model is obtained to find the parameters containing the weights of the self-attention layer, and then the first weight matrix of each self-attention head is found according to the first deep learning model. Thus, an absolute value operation is applied to each first weight matrix, and then summed to obtain the weight amplitude.

[0049] Exemplarily, after the forward propagation of the first deep learning model, back propagation is used to obtain the weight gradient of each attention head. And the activation value of each attention head is recorded in the forward propagation of the first deep learning model. Then, the weight amplitude, weight gradient and activation value are fused using weighted summation or other defined methods to obtain the first importance score.

[0050] Specifically, by obtaining the first weight matrix of each attention head and calculating the weight amplitude, it can help understand the importance and distribution of different attention heads in the model during the learning process. This helps to gain a deeper understanding of how the model processes different aspects of the input data and provides a basis for further model interpretation. By obtaining the weight gradient, we can understand the contribution of each attention head to the model output. If the gradient of some attention heads is large, it may indicate that they have a greater impact on the model's decision, which provides guidance for optimizing the model, such as improving the model effect through weight pruning or further feature selection. Obtaining the activation value of each attention head can help verify the attention distribution of the model when processing different inputs. This is crucial to ensure that the model pays attention to the correct information in the learning task, and helps to discover the reasons for possible deviations or misclassifications in the model. Finally, by calculating the importance score of each attention head through data fusion, the contribution of different attention heads in the model decision can be quantified. These scores can provide a basis for subsequent model optimization.

[0051] In an embodiment of the present application, after step S101, a strategy combining structured pruning and unstructured pruning is further adopted to reduce the model structure of the deep learning model, remove redundancy, and achieve model compression of the deep learning model. From the overall principle, first of all, it is necessary to perform structured pruning of the attention head and channel of the deep learning model, and then sparse the remaining weights (unstructured pruning) to further reduce the model parameters. Quantization refers to the representation of high-precision floating-point network parameters with low-precision numbers, thereby reducing the memory and processor time required for storage and calculation. Computational acceleration is achieved by modifying the expression granularity of the model weights, that is, a mixed precision quantization technology that applies different quantization levels to different layers. In combination with the above principles, the specific implementation steps are introduced in detail below.

[0052] Step S102: Prune the self-attention layer of the first deep learning model according to the first importance score to obtain a second deep learning model.

[0053] Exemplarily, according to the first importance score, a pruning strategy is formulated to determine which attention heads will be pruned (removed) and which will be retained. For example, the first importance scores are sorted from high to low, so that 80% of the sorted attention heads are retained, and the 20% of the attention heads with the lowest sorting are pruned, thereby obtaining a second deep learning model. Further optionally, the first deep learning model is pruned and fine-tuned multiple times, and the pruning optimization ratio involved in each fine-tuning process (i.e., the percentage corresponding to the sorting operation, etc.) is dynamically implemented and configured according to the model performance change rate of the deep learning model.

[0054] Thus, the pruning operation of channels and attention heads in the deep learning model is realized, completing the structured pruning of the deep learning model.

[0055] In some embodiments, pruning the self-attention layer of the deep learning model according to the first importance score to obtain a second deep learning model includes: setting a pruning threshold, comparing the pruning threshold with the first importance score, and when the first importance score is less than the pruning threshold, removing the self-attention layer corresponding to the first importance score to obtain the second deep learning model.

[0056] For example, first, a pruning threshold is defined based on expert experience or historical experience, and this threshold will be used to determine which attention heads need to be removed. Thus, the attention heads that need to be removed are screened out according to the set pruning threshold.

[0057] Exemplarily, the importance scores (such as weight amplitude, gradient, and entropy-based indicators) of the attention heads and feedforward neural networks in each self-attention layer are calculated. In practical applications, the weight amplitude is the absolute value of the weight. The smaller the absolute value of the weight corresponding to the neuron, the smaller the impact on the neural network, and the lower the importance of the neuron. The gradient index can be obtained by calculating the mean gradient in multiple batches or multiple rounds of training. The smaller the mean gradient, the less active the neuron is and the lower its importance. There are certain differences between entropy and weights and gradients. Weights and gradients are mainly used to evaluate unstructured single neurons, while entropy is mainly used to evaluate the output distribution of the entire layer or the attention distribution of a self-attention head. The corresponding entropy value is calculated according to the information entropy formula. It can be understood that the high or low entropy value does not directly indicate the importance. It needs to be determined according to the specific task objectives. In certain specific tasks, low entropy values ​​are more important, such as tasks that rely on identifying specific, critical patterns or features.

[0058] Exemplarily, the pruning threshold and the first importance score are compared. When the first importance score is less than the pruning threshold, the self-attention layer corresponding to the first importance score is removed. When the first importance score is greater than or equal to the pruning threshold, the self-attention layer corresponding to the first importance score is retained, thereby obtaining a second deep learning model.

[0059] For example, Figure 2 As shown, when the first importance score is less than the pruning threshold, the self-attention layer corresponding to the first importance score is removed, and when the first importance score is greater than or equal to the pruning threshold, the self-attention layer corresponding to the first importance score is retained.

[0060] Specifically, by removing unimportant attention heads, the number of model parameters and computational complexity can be significantly reduced, thereby achieving model compression and adapting to resource-constrained environments. In addition, by setting the pruning threshold and pruning the self-attention layer, the efficiency, interpretability, and generalization ability of the model can be effectively improved, which is an important step in deep learning model optimization.

[0061] Further optionally, the pruning threshold is adjusted in real time according to the rate of change of model performance after pruning. That is, the pruning threshold is dynamically adjusted based on the rate of change of performance after pruning, and is adjusted in real time according to the model performance after pruning to ensure that the model performance is maximized. Specifically, if the model performance loss is not significant, the threshold can be gradually lowered for deeper pruning. If the performance loss is too large, the threshold can be increased to reduce the number of pruned weights; and the binary search technique is used to quickly find the optimal threshold. Within a given range, the threshold is continuously binary-divided and the performance after each adjustment is evaluated until the optimal balance between performance and complexity is found.

[0062] For example, suppose a convolutional neural network contains multiple convolutional layers and fully connected layers. In order to reduce computational overhead, it is decided to prune the model, that is, to prune connections with smaller weights. For example, by dynamically adjusting the pruning threshold to find an optimal pruning strategy, it can reduce computational complexity while maintaining good model performance. In the initial setting stage, an initial range of the pruning threshold is set, such as a threshold between 0.01 and 0.5. The initial threshold can be set to the middle value of the range, such as 0.25.

[0063] During pruning and performance evaluation, the model is pruned using an initial threshold of 0.25, which means that connections with an absolute weight value less than 0.25 are pruned. Evaluate the pruned model, such as calculating the accuracy and loss of the model on the validation set. Based on the performance changes of the pruned model, decide whether to adjust the threshold. If the performance loss is not significant (for example, the accuracy drops by less than 1%), you can try to lower the threshold so that more connections are pruned. If the performance loss is large (for example, the accuracy drops by more than 5%), you need to increase the threshold and reduce the number of pruned connections to avoid over-pruning.

[0064] Set up a binary search mechanism, for example, divide the threshold range into two parts, such as 0.125 and 0.375. Then, prune the middle value and evaluate its performance. Finally, select the middle value range with better performance to continue the binary search. Repeat the above process in the new range to gradually find the best pruning threshold. This process will continue to narrow the threshold range until an ideal balance is reached between performance and complexity.

[0065] In the actual example, suppose that the following situations are encountered during pruning: for example, the accuracy of the model after pruning with an initial threshold of 0.25 drops by 3%. Or, for example, after adjusting the threshold to 0.2, the accuracy drop is reduced to 1%. For the range of thresholds between 0.2 and 0.25, more refined adjustments are made, such as 0.22 and 0.23. Through continuous adjustment and evaluation, a threshold of 0.22 is finally found, which achieves the best balance between the performance of the pruned model (accuracy drops by 0.5%) and the reduction of complexity.

[0066] By dynamically adjusting the pruning threshold and using binary search techniques, model performance and complexity can be optimized in real time during the pruning process. This process involves constantly evaluating the performance after pruning and making precise adjustments within a given threshold range to find the optimal pruning strategy.

[0067] Step S103: Retrain the second deep learning model using the training data to obtain a third deep learning model, and obtain a second importance score corresponding to the self-attention layer in the third deep learning model.

[0068] Exemplarily, the second deep learning model is retrained using the training data to obtain a third deep learning model.

[0069] Exemplarily, using a method similar to the previous one, the second importance score corresponding to each attention head is calculated based on the weight amplitude, weight gradient, and activation value of the self-attention layer of the third deep learning model.

[0070] In some embodiments, the method of performing precision quantization on the third deep learning model according to the second importance score to obtain a fourth deep learning model includes: obtaining a second weight matrix corresponding to each self-attention layer in the third deep learning model; performing unstructured pruning on the third deep learning model according to the second weight matrix to obtain a fifth deep learning model; and performing precision quantization on the fifth deep learning model using the second importance score to obtain the fourth deep learning model.

[0071] Exemplarily, the second weight matrix in the third deep learning model is obtained, and then the second weight matrix is ​​used to perform unstructured pruning, that is, weights less than a threshold are directly eliminated according to the matrix values ​​in the second weight matrix, so as to obtain the fifth deep learning model.

[0072] Exemplarily, the accuracy of the pruned fifth deep learning model is quantified according to the second importance score, and different second importance scores are assigned different accuracies, thereby obtaining the fourth deep learning model.

[0073] Specifically, by obtaining the second weight matrix of each self-attention layer in the third deep learning model and performing unstructured pruning, and then using the second importance score to quantify the accuracy of the pruned model, the model can be effectively compressed, the model reasoning speed can be improved, the computational cost can be reduced, the model's deployability and interpretability can be enhanced, and the model's adaptability and flexibility can be improved. These effects are very important for deep learning model optimization in practical applications.

[0074] In some embodiments, performing unstructured pruning on the third deep learning model according to the second weight matrix to obtain a fifth deep learning model includes: determining a weight threshold corresponding to unstructured pruning; calculating a weight amplitude corresponding to each weight in the second weight matrix; and when the weight amplitude is lower than the weight threshold, removing neurons corresponding to the weight amplitude to obtain the fifth deep learning model.

[0075] For example, a suitable weight threshold is selected based on experience and model characteristics, or the best weight threshold is determined through algorithms and experiments, for example, using a validation set to select a threshold so that the pruned model has the best performance on the validation set.

[0076] Exemplarily, after determining the pruning threshold, the weight amplitude corresponding to each weight in the second weight matrix can be calculated, and then the neurons with weight amplitudes lower than the pruning threshold can be removed according to the pruning threshold. Thus, according to the calculated weight amplitude and the set pruning threshold, the neurons with weight amplitudes lower than the threshold are removed, thereby obtaining the fifth deep learning model.

[0077] Specifically, determining the weight threshold of unstructured pruning can be optimized according to specific model requirements and task characteristics, thereby better balancing the size and performance of the model. By calculating the magnitude of each weight and setting a reasonable threshold, the degree of pruning can be precisely controlled, thereby minimizing the computational cost and inference time of the model. In addition, by adjusting the weight threshold and pruning strategy, the model structure can be dynamically adjusted according to actual needs to meet the performance and resource requirements of different application scenarios.

[0078] In some implementations, determining the weight threshold corresponding to unstructured pruning includes: determining a target percentage required for data screening; and determining the weight threshold from the second weight matrix according to the target percentage.

[0079] For example, a target percentage is defined, which is usually a value between 0 and 1, indicating the proportion of weights that you want to retain. For example, a target percentage of 0.8 means that you want to retain 80% of the weights.

[0080] Exemplarily, the weight threshold is determined from the second weight matrix according to the target percentage. For example, all weights in the second weight matrix are extracted and sorted. Thus, the number of weights to be retained is determined according to the target percentage, and then the corresponding weight value is found as the threshold.

[0081] Specifically, by setting the target percentage, the sparsity of the pruned model can be precisely controlled, thereby balancing the accuracy and efficiency of the model. In addition, determining the weight threshold according to the target percentage helps retain the weights that contribute more to the model performance and reduce the impact of pruning on the model accuracy.

[0082] For example, the above steps S102 and S103 may be executed repeatedly to achieve more accurate model control. Figure 2 It can be seen that after multiple adjustments, Figure 2 The model structure on the left side is sparse after pruning. Figure 2 The model structure on the right side of the figure is constructed, thereby removing the redundant nodes in the model structure on the left side, which can not only ensure the maximum performance of the model, but also further reduce the model parameters.

[0083] Step S104: quantify the accuracy of the third deep learning model according to the second importance score to obtain a fourth deep learning model.

[0084] Exemplarily, the quantization method may include reducing the number of weight digits, using fixed-point representation, and other techniques to reduce the storage requirements and computational costs of the model. According to the second importance score obtained by the evaluation, the third deep learning model is subjected to precision quantization. For example, when the second importance score is larger, the number of weight digits is larger, and when the second importance score is smaller, the number of weight digits is smaller.

[0085] In some embodiments, the use of the second importance score to quantify the accuracy of the fifth deep learning model to obtain the fourth deep learning model includes: determining the segmentation parameters required for accuracy quantization, comparing the second importance score with the segmentation parameters, when the second importance score is greater than or equal to the segmentation parameters, setting the information accuracy of the self-attention layer corresponding to the second importance score to the first accuracy information; when the second importance score is less than the segmentation parameter, setting the information accuracy of the self-attention layer corresponding to the second importance score to the second accuracy information; and obtaining the fourth deep learning model according to the first accuracy information and the second accuracy information.

[0086] Exemplarily, a suitable segmentation parameter strategy is selected according to the application scenario and model characteristics. The segmentation parameter may be a preset threshold value or a value dynamically determined according to the distribution characteristics of the second importance score.

[0087] Exemplarily, the second importance score of each self-attention layer is compared with the set segmentation parameter. When the second importance score is greater than or equal to the segmentation parameter, the information accuracy of the self-attention layer is set to the first accuracy information, and the first accuracy information is used to characterize a higher accuracy setting, such as an 8-bit data storage method; when the second importance score is less than the segmentation parameter, the information accuracy of the self-attention layer is set to the second accuracy information, and the second accuracy information is used to characterize a lower accuracy setting, such as a 4-bit data storage method.

[0088] Exemplarily, the fifth deep learning model is adjusted using the set first precision information and second precision information to obtain a fourth deep learning model.

[0089] Specifically, according to the segmentation parameters and the second importance score, the most suitable accuracy settings can be applied to different parts of the model, thereby minimizing the computation and storage requirements while maintaining model performance. In addition, accurate accuracy settings can reduce the computational cost and energy consumption of the model, especially in mobile devices or edge computing environments, which can significantly reduce energy consumption and operating costs.

[0090] The above steps can modify the expression granularity of the model weights to accelerate the calculation of the model, that is, to configure different quantization levels for different layers, thereby realizing mixed precision quantization of the model. In other words, according to the importance and computational complexity of the model structure, different quantization levels are applied to different layers, and the optimal balance between accuracy and compression ratio is achieved through mixed precision quantization technology.

[0091] Step S105: Use the first deep learning model to perform distillation processing on the fourth deep learning model to obtain a target deep learning model.

[0092] For example, the distillation technique used, such as knowledge distillation, usually involves transferring the knowledge of a complex model (teacher model) to a simpler model (student model). Figure 3 As shown, the first deep learning model is used as the teacher model and the fourth deep learning model is used as the student model. Thus, the loss function is designed to enable the student model to learn the prediction results and feature representations of the teacher model. For example, the loss function is the mean square error, cross entropy loss, etc. The specific choice depends on the task and model structure.

[0093] Exemplarily, the student model is trained using a training set with labeled data, and the parameters of the student model are adjusted according to the output of the teacher model. During the training process, the prediction results (soft targets) of the teacher model are introduced into the training of the student model as auxiliary information. Thus, the performance of the distilled student model is evaluated using a validation set or a test set, ensuring that the model size and computational efficiency are improved while retaining accuracy, thereby obtaining the target deep learning model.

[0094] In some embodiments, the use of the first deep learning model to perform distillation processing on the fourth deep learning model to obtain a target deep learning model includes: determining the first deep learning model as a teacher model, and determining the fourth deep learning model as a student model; training the teacher model and the student model respectively with the training data to obtain a first loss value corresponding to the self-attention layer between the teacher model and the student model, a second loss value corresponding to the hidden layer between the teacher model and the student model, a third loss value corresponding to the embedding layer between the teacher model and the student model, and a fourth loss value corresponding to the prediction layer between the teacher model and the student model; determining a first distillation loss function according to the first loss value, the second loss value, the third loss value, and the fourth loss value, and obtaining the target deep learning model according to the first distillation loss function.

[0095] Exemplarily, the first deep learning model is determined as the teacher model, and the fourth deep learning model is determined as the student model. Then, a function for calculating the loss values ​​of different layers between the teacher model and the student model is defined. These losses generally include self-attention layer loss, hidden layer loss, embedding layer loss, and prediction layer loss.

[0096] Exemplarily, a distillation loss function is defined based on the calculated losses of different layers, and the loss values ​​between different layers, including the first loss value, the second loss value, the third loss value and the fourth loss value, are combined, and then the first loss value, the second loss value, the third loss value and the fourth loss value are summed to obtain a first distillation loss function.

[0097] Exemplarily, an optimizer is used to minimize the defined first distillation loss function to adjust the parameters of the student model, thereby effectively transferring the knowledge of the first deep learning model to the fourth deep learning model to obtain the target deep learning model.

[0098] Specifically, through loss function definition and optimization, the complex features and predictive capabilities of the teacher model are transferred to the student model, helping the student model to reduce model complexity while retaining accuracy. According to the loss values ​​between different layers, the training process of the student model is precisely adjusted to improve the performance, speed and resource utilization of the model. This effectively realizes the distillation processing of the deep learning model, organically transfers the knowledge of the teacher model to the student model, and obtains a more optimized and efficient target deep learning model, which is suitable for various complex tasks and practical application scenarios.

[0099] In some embodiments, obtaining the target deep learning model according to the first distillation loss function includes: determining the output loss function between the output label corresponding to the student model and the sample hard label corresponding to the training sample; fusing the first distillation loss function and the output loss function to obtain the target loss function; and obtaining the target deep learning model according to the target loss function.

[0100] Exemplarily, an output loss function is obtained between an output label of a student model and a sample hard label corresponding to a training sample. The output loss function may be a cross entropy loss function.

[0101] Exemplarily, the first distillation loss function and the output loss function are weightedly summed to obtain a target loss function, and then when the loss value corresponding to the target loss function meets a preset condition, the target deep learning model is obtained.

[0102] Specifically, by adjusting the weights of the first distillation loss function and the output loss function, the best balance between complexity and performance can be found, so that the final model can achieve the best state in terms of resource consumption and task performance, thereby obtaining the optimal target deep learning model.

[0103] In some embodiments, the target deep learning model is obtained, and the method further includes: using the first deep learning model, the second deep learning model, and the third deep learning model as teacher models, and determining the fourth deep learning model as a student model; using the training data to train the first deep learning model and the student model respectively to obtain a first attention loss corresponding to the self-attention layer between the first deep learning model and the student model, a second attention loss corresponding to the self-attention layer between the second deep learning model and the student model, and a third attention loss corresponding to the self-attention layer between the third deep learning model and the student model, and determining a fifth loss value corresponding to the self-attention layer between the teacher model and the student model based on the first attention loss, the second attention loss, and the third attention loss; obtaining a first hidden loss corresponding to the hidden layer between the first deep learning model and the student model, a second hidden loss corresponding to the hidden layer between the second deep learning model and the student model, and a third hidden loss corresponding to the hidden layer between the third deep learning model and the student model, and determining a fifth loss value corresponding to the self-attention layer between the teacher model and the student model based on the first hidden loss, the second hidden loss, and the third The method comprises the following steps: determining a sixth loss value corresponding to the hidden layer between the teacher model and the student model by using a hidden loss; obtaining a first embedding loss corresponding to the embedding layer between the first deep learning model and the student model, a second embedding loss corresponding to the embedding layer between the second deep learning model and the student model, and a third embedding loss corresponding to the embedding layer between the third deep learning model and the student model, and determining a seventh loss value corresponding to the embedding layer between the teacher model and the student model according to the first embedding loss, the second embedding loss, and the third embedding loss; obtaining a first prediction loss corresponding to the prediction layer between the first deep learning model and the student model, a second prediction loss corresponding to the prediction layer between the second deep learning model and the student model, and a third prediction loss corresponding to the embedding layer between the third deep learning model and the student model, and determining an eighth loss value corresponding to the prediction layer between the teacher model and the student model according to the first prediction loss, the second prediction loss, and the third prediction loss; determining a second distillation loss function according to the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value, and obtaining the target deep learning model according to the second distillation loss function.

[0104] Exemplarily, the first deep learning model, the second deep learning model, and the third deep learning model are all defined as teacher models, and the first deep learning model, the second deep learning model, and the third deep learning model are in a parallel relationship in the teacher model. And the fourth deep learning model is also defined as a student model.

[0105] Exemplarily, the training data is used to respectively train the first deep learning model and the student model to obtain a first attention loss corresponding to the self-attention layer between the first deep learning model and the student model. Similarly, the second deep learning model is trained to obtain a second attention loss corresponding to the self-attention layer between the second deep learning model and the student model. Similarly, the third deep learning model is trained to obtain a third attention loss corresponding to the self-attention layer between the first deep learning model and the student model.

[0106] Exemplarily, since the model accuracies corresponding to the first deep learning model, the second deep learning model, and the third deep learning model are different, different weights are set for the first deep learning model, the second deep learning model, and the third deep learning model, respectively, and then the weights are used to determine the fifth loss value corresponding to the self-attention layer between the teacher model and the student model by taking a weighted average of the first attention loss, the second attention loss, and the third attention loss.

[0107] Exemplarily, the training data is used to respectively train the first deep learning model and the student model to obtain a first hidden loss corresponding to the hidden layer between the first deep learning model and the student model. Similarly, the second deep learning model is trained to obtain a second hidden loss corresponding to the hidden layer between the second deep learning model and the student model. Similarly, the third deep learning model is trained to obtain a third hidden loss corresponding to the hidden layer between the first deep learning model and the student model.

[0108] Exemplarily, since the model accuracies corresponding to the first deep learning model, the second deep learning model, and the third deep learning model are different, different weights are set for the first deep learning model, the second deep learning model, and the third deep learning model, respectively, and then the weights are used to determine the sixth loss value corresponding to the hidden layer between the teacher model and the student model by taking a weighted average of the first hidden loss, the second hidden loss, and the third hidden loss.

[0109] Exemplarily, the training data is used to respectively train the first deep learning model and the student model to obtain a first embedding loss corresponding to the embedding layer between the first deep learning model and the student model. Similarly, the second deep learning model is trained to obtain a second embedding loss corresponding to the embedding layer between the second deep learning model and the student model. Similarly, the third deep learning model is trained to obtain a third embedding loss corresponding to the embedding layer between the first deep learning model and the student model.

[0110] Exemplarily, since the model accuracies corresponding to the first deep learning model, the second deep learning model, and the third deep learning model are different, different weights are set for the first deep learning model, the second deep learning model, and the third deep learning model, respectively, and then the weights are used to determine the seventh loss value corresponding to the embedding layer between the teacher model and the student model by taking a weighted average of the first embedding loss, the second embedding loss, and the third embedding loss.

[0111] Exemplarily, the training data is used to train the first deep learning model and the student model respectively to obtain a first prediction loss corresponding to the prediction layer between the first deep learning model and the student model. Similarly, the second deep learning model is trained to obtain a second prediction loss corresponding to the prediction layer between the second deep learning model and the student model. Similarly, the third deep learning model is trained to obtain a third prediction loss corresponding to the prediction layer between the first deep learning model and the student model.

[0112] Exemplarily, since the model accuracies corresponding to the first deep learning model, the second deep learning model, and the third deep learning model are different, different weights are set for the first deep learning model, the second deep learning model, and the third deep learning model, respectively, and then the weights are used to determine the eighth loss value corresponding to the prediction layer between the teacher model and the student model by taking a weighted average of the first prediction loss, the second prediction loss, and the third prediction loss.

[0113] Exemplarily, a distillation loss function is defined according to the calculated losses of different layers, and the loss values ​​between different layers, including the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value, are combined, and then the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value are summed to obtain a second distillation loss function. Thus, the target deep learning model is obtained according to the second distillation loss function.

[0114] In some embodiments, determining the second distillation loss function based on the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value includes: determining the first weight information corresponding to the fifth loss value, the second weight information corresponding to the sixth loss value, the third weight information corresponding to the seventh loss value, and the fourth weight information corresponding to the eighth loss value; performing information fusion on the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value according to the first weight information, the second weight information, the third weight information, and the fourth weight information to obtain the second distillation loss function.

[0115] Exemplarily, the first weight information corresponding to the fifth loss value, the second weight information corresponding to the sixth loss value, the third weight information corresponding to the seventh loss value, and the fourth weight information corresponding to the eighth loss value are determined according to the different model accuracies corresponding to the first deep learning model, the second deep learning model, and the third deep learning model.

[0116] Exemplarily, the first weight information, the second weight information, the third weight information, and the fourth weight information are used to fuse the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value to obtain a second distillation loss function.

[0117] In some embodiments, after obtaining the target deep learning model, the method further includes: performing singular value decomposition on a third weight matrix corresponding to the target deep learning model to obtain a fourth weight matrix; and using the fourth weight matrix to replace the weights of the target deep learning model to obtain the replaced target deep learning model.

[0118] Exemplarily, singular value decomposition is performed on the third weight matrix corresponding to the target deep learning model. Singular value decomposition is a method of decomposing a matrix into a specific form, which is usually used to reduce the complexity of the matrix and extract key information.

[0119] Exemplarily, the fourth weight matrix can be obtained through the results obtained by singular value decomposition. This matrix may be a new matrix after dimensionality reduction or other forms of processing, and the specific form depends on the result of singular value decomposition. Then, the fourth weight matrix is ​​used to replace the original third weight matrix in the target deep learning model. This process will update the parameters of the model to reflect the information contained in the fourth weight matrix. After completing the weight replacement, the replaced target deep learning model is obtained. This model will reflect the characteristics and performance changes after using the fourth weight matrix.

[0120] See also Figure 4 , Figure 4A model compression device 200 is provided for an embodiment of the present application, and the model compression device 200 includes a first training module 201, a pruning processing module 202, a second training module 203, a precision quantization module 204, and a distillation processing module 205, wherein the first training module 201 is used to perform model training using training data to obtain a first deep learning model, and obtain a first importance score corresponding to the self-attention layer of the first deep learning model; the pruning processing module 202 is used to prune the self-attention layer of the first deep learning model according to the first importance score to obtain a second deep learning model; the second training module 203 is used to retrain the second deep learning model using the training data to obtain a third deep learning model, and obtain a second importance score corresponding to the self-attention layer in the third deep learning model; the precision quantization module 204 is used to perform precision quantization on the third deep learning model according to the second importance score to obtain a fourth deep learning model; the distillation processing module 205 is used to perform distillation processing on the fourth deep learning model using the first deep learning model to obtain a target deep learning model.

[0121] In some implementations, the first training module 201, during the process of obtaining the first importance score corresponding to the self-attention layer of the first deep learning model, performs:

[0122] Obtain a first weight matrix corresponding to each attention head of the self-attention layer of the first deep learning model, and calculate a weight amplitude corresponding to the first weight matrix;

[0123] Obtaining a weight gradient of an output of the first deep learning model relative to each attention head after forward propagation of a feedforward neural network of a self-attention layer of the first deep learning model;

[0124] Obtaining an activation value of each attention head of the self-attention layer of the first deep learning model;

[0125] The first importance score is determined by performing data fusion according to the weight magnitude, the weight gradient, and the activation value.

[0126] In some implementations, the pruning processing module 202, in the process of pruning the self-attention layer of the first deep learning model according to the first importance score to obtain the second deep learning model, performs:

[0127] A pruning threshold is set, and the pruning threshold is compared with the first importance score, and the pruning threshold is adjusted in real time according to the rate of change of model performance after pruning; when the first importance score is less than the pruning threshold, the self-attention layer corresponding to the first importance score is removed to obtain the second deep learning model.

[0128] In some implementations, during the process of performing precision quantization on the third deep learning model according to the second importance score to obtain a fourth deep learning model, the precision quantization module 204 executes:

[0129] Obtain a second weight matrix corresponding to each self-attention layer in the third deep learning model;

[0130] Performing unstructured pruning on the third deep learning model according to the second weight matrix to obtain a fifth deep learning model;

[0131] The fourth deep learning model is obtained by using the second importance score to quantify the accuracy of the fifth deep learning model.

[0132] In some implementations, the precision quantization module 204 performs, during the process of performing unstructured pruning on the third deep learning model according to the second weight matrix to obtain the fifth deep learning model:

[0133] Determine the weight threshold corresponding to unstructured pruning;

[0134] Calculating a weight magnitude corresponding to each weight in the second weight matrix;

[0135] When the weight amplitude is lower than the weight threshold, the neurons corresponding to the weight amplitude are removed to obtain the fifth deep learning model.

[0136] In some implementations, during the process of determining the weight threshold corresponding to the unstructured pruning, the precision quantization module 204 performs:

[0137] Determine the target percentage required for data screening;

[0138] The weight threshold is determined from the second weight matrix according to the target percentage.

[0139] In some implementations, during the process of using the second importance score to perform precision quantification on the fifth deep learning model to obtain the fourth deep learning model, the precision quantification module 204 executes:

[0140] Determine a segmentation parameter required for precision quantization, compare the second importance score with the segmentation parameter, and when the second importance score is greater than or equal to the segmentation parameter, set the information precision of the self-attention layer corresponding to the second importance score to the first precision information;

[0141] When the second importance score is less than the segmentation parameter, setting the information accuracy of the self-attention layer corresponding to the second importance score as the second accuracy information;

[0142] The fourth deep learning model is obtained according to the first precision information and the second precision information.

[0143] In some implementations, the distillation processing module 205 performs, during the process of using the first deep learning model to distill the fourth deep learning model to obtain a target deep learning model:

[0144] Determining the first deep learning model as a teacher model, and determining the fourth deep learning model as a student model;

[0145] Using the training data to train the teacher model and the student model respectively to obtain a first loss value corresponding to the self-attention layer between the teacher model and the student model, a second loss value corresponding to the hidden layer between the teacher model and the student model, a third loss value corresponding to the embedding layer between the teacher model and the student model, and a fourth loss value corresponding to the prediction layer between the teacher model and the student model;

[0146] A first distillation loss function is determined according to the first loss value, the second loss value, the third loss value, and the fourth loss value, and the target deep learning model is obtained according to the first distillation loss function.

[0147] In some implementations, the distillation processing module 205 performs, during the process of obtaining the target deep learning model according to the first distillation loss function:

[0148] Determine an output loss function between an output label corresponding to the student model and a sample hard label corresponding to the training sample;

[0149] Fusion of the first distillation loss function and the output loss function to obtain a target loss function;

[0150] The target deep learning model is obtained according to the target loss function.

[0151] In some implementations, after obtaining the target deep learning model, the distillation processing module 205 further performs:

[0152] The first deep learning model, the second deep learning model, and the third deep learning model are all used as teacher models, and the fourth deep learning model is determined as a student model;

[0153] Using the training data to train the first deep learning model and the student model respectively to obtain a first attention loss corresponding to the self-attention layer between the first deep learning model and the student model, a second attention loss corresponding to the self-attention layer between the second deep learning model and the student model, and a third attention loss corresponding to the self-attention layer between the third deep learning model and the student model, and determine a fifth loss value corresponding to the self-attention layer between the teacher model and the student model according to the first attention loss, the second attention loss, and the third attention loss;

[0154] Obtain a first hidden loss corresponding to a hidden layer between the first deep learning model and the student model, a second hidden loss corresponding to a hidden layer between the second deep learning model and the student model, and a third hidden loss corresponding to a hidden layer between the third deep learning model and the student model, and determine a sixth loss value corresponding to a hidden layer between the teacher model and the student model according to the first hidden loss, the second hidden loss, and the third hidden loss;

[0155] Obtain a first embedding loss corresponding to the embedding layer between the first deep learning model and the student model, a second embedding loss corresponding to the embedding layer between the second deep learning model and the student model, and a third embedding loss corresponding to the embedding layer between the third deep learning model and the student model, and determine a seventh loss value corresponding to the embedding layer between the teacher model and the student model according to the first embedding loss, the second embedding loss, and the third embedding loss;

[0156] Obtain a first prediction loss corresponding to the prediction layer between the first deep learning model and the student model, a second prediction loss corresponding to the prediction layer between the second deep learning model and the student model, and a third prediction loss corresponding to the embedding layer between the third deep learning model and the student model, and determine an eighth loss value corresponding to the prediction layer between the teacher model and the student model according to the first prediction loss, the second prediction loss, and the third prediction loss;

[0157] A second distillation loss function is determined according to the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value, and the target deep learning model is obtained according to the second distillation loss function.

[0158] In some implementations, the distillation processing module 205, in the process of determining the second distillation loss function according to the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value, performs:

[0159] Determine first weight information corresponding to the fifth loss value, second weight information corresponding to the sixth loss value, third weight information corresponding to the seventh loss value, and fourth weight information corresponding to the eighth loss value;

[0160] The fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value are fused according to the first weight information, the second weight information, the third weight information, and the fourth weight information to obtain the second distillation loss function.

[0161] In some implementations, after obtaining the target deep learning model, the distillation processing module 205 further performs:

[0162] Performing singular value decomposition on a third weight matrix corresponding to the target deep learning model to obtain a fourth weight matrix;

[0163] The fourth weight matrix is ​​used to perform weight replacement on the target deep learning model to obtain the replaced target deep learning model.

[0164] In some implementations, the model compression apparatus 200 may be applied to a terminal device.

[0165] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the model compression device 200 described above can refer to the corresponding process in the aforementioned model compression method embodiment, and will not be repeated here.

[0166] See also Figure 5 , Figure 5 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present invention.

[0167] like Figure 5 As shown, the terminal device 300 includes a processor 301 and a memory 302 , and the processor 301 and the memory 302 are connected via a bus 303 , such as an I2C (Inter-integrated Circuit) bus.

[0168] Specifically, the processor 301 is used to provide computing and control capabilities to support the operation of the entire terminal device. The processor 301 can be a central processing unit (CPU), and the processor 301 can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0169] Specifically, the memory 302 may be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a mobile hard disk.

[0170] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a partial structure related to the embodiment of the present invention, and does not constitute a limitation on the terminal device to which the embodiment of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0171] The processor is used to run a computer program stored in the memory, and implement any one of the model compression methods provided by the embodiments of the present invention when executing the computer program.

[0172] In one embodiment, the processor is used to run a computer program stored in the memory, and implements the following steps when executing the computer program:

[0173] Performing model training using the training data to obtain a first deep learning model, and obtaining a first importance score corresponding to a self-attention layer of the first deep learning model;

[0174] Pruning the self-attention layer of the first deep learning model according to the first importance score to obtain a second deep learning model;

[0175] Retraining the second deep learning model using the training data to obtain a third deep learning model, and obtaining a second importance score corresponding to the self-attention layer in the third deep learning model;

[0176] quantifying the accuracy of the third deep learning model according to the second importance score to obtain a fourth deep learning model;

[0177] The fourth deep learning model is distilled using the first deep learning model to obtain a target deep learning model.

[0178] In some implementations, the processor 301, in the process of obtaining the first importance score corresponding to the self-attention layer of the first deep learning model, executes:

[0179] Obtain a first weight matrix corresponding to each attention head of the self-attention layer of the first deep learning model, and calculate a weight amplitude corresponding to the first weight matrix;

[0180] Obtaining a weight gradient of an output of the first deep learning model relative to each attention head after forward propagation of a feedforward neural network of a self-attention layer of the first deep learning model;

[0181] Obtaining an activation value of each attention head of the self-attention layer of the first deep learning model;

[0182] The first importance score is determined by performing data fusion according to the weight magnitude, the weight gradient, and the activation value.

[0183] In some implementations, the processor 301, in the process of pruning the self-attention layer of the first deep learning model according to the first importance score to obtain the second deep learning model, executes:

[0184] A pruning threshold is set, and the pruning threshold is compared with the first importance score; the pruning threshold is adjusted in real time according to the rate of change of model performance after pruning; when the first importance score is less than the pruning threshold, the self-attention layer corresponding to the first importance score is removed to obtain the second deep learning model.

[0185] In some implementations, during the process of quantifying the accuracy of the third deep learning model according to the second importance score to obtain a fourth deep learning model, the processor 301 executes:

[0186] Obtain a second weight matrix corresponding to each self-attention layer in the third deep learning model;

[0187] Performing unstructured pruning on the third deep learning model according to the second weight matrix to obtain a fifth deep learning model;

[0188] The fourth deep learning model is obtained by using the second importance score to quantify the accuracy of the fifth deep learning model.

[0189] In some implementations, the processor 301, during the process of performing unstructured pruning on the third deep learning model according to the second weight matrix to obtain the fifth deep learning model, executes:

[0190] Determine the weight threshold corresponding to unstructured pruning;

[0191] Calculating a weight magnitude corresponding to each weight in the second weight matrix;

[0192] When the weight amplitude is lower than the weight threshold, the neurons corresponding to the weight amplitude are removed to obtain the fifth deep learning model.

[0193] In some implementations, the processor 301, during the process of determining the weight threshold corresponding to the unstructured pruning, executes:

[0194] Determine the target percentage required for data screening;

[0195] The weight threshold is determined from the second weight matrix according to the target percentage.

[0196] In some implementations, during the process of using the second importance score to perform precision quantization on the fifth deep learning model to obtain the fourth deep learning model, the processor 301 executes:

[0197] Determine a segmentation parameter required for precision quantization, compare the second importance score with the segmentation parameter, and when the second importance score is greater than or equal to the segmentation parameter, set the information precision of the self-attention layer corresponding to the second importance score to the first precision information;

[0198] When the second importance score is less than the segmentation parameter, setting the information accuracy of the self-attention layer corresponding to the second importance score as the second accuracy information;

[0199] The fourth deep learning model is obtained according to the first precision information and the second precision information.

[0200] In some implementations, the processor 301, in the process of using the first deep learning model to perform distillation processing on the fourth deep learning model to obtain a target deep learning model, executes:

[0201] Determining the first deep learning model as a teacher model, and determining the fourth deep learning model as a student model;

[0202] Using the training data to train the teacher model and the student model respectively to obtain a first loss value corresponding to the self-attention layer between the teacher model and the student model, a second loss value corresponding to the hidden layer between the teacher model and the student model, a third loss value corresponding to the embedding layer between the teacher model and the student model, and a fourth loss value corresponding to the prediction layer between the teacher model and the student model;

[0203] A first distillation loss function is determined according to the first loss value, the second loss value, the third loss value, and the fourth loss value, and the target deep learning model is obtained according to the first distillation loss function.

[0204] In some implementations, the processor 301, in the process of obtaining the target deep learning model according to the first distillation loss function, executes:

[0205] Determine an output loss function between an output label corresponding to the student model and a sample hard label corresponding to the training sample;

[0206] Fusion of the first distillation loss function and the output loss function to obtain a target loss function;

[0207] The target deep learning model is obtained according to the target loss function.

[0208] In some implementations, the processor 301, in the process of obtaining the target deep learning model, further executes:

[0209] The first deep learning model, the second deep learning model, and the third deep learning model are all used as teacher models, and the fourth deep learning model is determined as a student model;

[0210] Using the training data to train the first deep learning model and the student model respectively to obtain a first attention loss corresponding to the self-attention layer between the first deep learning model and the student model, a second attention loss corresponding to the self-attention layer between the second deep learning model and the student model, and a third attention loss corresponding to the self-attention layer between the third deep learning model and the student model, and determine a fifth loss value corresponding to the self-attention layer between the teacher model and the student model according to the first attention loss, the second attention loss, and the third attention loss;

[0211] Obtain a first hidden loss corresponding to a hidden layer between the first deep learning model and the student model, a second hidden loss corresponding to a hidden layer between the second deep learning model and the student model, and a third hidden loss corresponding to a hidden layer between the third deep learning model and the student model, and determine a sixth loss value corresponding to a hidden layer between the teacher model and the student model according to the first hidden loss, the second hidden loss, and the third hidden loss;

[0212] Obtain a first embedding loss corresponding to the embedding layer between the first deep learning model and the student model, a second embedding loss corresponding to the embedding layer between the second deep learning model and the student model, and a third embedding loss corresponding to the embedding layer between the third deep learning model and the student model, and determine a seventh loss value corresponding to the embedding layer between the teacher model and the student model according to the first embedding loss, the second embedding loss, and the third embedding loss;

[0213] Obtain a first prediction loss corresponding to the prediction layer between the first deep learning model and the student model, a second prediction loss corresponding to the prediction layer between the second deep learning model and the student model, and a third prediction loss corresponding to the embedding layer between the third deep learning model and the student model, and determine an eighth loss value corresponding to the prediction layer between the teacher model and the student model according to the first prediction loss, the second prediction loss, and the third prediction loss;

[0214] A second distillation loss function is determined according to the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value, and the target deep learning model is obtained according to the second distillation loss function.

[0215] In some implementations, the processor 301 performs, in the process of determining the second distillation loss function according to the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value:

[0216] Determine first weight information corresponding to the fifth loss value, second weight information corresponding to the sixth loss value, third weight information corresponding to the seventh loss value, and fourth weight information corresponding to the eighth loss value;

[0217] The fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value are fused according to the first weight information, the second weight information, the third weight information, and the fourth weight information to obtain the second distillation loss function.

[0218] In some implementations, after obtaining the target deep learning model, the processor 301 further executes:

[0219] Performing singular value decomposition on a third weight matrix corresponding to the target deep learning model to obtain a fourth weight matrix;

[0220] The fourth weight matrix is ​​used to perform weight replacement on the target deep learning model to obtain the replaced target deep learning model.

[0221] It should be noted that those skilled in the art can clearly understand that, for the sake of convenience and brevity of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned model compression method embodiment, and will not be repeated here.

[0222] An embodiment of the present invention also provides a storage medium for computer-readable storage, wherein the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any model compression method provided in the description of the embodiment of the present invention.

[0223] The storage medium may be an internal storage unit of the terminal device in the aforementioned embodiment, such as a hard disk or memory of the terminal device. The storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal device.

[0224] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In a hardware embodiment, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0225] It should be understood that the term "and / or" used in the present specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0226] The serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A model compression method, characterized in that: The method comprises: A first deep learning model is obtained by performing model training using the training data, and a first importance score corresponding to a self-attention layer of the first deep learning model is obtained; a deep learning framework of the first deep learning model is obtained based on task selection, the task includes at least an image classification task, and the deep learning framework is used to define a model structure; Pruning the self-attention layer of the first deep learning model according to the first importance score to obtain a second deep learning model; Retraining the second deep learning model using the training data to obtain a third deep learning model, and obtaining a second importance score corresponding to the self-attention layer in the third deep learning model; quantifying the accuracy of the third deep learning model according to the second importance score to obtain a fourth deep learning model; Using the first deep learning model to perform distillation processing on the fourth deep learning model to obtain a target deep learning model; The step of quantifying the accuracy of the third deep learning model according to the second importance score to obtain a fourth deep learning model includes: Obtain a second weight matrix corresponding to each self-attention layer in the third deep learning model; Performing unstructured pruning on the third deep learning model according to the second weight matrix to obtain a fifth deep learning model; quantifying the accuracy of the fifth deep learning model using the second importance score to obtain the fourth deep learning model; The performing unstructured pruning on the third deep learning model according to the second weight matrix to obtain a fifth deep learning model includes: Determine the weight threshold corresponding to unstructured pruning; Calculating a weight magnitude corresponding to each weight in the second weight matrix; When the weight amplitude is lower than the weight threshold, removing the neurons corresponding to the weight amplitude to obtain the fifth deep learning model; The step of using the second importance score to quantify the accuracy of the fifth deep learning model to obtain the fourth deep learning model includes: Determine a segmentation parameter required for precision quantization, compare the second importance score with the segmentation parameter, and when the second importance score is greater than or equal to the segmentation parameter, set the information precision of the self-attention layer corresponding to the second importance score to the first precision information; When the second importance score is less than the segmentation parameter, setting the information accuracy of the self-attention layer corresponding to the second importance score as the second accuracy information; The fourth deep learning model is obtained according to the first precision information and the second precision information.

2. The method according to claim 1, characterized in that The obtaining a first importance score corresponding to the self-attention layer of the first deep learning model includes: Obtain a first weight matrix corresponding to each attention head of the self-attention layer of the first deep learning model, and calculate a weight amplitude corresponding to the first weight matrix; Obtaining a weight gradient of an output of the first deep learning model relative to each attention head after forward propagation of a feedforward neural network of a self-attention layer of the first deep learning model; Obtaining an activation value of each attention head of the self-attention layer of the first deep learning model; The first importance score is determined by performing data fusion according to the weight magnitude, the weight gradient, and the activation value.

3. The method according to claim 1, characterized in that The step of pruning the self-attention layer of the first deep learning model according to the first importance score to obtain a second deep learning model includes: Setting a pruning threshold, and comparing the pruning threshold with the first importance score; the pruning threshold is adjusted in real time according to the rate of change of model performance after pruning; When the first importance score is less than the pruning threshold, the self-attention layer corresponding to the first importance score is removed to obtain the second deep learning model.

4. The method according to claim 1, characterized in that The determining of the weight threshold corresponding to the unstructured pruning includes: Determine the target percentage required for data screening; The weight threshold is determined from the second weight matrix according to the target percentage.

5. The method according to claim 1, characterized in that The step of performing distillation processing on the fourth deep learning model using the first deep learning model to obtain a target deep learning model includes: Determining the first deep learning model as a teacher model, and determining the fourth deep learning model as a student model; Using the training data to train the teacher model and the student model respectively to obtain a first loss value corresponding to the self-attention layer between the teacher model and the student model, a second loss value corresponding to the hidden layer between the teacher model and the student model, a third loss value corresponding to the embedding layer between the teacher model and the student model, and a fourth loss value corresponding to the prediction layer between the teacher model and the student model; A first distillation loss function is determined according to the first loss value, the second loss value, the third loss value, and the fourth loss value, and the target deep learning model is obtained according to the first distillation loss function.

6. The method according to claim 5, characterized in that The obtaining the target deep learning model according to the first distillation loss function includes: Determine an output loss function between an output label corresponding to the student model and a sample hard label corresponding to the training sample; Fusion of the first distillation loss function and the output loss function to obtain a target loss function; The target deep learning model is obtained according to the target loss function.

7. The method according to claim 1, characterized in that Obtaining the target deep learning model, the method further includes: The first deep learning model, the second deep learning model, and the third deep learning model are all used as teacher models, and the fourth deep learning model is determined as a student model; Using the training data to train the first deep learning model and the student model respectively to obtain a first attention loss corresponding to the self-attention layer between the first deep learning model and the student model, a second attention loss corresponding to the self-attention layer between the second deep learning model and the student model, and a third attention loss corresponding to the self-attention layer between the third deep learning model and the student model, and determine a fifth loss value corresponding to the self-attention layer between the teacher model and the student model according to the first attention loss, the second attention loss, and the third attention loss; Obtain a first hidden loss corresponding to a hidden layer between the first deep learning model and the student model, a second hidden loss corresponding to a hidden layer between the second deep learning model and the student model, and a third hidden loss corresponding to a hidden layer between the third deep learning model and the student model, and determine a sixth loss value corresponding to a hidden layer between the teacher model and the student model according to the first hidden loss, the second hidden loss, and the third hidden loss; Obtain a first embedding loss corresponding to the embedding layer between the first deep learning model and the student model, a second embedding loss corresponding to the embedding layer between the second deep learning model and the student model, and a third embedding loss corresponding to the embedding layer between the third deep learning model and the student model, and determine a seventh loss value corresponding to the embedding layer between the teacher model and the student model according to the first embedding loss, the second embedding loss, and the third embedding loss; Obtain a first prediction loss corresponding to the prediction layer between the first deep learning model and the student model, a second prediction loss corresponding to the prediction layer between the second deep learning model and the student model, and a third prediction loss corresponding to the embedding layer between the third deep learning model and the student model, and determine an eighth loss value corresponding to the prediction layer between the teacher model and the student model according to the first prediction loss, the second prediction loss, and the third prediction loss; A second distillation loss function is determined according to the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value, and the target deep learning model is obtained according to the second distillation loss function.

8. The method according to claim 7, characterized in that The determining a second distillation loss function according to the fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value comprises: Determine first weight information corresponding to the fifth loss value, second weight information corresponding to the sixth loss value, third weight information corresponding to the seventh loss value, and fourth weight information corresponding to the eighth loss value; The fifth loss value, the sixth loss value, the seventh loss value, and the eighth loss value are fused according to the first weight information, the second weight information, the third weight information, and the fourth weight information to obtain the second distillation loss function.

9. The method according to claim 1, characterized in that: After obtaining the target deep learning model, the method further includes: Performing singular value decomposition on a third weight matrix corresponding to the target deep learning model to obtain a fourth weight matrix; The fourth weight matrix is ​​used to perform weight replacement on the target deep learning model to obtain the replaced target deep learning model.

10. A model compression device, characterized in that: include: A first training module is used to perform model training using the training data to obtain a first deep learning model, and obtain a first importance score corresponding to a self-attention layer of the first deep learning model; The deep learning framework of the first deep learning model is obtained based on task selection, the task at least includes an image classification task, and the deep learning framework is used to define a model structure; A pruning processing module, used for pruning the self-attention layer of the deep learning model according to the first importance score to obtain a second deep learning model; A second training module is used to retrain the second deep learning model using the training data to obtain a third deep learning model, and obtain a second importance score corresponding to the self-attention layer in the third deep learning model; A precision quantization module, configured to perform precision quantization on the third deep learning model according to the second importance score to obtain a fourth deep learning model; A distillation processing module, used to perform distillation processing on the fourth deep learning model using the first deep learning model to obtain a target deep learning model; The precision quantization module performs precision quantization on the third deep learning model according to the second importance score to obtain a fourth deep learning model, and is specifically used for: Obtain a second weight matrix corresponding to each self-attention layer in the third deep learning model; Performing unstructured pruning on the third deep learning model according to the second weight matrix to obtain a fifth deep learning model; quantifying the accuracy of the fifth deep learning model using the second importance score to obtain the fourth deep learning model; The precision quantization module performs unstructured pruning on the third deep learning model according to the second weight matrix to obtain a fifth deep learning model, which is specifically used for: Determine the weight threshold corresponding to unstructured pruning; Calculating a weight magnitude corresponding to each weight in the second weight matrix; When the weight amplitude is lower than the weight threshold, removing the neurons corresponding to the weight amplitude to obtain the fifth deep learning model; The precision quantification module uses the second importance score to perform precision quantification on the fifth deep learning model to obtain the fourth deep learning model, which is specifically used for: Determine a segmentation parameter required for precision quantization, compare the second importance score with the segmentation parameter, and when the second importance score is greater than or equal to the segmentation parameter, set the information precision of the self-attention layer corresponding to the second importance score to the first precision information; When the second importance score is less than the segmentation parameter, setting the information accuracy of the self-attention layer corresponding to the second importance score as the second accuracy information; The fourth deep learning model is obtained according to the first precision information and the second precision information.

11. A terminal device, characterized in that: The terminal device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the model compression method according to any one of claims 1 to 9 when executing the computer program.

12. A computer storage medium for computer storage, characterized in that: The computer storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the model compression method according to any one of claims 1 to 9.

Images