Model compression method and device

Abstract

The invention discloses a model compression method and device, and the method comprises the steps: obtaining a to-be-compressed model; pruning the to-be-compressed model based on a preset pruning strategy to obtain a pruning model; and performing knowledge distillation on the pruning model based on the to-be-compressed model to obtain a target compression model corresponding to the pruning model. After the to-be-compressed model is obtained, the to-be-compressed model is pruned based on the preset pruning strategy, and then the knowledge of the to-be-compressed model is transmitted to the pruning model through knowledge distillation, so that pruning and knowledge distillation can be organically combined, and the compressed model simultaneously has the flexibility of pruning and the relatively high compression performance of knowledge distillation; the model compression effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model compression method and device. Background Art

[0002] With the development of deep learning, model compression can effectively reduce the size of the model and facilitate industrial implementation. Existing technologies mainly rely on pruning, distillation and other methods to compress the model. Both methods have their own limitations. For the pruning compression method, it has the problem of insufficient compression performance. Specifically, while ensuring that the performance does not decrease or the decrease is acceptable, it can only provide a speed increase of 2-3 times at most. For the knowledge distillation compression method, although it has good compression performance, its implementation method is to first fix the student model size and then perform distillation training from scratch. This leads to the need for retraining when the model size needs to be adjusted for different scenarios, which brings about the problems of insufficient flexibility and excessive training overhead. That is, the existing technology cannot take into account the flexibility and compression performance of the compression model, resulting in poor model compression effect.

[0003] That is, the model compression effect in the prior art is relatively poor. Summary of the invention

[0004] The embodiments of the present application provide a model compression method and device, which can improve the effectiveness of model compression.

[0005] In a first aspect, the model compression method provided by the present application comprises:

[0006] Get the model to be compressed;

[0007] Pruning the model to be compressed based on a preset pruning strategy to obtain a pruned model;

[0008] Based on the model to be compressed, knowledge distillation is performed on the pruned model to obtain a target compression model corresponding to the pruned model.

[0009] In a second aspect, the model compression device provided by the present application comprises:

[0010] An acquisition module, used to acquire the model to be compressed;

[0011] A model pruning module, used to prune the model to be compressed based on a preset pruning strategy to obtain a pruned model;

[0012] The knowledge distillation module is used to perform knowledge distillation on the pruned model based on the model to be compressed to obtain a target compression model corresponding to the pruned model.

[0013] In a third aspect, the electronic device provided in the present application includes a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the model compression method provided in the present application.

[0014] In a fourth aspect, the computer-readable storage medium provided in the present application stores a plurality of instructions, which are suitable for loading by a processor to implement the steps in the model compression method provided in the present application.

[0015] In a fifth aspect, the computer program product provided in the present application includes a computer program or instructions, which, when executed by a processor, implement the steps in the model compression method provided in the present application.

[0016] In this application, compared with the related art, a model to be compressed is obtained; the model to be compressed is pruned based on a preset pruning strategy to obtain a pruned model; knowledge distillation is performed on the pruned model based on the model to be compressed to obtain a target compression model corresponding to the pruned model. After obtaining the model to be compressed, the present application first prunes the model to be compressed based on a preset pruning strategy, and then transfers the knowledge of the model to be compressed to the pruned model through knowledge distillation. It can organically combine pruning and knowledge distillation, so that the compressed model has both the flexibility of pruning and the high compression performance of knowledge distillation, which can improve the effect of model compression. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 is a schematic diagram of a scenario of a model compression system provided in an embodiment of the present application;

[0019] Figure 2 It is a flowchart of an embodiment of the model compression method provided in the embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of a cropping in an embodiment of a model compression method provided in an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of block clipping in an embodiment of a model compression method provided in an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of layer clipping in an embodiment of a model compression method provided in an embodiment of the present application;

[0023] Figure 6 This is a flow chart of an embodiment of the model compression method provided by the embodiment of the present application, in which a fine-tuning model is used as a teacher neural network, a pruned model is used as a student neural network, knowledge distillation is performed on the pruned model, and a target compression model corresponding to the pruned model is obtained;

[0024] Figure 7 It is a schematic diagram of a process for determining a target loss based on predicted distillation loss in one embodiment of a model compression method provided in an embodiment of the present application;

[0025] Figure 8 It is a partial schematic diagram of a fine-tuning model and a pruning model in one embodiment of a model compression method provided in an embodiment of the present application;

[0026] Fig. 9 is a flow chart of another embodiment of the model compression method provided in the embodiment of the present application;

[0027] Fig.10 is a schematic diagram of the structure of a model compression device provided in an embodiment of the present application;

[0028] Fig.11 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0030] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0031] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0032] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0033] In order to improve the effect of model compression, the embodiments of the present application provide a model compression method, a model compression device, an electronic device, a computer-readable storage medium, and a computer program product. The model compression method can be executed by the model compression device, or by an electronic device integrated with the model compression device.

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0035] Please refer to Figure 1 , the present application also provides a model compression system, such as Figure 1 As shown, the model compression system includes an electronic device 100, and the model compression device provided by the present application is integrated in the electronic device 100.

[0036] Among them, the electronic device 100 can be any device equipped with a processor and has processing capabilities, such as mobile electronic devices with processors such as smart phones, tablet computers, PDAs, laptops, smart speakers, or fixed electronic devices with processors such as desktop computers, televisions, servers, industrial equipment, etc.

[0037] In addition, if Figure 1 As shown, the model compression system may further include a memory 200 for storing original data, intermediate data, and result data.

[0038] In the embodiment of the present application, the memory 200 can be a cloud storage. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as the storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0039] At present, the storage method of the storage system is: create a logical volume, and when creating a logical volume, allocate physical storage space for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID, ID entity). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

[0040] The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of independent redundant disk arrays (RAID, Redundant Array of Independent Disks), the physical storage space is pre-divided into stripes. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0041] It should be noted that Figure 1 The scenario diagram of the model compression system shown is only an example. The model compression system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the model compression system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0042] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0043] Please refer to Figure 2 , Figure 2 is a flow chart of an embodiment of the model compression method provided in the embodiment of the present application, such as Figure 2 As shown, the process of the model compression method provided by this application is as follows:

[0044] 201. Obtain a model to be compressed.

[0045] In an embodiment of the present application, the model to be compressed is a pre-trained model, and the model to be compressed can be a Transformer model, a BERT (Bidirectional Encoder Representations from Transformers) model, or a BART (Bidirectional and Auto-Regressive Transformers) model. BERT is a pre-trained language representation model. It emphasizes that it is no longer pre-trained by using the traditional unidirectional language model or the shallow splicing method of two unidirectional language models as in the past, but a new MLM (masked language simulation, masked language model) is used to generate deep bidirectional language representations. When processing a certain word, it can use both the previous word and the following word information at the same time. The BART model uses a standard Transformer-based sequence-to-sequence structure. The main difference is that the activation function in the original structure is replaced by the GeLU (Gaussian Error Linerar Units) activation function, and the parameters are initialized according to the normal distribution. BART combines a bidirectional Transformer encoder with a unidirectional autoregressive Transformer decoder. It is a typical denoising autoencoder by pre-training the denoising and reconstruction of noisy input text.

[0046] 202. Prune the model to be compressed based on a preset pruning strategy to obtain a pruned model.

[0047] In the embodiment of the present application, the preset pruning strategy includes the importance parameters and pruning methods of each pruning unit in the model to be compressed.

[0048] Among them, the pruning unit can be an attention head, a set of weight parameters, a weight block or a network layer in the model to be compressed, etc., and each pruning unit can be set according to specific conditions.

[0049] The importance parameters of the pruned units can be set by manual experience or obtained through model training iteration. The importance parameters of the pruned units are initialized to preset values ​​or randomly initialized when the model is trained for the first time. For example, the importance parameters of the pruned units are all initialized to 1 when the model is trained for the first time, and its semantics is that all prunable units are fully retained.

[0050] The pruning method of the pruning unit can be head pruning, weight pruning, block pruning, layer pruning or a combination thereof. For example, if you want to maximize the accuracy, you can choose weight pruning and block pruning, which have less impact on performance. If you want to maximize model acceleration, you can sacrifice some model performance and choose layer pruning and head pruning, which have more obvious acceleration effects.

[0051] In pruning models of Transformer-like structures, two main types of pruning methods can be used: one-shot pruning and non-one-shot pruning. One-shot pruning: This method refers to performing a global pruning once during training, and using the pruned model for the subsequent inference phase. In this method, the entire model is compressed into a small sub-network, so the pruned model has a smaller size and computational overhead. However, since only one pruning is performed, the performance of the model may not be improved further. Non-one-shot pruning: This method performs multiple prunings during training, and retrains the model after each pruning. This allows the model to be optimized gradually while maintaining a small size and computational overhead. In this method, pruning is a dynamic process, so the model performance can be further improved as the number of prunings increases.

[0052] Head pruning is a pruning method that uses attention heads as pruning units. This method prunes some attention heads in the Transformer model. Each attention head has an independent set of parameters, so the number of parameters can be reduced by pruning the attention heads.

[0053] Weight pruning performs global pruning on the Transformer model based on the size of the weights. Similar to the convolutional model, connections with smaller weights are considered unimportant and can be pruned. Figure 3 As shown, Figure 3 This is a schematic diagram of row clipping in an embodiment of the model compression method provided in an embodiment of the present application. Row clipping of a linear layer weight matrix will also cause the same offset columns of the next linear layer to be clipped.

[0054] Block pruning divides the Transformer model into multiple blocks and then prunes the entire block. A block can be a multi-head attention or a feed-forward network. After pruning, the unpruned blocks will continue to be used, but the number of input and output channels also needs to be adjusted accordingly. Figure 4 As shown, Figure 4 Schematic diagram of block clipping in an embodiment of the model compression method provided in the embodiment of the present application. Block clipping of a linear layer weight matrix will also cause the same offset block of the next linear layer to be clipped.

[0055] Layer pruning is a pruning method that uses the weight of the entire layer as the pruning unit. It sets importance indicators for different layers of the Transformer model and then prunes the unimportant layers. This is a coarse-grained pruning method that can greatly accelerate the performance, but it also has a greater impact on performance. Figure 5 As shown, Figure 5 1 is a schematic diagram of a layer clipping in an embodiment of the model compression method provided in the embodiment of the present application, and all weight matrices constituting the layer are clipped.

[0056] It should be noted that artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0057] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in the fields of machine vision and natural speech processing.

[0058] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sound. Using deep learning technology and corresponding sample sets, network models that can achieve different functions can be trained.

[0059] 203. Perform knowledge distillation on the pruned model based on the model to be compressed to obtain a target compression model corresponding to the pruned model.

[0060] Knowledge distillation is also a model compression method. Model compression methods such as parameter clipping, precision conversion, and neural network structure search will destroy the structure of the original model, that is, they will damage the knowledge learned by the original model from the data. Knowledge distillation protects the knowledge learned in the original model through distillation, and then transfers this knowledge to the compressed model. Although the compressed model is much smaller than the original model, it can also learn the same knowledge.

[0061] Analogous to the human learning process, in knowledge distillation, the model to be compressed is called the teacher neural network (Teacher Model), and the model after compression is called the student neural network (Student Model). In general, the size of the teacher neural network is much larger than that of the student neural network. The general knowledge distillation process is to first use the data set to train the teacher neural network, so that the teacher neural network can fully learn the knowledge contained in the data; then when using the data set to train the student neural network, the knowledge that has been learned in the teacher neural network is extracted through the distillation method to guide the training of the student neural network, so that the student neural network is equivalent to obtaining prior information about the data set from the teacher neural network. That is, in knowledge distillation, the teacher neural network is pre-trained on the data set, and then the knowledge learned by itself is used to guide the student neural network during its training process, helping to improve the accuracy of the student neural network.

[0062] In a specific embodiment, the model to be compressed is used as a teacher neural network, the pruned model is used as a student neural network, and knowledge distillation is performed on the pruned model to obtain a target compression model corresponding to the pruned model.

[0063] In another specific embodiment, the model to be compressed is fine-tuned to obtain a fine-tuned model; the fine-tuned model is used as the teacher neural network, the pruned model is used as the student neural network, and the pruned model is subjected to knowledge distillation to obtain a target compression model corresponding to the pruned model. Specifically, the model to be compressed is fine-tuned based on preset training samples to obtain a fine-tuned model. The number of preset training samples can be multiple, and the type of preset training samples is determined according to the downstream tasks to be performed by the target compression model after compression. For example, if the downstream task is text prediction, the preset training samples are text samples, and if the downstream task is image classification, the preset training samples are image samples, which can be set according to the specific situation.

[0064] Model fine-tuning improves accuracy by using a pre-trained model obtained on big data to initialize the weights of your own model. This requires that the quality of the pre-trained model must be guaranteed. Fine-tuning is usually faster and more accurate. Of course, you can also use your own trained model as a pre-trained model, and then train it on your own dataset to make the model suitable for your own scenarios and tasks.

[0065] Since the model before pruning and the model after pruning have better structural similarity than other large models, this application uses the fine-tuned unpruned model as the teacher neural network and the pruned model after pruning as the student neural network to help find the best substructure during the pruning process, and to perform accuracy recovery to improve compression efficiency.

[0066] See also Figure 6 , Figure 6 This is a flow chart of an embodiment of the model compression method provided by the embodiment of the present application, in which the fine-tuning model is used as the teacher neural network, the pruned model is used as the student neural network, the knowledge distillation of the pruned model is performed, and the target compression model corresponding to the pruned model is obtained. In a specific embodiment, the fine-tuning model is used as the teacher neural network, the pruned model is used as the student neural network, the knowledge distillation of the pruned model is performed, and the target compression model corresponding to the pruned model is obtained, including:

[0067] 301. Input a preset training sample into a fine-tuning model to obtain a first prediction result.

[0068] The fine-tuning model includes a first network layer and a first prediction layer. The first prediction result is determined according to the type of the preset training sample. For example, if the downstream task is text prediction, the preset training sample is a text sample. The first prediction result is the first prediction probability distribution of the first prediction layer of the fine-tuning model predicting the preset training sample. The first prediction probability distribution indicates the probability of the preset training sample being each text category.

[0069] 302. Input a preset training sample into the pruning model to obtain a second prediction result.

[0070] The fine-tuning model includes a second network layer and a second prediction layer. The second prediction result is determined according to the type of the preset training sample. For example, if the downstream task is text prediction, the preset training sample is a text sample. The second prediction result is a second prediction probability distribution of the second prediction layer of the fine-tuning model predicting the preset training sample. The second prediction probability distribution indicates the probability of the preset training sample being each text category.

[0071] 303. Calculate a predicted distillation loss based on the first prediction result and the second prediction result.

[0072] In the embodiment of the present application, a preset loss function is used to calculate the prediction distillation loss based on the first prediction result and the second prediction result. The preset loss function is a cross entropy loss function, a mean square error loss function, an exponential loss function, or other loss functions.

[0073] 304. Determine a target loss based on the predicted distillation loss.

[0074] In a specific embodiment, the predicted distillation loss is determined as the target loss.

[0075] In another specific embodiment, determining the target loss based on the predicted distillation loss includes: calculating the sparse loss based on the first prediction result; and calculating the target total loss based on the sparse loss and the predicted distillation loss.

[0076] Specifically, the L0 norm and the supervision loss are calculated according to the first prediction result and the corresponding annotation label, and the sparse loss is determined based on the L0 norm and the supervision loss. The supervision loss can be calculated according to a preset loss function, which is a cross entropy loss function, a mean square error loss function, an exponential loss function, or the like, and the L0 norm refers to the number of non-zero elements in the vector.

[0077] Specifically, the sparse loss and the predicted distillation loss are weighted and summed based on two preset weight coefficients to obtain the target total loss.

[0078] 305. Iteratively train the pruning model based on the target loss until a preset stopping condition is met to obtain a target compression model corresponding to the pruning model.

[0079] The preset stop condition may be that the number of iterations reaches a preset number or the target loss is less than a preset loss value. The preset number and the preset loss value may be set according to specific circumstances.

[0080] In a specific embodiment, if the pruned model does not meet the preset stopping condition, the model parameters and the preset pruning strategy of the pruned model are updated, and the updated pruned model is iteratively trained until the preset stopping condition is met to obtain the target compression model corresponding to the pruned model.

[0081] Among them, the hyperparameters for training the pruning model include: learning rate of 3e-5, sparsity, optimizer selection as AdamW, batch size (batchSize), 3 preset weight coefficients, and number of training rounds (epoch).

[0082] For further information, see Figure 7 , Figure 7 : is a flow chart of determining a target loss based on predicted distillation loss in an embodiment of a model compression method provided by an embodiment of the present application. Determining the target loss based on predicted distillation loss includes:

[0083] 401. Obtain at least one first network parameter corresponding to at least one first network layer in a fine-tuning model.

[0084] Among them, the first network parameter can be a parameter of the first network layer itself, or a parameter output by the first network layer. For example, the first network layer includes an FFN (Feedforward Neural Network, Frame Flexible Network) module and an MHA (Multi-Head Attention) module. The function of the FFN module is mainly to provide nonlinear transformation and map the output results of the multi-head attention layer to a higher dimensional space. Not only can the model dynamically adjust the amount of calculation according to the number of input frames, but it can also significantly reduce the memory cost of storing multiple models. MHA is a multi-head attention model that extends the attention mechanism to multiple heads, thereby enhancing the model's attention to different features.

[0085] 402. Obtain at least one second network parameter corresponding to at least one second network layer in the pruning model.

[0086] The second network parameter may be a parameter of the second network layer itself or a parameter output by the second network layer. For example, the second network layer includes a FFN (Feedforward Neural Network, Frame Flexible Network) module and an MHA (Multi-Head Attention, Multi-Head Attention) module.

[0087] The number of the second network layers is less than the number of the first network layers. Figure 8 As shown, the fine-tuning model includes an Embeddings layer and 4 first network layers, the first network layer includes an FFN module and an MHA module; the pruned model includes an Embeddings layer and 2 second network layers, the second network layer includes an FFN module and an MHA module.

[0088] In the implementation of this application, the model is parsed according to the importance parameters of the pruned units, that is, the importance parameters of the pruned units are sorted, and the unimportant pruned units are ignored, that is, no knowledge transfer is performed on the unit during this distillation process. For example, the pruned model includes 10 second network layers, and all 10 second network layers are pruned units. The 10 second network layers are sorted according to their importance parameters, and the 8 second network layers with the lowest ranking are ignored, that is, no knowledge transfer is performed on the unit during this distillation process, and only the 2 second network layers with the highest ranking are performed.

[0089] 403. Calculate a migration index between each first network layer and each second network layer based on at least one first network parameter and at least one second network parameter.

[0090] The migration index may be a migration speed, a migration difficulty, and other indicators, which are used to quantify the difficulty of migrating each first network layer from the teacher neural network to each second network layer of the student neural network. The larger the migration index, the slower the knowledge migration speed and the greater the difficulty of knowledge migration.

[0091] The migration index is the difference. The greater the difference between the first network layer and the second network layer, the smaller the similarity between the first network layer and the second network layer. The difference can be EMD (Earth Mover's Distance) distance, Euclidean distance, cosine similarity, Pearson correlation coefficient, etc., which can be set according to specific circumstances.

[0092] In a specific embodiment, the EMD distance between the first network parameter of the first network layer and the second network parameter of the second network layer is calculated, and the EMD distance between the first network layer and the second network layer is determined as the migration index of the first network layer and the second network layer. In computer science and technology, the EMD distance is a measure of the distance between two probability distributions on a preset area, which is the well-known Wasserstein metric. If the two distributions are regarded as two different ways of stacking a certain number of mountains on a preset area, then the EMD is the sum of the minimum distances of the unit blocks required to move one pile into another pile.

[0093] Specifically, since the dimensions of the first network parameter of the first network layer are aligned with the dimensions of the second network parameter of the second network layer, the EMD distance between the first network parameter of the first network layer and the second network parameter of the second network layer is calculated. Since pruning will lead to fragmentation of the model, there is a dimensional difference between the student neural network and the teacher neural network, making it difficult to transfer knowledge. Therefore, the necessary linear layers are added to map the inter-layer outputs of the student neural network to the same vector space as the teacher neural network for knowledge transfer.

[0094] 404. Determine a network layer distillation loss based on a first network parameter of each first network layer, a second network parameter of each second network layer, and a migration index between each first network layer and each second network layer.

[0095] In a specific embodiment, determining the network layer distillation loss based on the first network parameters of each first network layer, the second network parameters of each second network layer, and the migration index between each first network layer and each second network layer includes:

[0096] (1) Calculating the single-layer inter-layer distillation loss between each first network layer and each second network layer based on the first network parameters of each first network layer and the second network parameters of each second network layer.

[0097] Specifically, the single-layer inter-layer distillation loss between each first network layer and each second network layer is calculated based on the first network parameters of each first network layer and the second network parameters of each second network layer based on the preset loss function. Among them, the preset loss function is a cross entropy loss function, a mean square error loss function, an exponential loss function, and other loss functions.

[0098] (2) Determine the migration weights between each first network layer and each second network layer based on the migration indicators between each first network layer and each second network layer.

[0099] The larger the migration index, the smaller the migration weight. The larger the EMD distance between the first network layer and the second network layer, the larger the migration index of the first network layer and the second network layer, and the greater the difference between the first network layer and the second network layer, the migration weight needs to be reduced.

[0100] (3) Based on the migration weights between each first network layer and each second network layer, the inter-layer distillation loss of each single layer is weighted summed to obtain the network layer distillation loss.

[0101] Among them, the smaller the migration index, the slower the knowledge migration speed and the greater the difficulty of knowledge migration, and the migration weight. When the migration index between the first network layer and the second network layer is small, it means that the knowledge migration speed is slower and the knowledge migration is more difficult. At this time, a smaller migration weight is used to reduce the single-layer inter-layer distillation loss between the first network layer and the second network layer, thereby reducing the amount of knowledge transfer between the first network layer and the second network layer.

[0102] 405. Determine the target loss based on the network layer distillation loss and the predicted distillation loss.

[0103] Specifically, the inter-layer distillation loss and the predicted distillation loss are weighted and summed based on two preset weight coefficients to obtain the target loss. The two preset weight coefficients can be set according to specific circumstances.

[0104] Since the core knowledge in different tasks is located in different layers, and the knowledge capacity that different layers of the student neural network can accept is different after pruning, this application quantifies the difficulty of migrating different layers of the teacher neural network to different layers of the student neural network by using the EMD distance. The more difficult ones are allocated less knowledge for migration. For example, the last layer of the teacher neural network is quite different from the first layer of the student neural network, which is specifically quantified as a larger EMD distance. The amount of knowledge allocated from the last layer of the teacher neural network to the first layer of the student neural network should be adjusted accordingly. By calculating the migration allocation ratio of different layers of the teacher neural network, the corresponding adaptive matching network layer distillation loss can be obtained.

[0105] See also Fig. 9 , Fig. 9: is a flow chart of another embodiment of the model compression method provided in the embodiment of the present application. In the embodiment of the present application, the model compression method may include:

[0106] 501. Obtain the model to be compressed.

[0107] In an embodiment of the present application, the model to be compressed is a pre-trained model, and the model to be compressed can be a Transformer model, a BERT (Bidirectional Encoder Representations from Transformers) model, or a BART (Bidirectional and Auto-Regressive Transformers) model, which can be set according to the specific situation.

[0108] 502. Fine-tune the model to be compressed to obtain a fine-tuned model.

[0109] Specifically, the model to be compressed is fine-tuned based on preset training samples to obtain a fine-tuned model.

[0110] 503. Prune the model to be compressed based on a preset pruning strategy to obtain a pruned model.

[0111] In the embodiment of the present application, the preset pruning strategy includes the importance parameters and pruning methods of each pruning unit in the model to be compressed.

[0112] Among them, the pruning unit can be an attention head, a set of weight parameters, a weight block or a network layer in the model to be compressed, etc., and each pruning unit can be set according to specific conditions.

[0113] Among them, the importance parameters of the pruned units are initialized to preset values ​​or randomly initialized when the model is trained for the first time. For example, the importance parameters of the pruned units are all initialized to 1 when the model is trained for the first time, and its semantics is that all prunable units are retained.

[0114] Specifically, the importance parameters of each pruned unit are binarized to obtain a pruning mask, and the model to be compressed is pruned based on the pruning mask to obtain a pruned model. Specifically, the importance parameters of each pruned unit are sorted from large to small, the importance parameters of the top-ranked pruning units are determined to be 1, and the importance parameters of the remaining pruning units are determined to be 0 to obtain a pruning mask.

[0115] The pruning method of the pruning unit can be head pruning, weight pruning, block pruning, layer pruning or a combination thereof. For example, if you want to maximize the accuracy, you can choose weight pruning and block pruning, which have less impact on performance. If you want to maximize model acceleration, you can sacrifice some model performance and choose layer pruning and head pruning, which have more obvious acceleration effects.

[0116] 504. Input the preset training sample into the fine-tuning model to obtain a first prediction result.

[0117] The fine-tuning model includes a plurality of first network layers and a first prediction layer. The first prediction result is determined according to the type of the preset training sample. For example, if the downstream task is text prediction, the preset training sample is a text sample. The first prediction result is the first prediction probability distribution of the first prediction layer of the fine-tuning model predicting the preset training sample. The first prediction probability distribution indicates the probability of the preset training sample being each text category.

[0118] 505. Input the preset training sample into the pruning model to obtain a second prediction result.

[0119] The fine-tuning model includes multiple second network layers and second prediction layers. The second prediction result is determined according to the type of the preset training sample. For example, if the downstream task is text prediction, the preset training sample is a text sample, and the second prediction result is the second prediction probability distribution of the second prediction layer of the fine-tuning model predicting the preset training sample. The second prediction probability distribution indicates the probability of the preset training sample being each text category.

[0120] 506. Calculate a predicted distillation loss based on the first prediction result and the second prediction result.

[0121] In the embodiment of the present application, a preset loss function is used to calculate the prediction distillation loss based on the first prediction result and the second prediction result. The preset loss function is a cross entropy loss function, a mean square error loss function, an exponential loss function, or other loss functions.

[0122] 507. Calculate the sparse loss based on the first prediction result.

[0123] 508. Obtain multiple first network parameters corresponding to multiple first network layers in the fine-tuning model.

[0124] like Figure 8 As shown, the fine-tuning model includes an Embeddings layer and four first network layers, and the first network layer includes an FFN module and an MHA module.

[0125] 509. Obtain multiple second network parameters corresponding to multiple second network layers in the pruning model.

[0126] like Figure 8 As shown, the pruned model includes an Embeddings layer and two second network layers, and the second network layer includes an FFN module and an MHA module.

[0127] 510. Calculate migration indicators between each first network layer and each second network layer based on the multiple first network parameters and the multiple second network parameters.

[0128] In a specific embodiment, the EMD distance between the first network parameter of the first network layer and the second network parameter of the second network layer is calculated, and the EMD distance between the first network layer and the second network layer is determined as the migration index of the first network layer and the second network layer.

[0129] 511. Determine a network layer distillation loss based on a first network parameter of each first network layer, a second network parameter of each second network layer, and a migration index between each first network layer and each second network layer.

[0130] In a specific embodiment, determining the network layer distillation loss based on the first network parameters of each first network layer, the second network parameters of each second network layer, and the migration index between each first network layer and each second network layer includes:

[0131] (1) Calculating the single-layer inter-layer distillation loss between each first network layer and each second network layer based on the first network parameters of each first network layer and the second network parameters of each second network layer.

[0132] Specifically, the single-layer inter-layer distillation loss between each first network layer and each second network layer is calculated based on the first network parameters of each first network layer and the second network parameters of each second network layer based on the preset loss function. Among them, the preset loss function is a cross entropy loss function, a mean square error loss function, an exponential loss function, and other loss functions.

[0133] (2) Determine the migration weights between each first network layer and each second network layer based on the migration indicators between each first network layer and each second network layer.

[0134] The larger the migration index, the smaller the migration weight. The larger the EMD distance between the first network layer and the second network layer, the larger the migration index of the first network layer and the second network layer, and the greater the difference between the first network layer and the second network layer, the migration weight needs to be reduced.

[0135] (3) Based on the migration weights between each first network layer and each second network layer, the inter-layer distillation loss of each single layer is weighted summed to obtain the network layer distillation loss.

[0136] Among them, the smaller the migration index, the slower the knowledge migration speed and the greater the difficulty of knowledge migration, and the migration weight. When the migration index between the first network layer and the second network layer is small, it means that the knowledge migration speed is slower and the knowledge migration is more difficult. At this time, a smaller migration weight is used to reduce the single-layer inter-layer distillation loss between the first network layer and the second network layer, thereby reducing the amount of knowledge transfer between the first network layer and the second network layer.

[0137] 512. Determine the target loss based on the sparse loss, the network layer distillation loss and the prediction distillation loss.

[0138] Specifically, the sparse loss, the network layer distillation loss, and the prediction distillation loss are weighted and summed based on the three preset weight coefficients to obtain the target loss. The three preset weight coefficients can be set according to the specific situation.

[0139] 513. Determine whether the target loss is lower than the preset loss value.

[0140] Among them, the preset loss value is set according to the specific situation.

[0141] 514. Determine the trained pruning model as the target compression model

[0142] If the target loss is lower than the preset loss value, the pruned model obtained by iterative training is determined as the target compression model to complete the model compression.

[0143] 515. Update the preset pruning strategy and model parameters of the pruning model.

[0144] In an embodiment of the present application, if the target loss is not lower than the preset loss value, it indicates that the model has not been pruned to the preset requirements at this time, and further pruning is required and the network parameters of the model itself are updated. Then, the preset pruning strategy and the model parameters of the pruning model are updated, and then 503-513 are repeatedly executed until the target loss is lower than the preset loss value. The pruned model obtained by iterative training is determined as the target compression model to complete the model compression.

[0145] In order to better implement the model compression method provided in the embodiment of the present application, the embodiment of the present application also provides a model compression device based on the above model compression method. The meanings of the terms are the same as those in the above model compression method. For specific implementation details, please refer to the description in the above method embodiment.

[0146] Please refer to Fig.10 , Fig.10 The schematic diagram of the structure of the model compression device provided in the embodiment of the present application is as follows. The model compression device may include an acquisition module 701, a model pruning module 702 and a knowledge distillation module 703, wherein:

[0147] An acquisition module 701 is used to acquire a model to be compressed;

[0148] A model pruning module 702 is used to prune the compression model based on a preset pruning strategy to obtain a pruned model;

[0149] The knowledge distillation module 703 is used to perform knowledge distillation on the pruned model based on the model to be compressed to obtain a target compression model corresponding to the pruned model.

[0150] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described in detail here.

[0151] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps in the model compression method provided in this embodiment by calling a computer program stored in the memory.

[0152] Please refer to Fig.11 , Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0153] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0154] The processor 101 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 101 may include one or more processing cores; optionally, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 101.

[0155] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0156] The electronic device also includes a power supply 103 for supplying power to each component. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 103 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0157] The electronic device may further include an input unit 104, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0158] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps in the model compression method provided in this application, such as:

[0159] Get the model to be compressed;

[0160] Pruning the model to be compressed based on a preset pruning strategy to obtain a pruned model;

[0161] The knowledge distillation of the pruned model is performed based on the model to be compressed to obtain the target compression model corresponding to the pruned model.

[0162] It should be noted that the electronic device provided in the embodiment of the present application and the model compression method in the above embodiment belong to the same concept, and its specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0163] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on a processor of an electronic device provided in an embodiment of the present application, the processor of the electronic device executes the steps in the model compression method provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0164] The present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes various optional implementations of the above-mentioned model compression method.

[0165] The above is a detailed introduction to a model compression method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0166] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. A model compression method, characterized in that: include: Get the model to be compressed; Pruning the model to be compressed based on a preset pruning strategy to obtain a pruned model; Based on the model to be compressed, knowledge distillation is performed on the pruned model to obtain a target compression model corresponding to the pruned model.

2. The model compression method according to claim 1, characterized in that: The performing knowledge distillation on the pruned model based on the model to be compressed to obtain a target compression model corresponding to the pruned model includes: Fine-tuning the model to be compressed to obtain a fine-tuned model; The fine-tuning model is used as the teacher neural network, the pruned model is used as the student neural network, knowledge distillation is performed on the pruned model, and a target compression model corresponding to the pruned model is obtained.

3. The model compression method according to claim 2, characterized in that: The method uses the fine-tuning model as a teacher neural network, uses the pruning model as a student neural network, performs knowledge distillation on the pruning model, and obtains a target compression model corresponding to the pruning model, including: Inputting a preset training sample into the fine-tuning model to obtain a first prediction result; Inputting the preset training sample into the pruning model to obtain a second prediction result; Calculate a predicted distillation loss based on the first prediction result and the first prediction result; determining a target loss based on the predicted distillation loss; The pruning model is iteratively trained based on the target loss until a preset stopping condition is met, so as to obtain a target compression model corresponding to the pruning model.

4. The model compression method according to claim 3, characterized in that: The determining the target loss based on the predicted distillation loss includes: Calculating a sparse loss based on the first prediction result; A target total loss is determined based on the sparse loss and the predicted distillation loss.

5. The model compression method according to claim 3, characterized in that: The determining the target loss based on the predicted distillation loss includes: Obtaining at least one first network parameter corresponding to at least one first network layer in the fine-tuning model; Obtaining at least one second network parameter corresponding to at least one second network layer in the pruning model; Calculate a migration index between each of the first network layers and each of the second network layers based on at least one of the first network parameters and at least one of the second network parameters; Determine a network layer distillation loss based on a first network parameter of each of the first network layers, a second network parameter of each of the second network layers, and a migration index between each of the first network layers and each of the second network layers; A target loss is determined based on the network layer distillation loss and the prediction distillation loss.

6. The model compression method according to claim 3, characterized in that: The iterative training of the pruning model based on the target loss until a preset stop condition is satisfied to obtain a target compression model corresponding to the pruning model includes: If the pruned model does not meet the preset stopping condition, the model parameters of the pruned model and the preset pruning strategy are updated, and the updated pruned model is iteratively trained until the preset stopping condition is met to obtain a target compression model corresponding to the pruned model.

7. The model compression method according to claim 1, characterized in that: The preset pruning strategy includes the importance parameters and pruning methods of each pruning unit in the model to be compressed.

8. A model compression device, characterized in that: include: An acquisition module, used to acquire the model to be compressed; A model pruning module, used to prune the model to be compressed based on a preset pruning strategy to obtain a pruned model; The knowledge distillation module is used to perform knowledge distillation on the pruned model based on the model to be compressed to obtain a target compression model corresponding to the pruned model.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the model compression method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the model compression method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the model compression method described in any one of claims 1 to 7 are implemented.

Images