Model processing method and device, computer equipment and storage medium

Abstract

The invention relates to a model processing method and device, computer equipment and a storage medium, and relates to the technical field of AI. The method comprises the steps of obtaining model training data of a target model; the target model comprises an embedded layer and a plurality of feature processing blocks connected behind the embedded layer, and each feature processing block comprises one or more attention layers and one or more feedforward layers; pruning at least one of an attention layer and a feed-forward layer in the plurality of feature processing blocks based on the model training data; and after pruning of at least one of the attention layer and the feed-forward layer in the plurality of feature processing blocks is completed, pruning processing is performed on the embedded layer.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of AI technology, and in particular to a model processing method, apparatus, computer equipment and storage medium. Background Art

[0002] With the continuous development of artificial intelligence technology, the structure of machine learning models is becoming more and more complex, and the parameter scale is becoming larger and larger, which brings great challenges to the application and deployment of machine learning models.

[0003] In the related art, the parameter scale of the machine learning model is usually reduced by means of network compression, wherein pruning is an important step in the network compression process. The pruning process in the related art can be as follows: for the weight parameters of the convolutional neural network model, the importance of each weight parameter is determined by a specific calculation method, and then multiple weight parameters with lower importance are pruned, leaving weight parameters with higher importance, thereby reducing the parameter scale of the model while ensuring the accuracy of the model.

[0004] However, the solution shown in the above-mentioned related technology is only applicable to pruning convolutional neural network models, and the pruning effect is poor for networks with more complex structures, thereby limiting the application scope of model compression. Summary of the invention

[0005] The embodiment of the present application provides a model processing method, device, computer equipment and storage medium, which can expand the application scope of model compression. The technical solution is as follows:

[0006] In one aspect, a model processing method is provided, the method comprising:

[0007] Inputting training samples into a teacher network and a student network respectively; the teacher network is a trained neural network, the hierarchical structure of the student network is the same as that of the teacher network, and the model parameter scale of the student network is smaller than the model parameter scale of the teacher network;

[0008] Acquire first network data of the teacher network, the first network data including a first model parameter of the i-th layer in the teacher network and a first feature map of the i-th layer in the teacher network; 1≤i≤N, where i is an integer and N is the number of network layers of the teacher network;

[0009] Acquire second network data of the student network, wherein the second network data includes a second model parameter of the i-th layer in the student network and a second feature graph of the i-th layer in the student network;

[0010] Cross-calculate the first network data and the second network data to obtain a loss function value;

[0011] The second model parameters of the i-th layer in the student network are updated according to the loss function value.

[0012] In another aspect, a model processing device is provided, the device comprising:

[0013] A data acquisition module, used to acquire model training data of a target model; the target model includes an embedding layer, and a plurality of feature processing blocks connected after the embedding layer, the feature processing blocks include one or more attention layers, and one or more feed-forward layers;

[0014] A first pruning module, configured to prune at least one of the attention layer and the feedforward layer in the plurality of feature processing blocks based on the model training data;

[0015] The second pruning module is used to prune the embedding layer after pruning at least one of the attention layer and the feedforward layer in the multiple feature processing blocks.

[0016] In a possible implementation, the first pruning module is used to:

[0017] Based on the model training data, pruning the attention layers in the plurality of feature processing blocks;

[0018] After the pruning process of the attention layers in the multiple feature processing blocks is completed, the pruning process of the feedforward layers in the multiple feature processing blocks is performed based on the model training data.

[0019] In a possible implementation, the first pruning module is configured to iteratively perform the following steps:

[0020] Based on the model training data, forward reverse gradient calculation is performed on the weight of the target model to obtain updated values ​​of each weight parameter in the weight matrix of the attention layer, wherein the weight matrix includes multiple channels, and each channel includes multiple weight parameters;

[0021] Arrange the channels of the weight matrix of the attention layer in descending order according to the absolute values ​​of the weight parameters;

[0022] For each channel of the weight matrix of the attention layer, a weight parameter of at least one channel arranged at the end is pruned.

[0023] In a possible implementation, the first pruning module is used to:

[0024] Obtain the sum of the absolute values ​​of the weight parameters contained in each channel of the weight matrix of the attention layer;

[0025] Arrange the channels of the weight matrix of the attention layer in descending order according to the sum of the absolute values ​​of the weight parameters they contain.

[0026] In a possible implementation, the first pruning module is configured to iteratively perform the following steps:

[0027] Based on the model training data, forward reverse gradient calculation is performed on the weight of the target model to obtain the gradient change of each weight parameter in the weight matrix of the feedforward layer, wherein the weight matrix includes multiple channels, and each channel includes multiple weight parameters;

[0028] Arrange the channels of the weight matrix of the feedforward layer in descending order according to the amount of change of the weight parameter on the gradient;

[0029] A pruning process is performed on the weight parameters of at least one channel arranged at the end in each channel of the weight matrix of the feed-forward layer.

[0030] In a possible implementation, the first pruning module is used to:

[0031] Obtaining the sum of the absolute values ​​of the gradient changes of the weight parameters contained in each channel of the weight matrix of the feedforward layer;

[0032] Arrange the channels of the weight matrix of the feedforward layer in descending order according to the sum of the absolute values ​​of the gradient changes of the weight parameters contained in each channel.

[0033] In one possible implementation, the change in gradient of each weight parameter in the weight matrix of the feedforward layer is determined by the trace of the Hessian matrix of the weight parameter and the change of the weight parameter before and after the forward reverse gradient calculation.

[0034] In a possible implementation, the second pruning module is used to:

[0035] Deleting embeddings corresponding to low-frequency words in the vocabulary of the embedding layer; the low-frequency words are at least one word in the vocabulary that is arranged in descending order of frequency of occurrence;

[0036] Performing dimensionality reduction processing on the embedding layer by principal component analysis;

[0037] The embedding layer after dimensionality reduction is decomposed by singular value decomposition (SVD) to obtain the embedding layer after pruned.

[0038] In a possible implementation manner, the device further includes:

[0039] A training module, configured to, after pruning the attention layer, the feedforward layer or the embedding layer, train and update the remaining model parameters in the target model based on the model training data until the target model converges;

[0040] The distillation module is used to restore the accuracy of the target model through distillation learning.

[0041] In a possible implementation manner, the device further includes:

[0042] The quantization module is used to quantize and store the parameters in the target model.

[0043] In a possible implementation manner, the quantizing and storing the parameters in the target model includes:

[0044] The parameters in the target model are quantized and stored in FP16 or int8 format.

[0045] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the model processing method as described above.

[0046] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction, at least one program, code set or instruction set is stored in the storage medium, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the model processing method as described above.

[0047] In another aspect, a computer program product is provided, the computer program product comprising a computer program, the computer program being stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device executes the model processing method provided in the above-mentioned various optional implementations.

[0048] The technical solution provided by this application may have the following beneficial effects:

[0049] For a target model including an embedding layer and multiple feature processing blocks connected after the embedding layer, the feature processing blocks include one or more attention layers and one or more feed-forward layers, at least one of the attention layer and the feed-forward layer is pruned first, and then the embedding layer is pruned. The pruning processes of different network layers can be executed separately to ensure the accuracy of model pruning. At the same time, the pruning order of first the attention layer and the feed-forward layer, and then the embedding layer is adopted to ensure the accuracy and efficiency of pruning.

[0050] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] Figure 1 It is a schematic diagram of a transformer network structure involved in this application;

[0053] Figure 2 is a system configuration diagram of a model processing system involved in various embodiments of the present application;

[0054] Figure 3 It is a framework diagram of the network compression process involved in the embodiment of the present application;

[0055] Figure 4 is a method flow chart of a model processing method according to an exemplary embodiment;

[0056] Figure 5 is a schematic diagram of an evaluation indicator involved in this application;

[0057] Figure 6 is a method flow chart of a model processing method according to an exemplary embodiment;

[0058] Figure 7 is a schematic diagram of the layered distillation scheme;

[0059] Figure 8 is a method flow chart of a model processing method according to an exemplary embodiment;

[0060] Fig. 9 is a schematic diagram of PCA involved in this application;

[0061] Fig.10 is a schematic diagram of the SVD decomposition involved in this application;

[0062] Fig.11 is a schematic diagram of the adaptive distillation involved in this embodiment;

[0063] Fig.12 is a method flow chart of a model processing method according to an exemplary embodiment;

[0064] Fig.13 It is a flow chart of the transformer network structure on the application side involved in this application;

[0065] Fig.14 is a block diagram of a model processing device provided by an exemplary embodiment of the present application;

[0066] Fig.15 The diagram is a schematic structural diagram of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION

[0067] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0068] This application proposes a model processing solution that can prune a machine learning model with a complex structure, thereby expanding the application scope of network model compression. For ease of understanding, some concepts involved in this application are explained below.

[0069] (1) Artificial Intelligence (AI)

[0070] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0071] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The solution provided in the embodiments of this application mainly involves technologies such as machine learning / deep learning in artificial intelligence.

[0072] (2) Machine Learning (ML)

[0073] Machine learning is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0074] (3) Neural Network

[0075] Neural network, also known as artificial neural network (ANNs) or connection model, is an algorithmic mathematical model that imitates the behavioral characteristics of neural networks of animals such as humans and performs distributed parallel information processing. This network relies on the complexity of the system and adjusts the interconnected relationships between a large number of internal nodes to achieve the purpose of processing information.

[0076] (4) Modality

[0077] Modality is the organization form or source of information. Common modalities include vision, hearing, text, etc. Processing multiple modal information at the same time is called multimodal content understanding.

[0078] (5) Pre-trained Model

[0079] The pre-trained model is a model that is independent of specific tasks and is trained from large-scale data through self-supervised learning. Since rich information is learned during the pre-training process, the pre-trained model can achieve good results in downstream tasks after simple fine-tuning.

[0080] (6) Model compression

[0081] Model compression refers to using data sets to streamline already trained deep models to obtain a lightweight network with comparable accuracy. The compressed network has a smaller structure and fewer parameters, which can effectively reduce computing and storage overhead and facilitate deployment in restricted hardware environments.

[0082] (7) Pruning

[0083] The principle of model pruning is to reduce the number of parameters and calculations by removing "unimportant" weights in the model, while trying to ensure that the accuracy of the model is not affected.

[0084] In related technologies, pruning includes pruning of weights and activations. It can be divided into unstructured / structured pruning according to the pruning granularity, and can be roughly divided into metric-based / reconstruction error-based / sparse training-based pruning according to the implementation method. There is a gradual trend towards AutoML.

[0085] 1) Metric-based pruning

[0086] This type of method usually proposes a metric to determine whether a neuron is important, calculates the value of measuring the importance of the neuron based on this metric, and prunes the unimportant neurons. The values ​​that can be used for measurement in neural networks are mainly divided into three parts: weight, activation, and gradient. The neuron here can be a single weight in unstructured pruning or an entire filter in structured pruning.

[0087] a. Weight-based pruning

[0088] The classic method in structured pruning is norm pruning, which judges the importance of filters based on L1-norm. Related technologies reduce absolute importance to a relative level, and consider filters that are too similar to other filters to be unimportant.

[0089] b. Activation-based pruning

[0090] The related technology uses the average percentage of zeros (APoZ) in the activation value as a metric, while the related technology uses information entropy for pruning.

[0091] c. Gradient-based pruning

[0092] This type of method usually starts from the loss to find the neuron with the least impact on the loss. In recent years, a more representative solution is to perform Taylor expansion on the activation value at point 0, convert the activation value into the expansion of the weight and then add the square. Similar methods include using Fisher information to approximate the Hessian matrix and directly using derivatives to perform unstructured pruning on randomly initialized weights.

[0093] 2) Pruning based on reconstruction error

[0094] This type of method determines which filters to prune by minimizing the reconstruction error of the feature output, that is, finding the information in the current layer that has less impact on the output of the subsequent network layer, for example, using the greedy method or the Lassoregression algorithm. Related technologies determine which filters need to be pruned by minimizing the reconstruction error of the penultimate layer of the network and taking into account the accumulated error of back propagation.

[0095] 3) Pruning based on sparse training

[0096] This type of method uses training and various regularization terms to make the network weights sparse, so that values ​​close to 0 can be pruned. For example, group Lasso can be used for structured sparseness, including filters, channels, filter shapes, and depth. Some solutions also use the APG algorithm to sparse the mask by introducing a learnable mask to achieve structured pruning. Some solutions can also be solved using the classic algorithm ADMM in constrained optimization. Since the output of each channel will pass through BN (Batch Normalization), the scaling factor of BN can be cleverly sparse directly, such as using L1 regularizer or ISTA (Iterative Shrinkage-Thresholding Algorithm) for sparseness. There are also some solutions that can directly use L1regularizer, combined with the width-multiplier in MobileNet, and add shink-expand operations to achieve pruning, which can better meet resource constraints.

[0097] (8) Distillation

[0098] Distillation refers to the process of extracting knowledge from the original model through certain technical means to learn a small model.

[0099] (9) Transformer

[0100] Transformer is a feature extractor, which was originally used in NLP (Natural Language Processing) tasks. Recently, it has been gradually popularized in CV tasks and has gradually become the standard for various tasks. Its structure includes an encoder and a decoder. Please refer to Figure 1, which shows a schematic diagram of a transformer network structure involved in this application, such as Figure 1 As shown on the left. The commonly used encoder module includes a multi-head attention module (such as Figure 1 right) and a feed-forward network layer with a residual link.

[0101] Taking BERT base as an example, the main parameters of the network structure include: embedding layer, encoder layer and optional classification application layer. For example, the embedding layer and encoder layer parameter structure can be as follows:

[0102]

[0103]

[0104] As shown above, among the total 95M parameters, the embedding layer has 13M parameters, accounting for 13.68%, the vocabulary dimension is 18000*768, the single encoder layer has 6.76M parameters, accounting for 7.11%, and the 12-layer encoder has 81M parameters, accounting for 85.26%. In a single encoder layer, the multi-head attention module contains 2.25M parameters, the feedforward layer contains 2.25M parameters, and the output layer contains 2.25M parameters, each accounting for one third.

[0105] Figure 2 1 is a system structure diagram of a model processing system involved in each embodiment of the present application. Figure 1 As shown, the system includes a network compression requester device 220 and a network compression servicer device 240 .

[0106] The network compression requester device 220 may be a personal computer device such as a desktop computer, a laptop computer, a personal workstation, etc.

[0107] The network compression service provider device 240 may be a server, or may be composed of several servers, or may be a virtualization platform, or may be a cloud computing service center.

[0108] The network compression requester device 220 and the network compression servicer device 240 may be connected via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0109] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to any combination of local area network (LAN), metropolitan area network (MAN), wide area network (WAN), mobile, wired or wireless network, private network or virtual private network. In some embodiments, the data exchanged through the network is represented by technology and / or format including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technology can also be used to replace or supplement the above-mentioned data communication technology.

[0110] exist Figure 2 In the system shown, the above-mentioned network compression requester device 220 can provide the network model to be compressed (target model) to the network compression service provider device 240, and optionally, can also provide training data. The network compression service provider device 240 prunes and distills the network model through the training data to achieve network compression, obtains the compressed network model, and returns the compressed network model to the network compression requester device 220.

[0111] For example, see Figure 3 , which shows a network compression process framework diagram involved in the embodiment of the present application. Figure 3As shown, in the network compression requester device 220, the developer of the network compression requester trains the target model through a large amount of training data to obtain a trained target model, and then sends the trained target model and the training data to the network compression service device 240 through the interface provided by the network compression service device 240. In the network compression service device 240, the trained target model is pruned according to the training data to obtain a lightweight, compressed target model. Afterwards, the network compression service device 240 returns the compressed lightweight target model to the network compression requester device 220, and the network compression requester corresponding to the network compression requester device 220 applies the target model in a scenario with low computing performance.

[0112] From a general perspective, consider that the user has trained the neural network they need, but because the model size and computational complexity are too large, the model needs to be pruned and compressed. Since some users do not know how to prune the model, they may resort to this part of the work to a third party that provides model compression services. As a third party, to prune the model, the user is usually required to provide both the model and the training data to ensure the accuracy of the pruned model. Due to data security and privacy issues, users are usually only willing to provide trained models and a small amount of training data.

[0113] The compressed target model in the solution shown in this application can be applied in the following scenarios, including but not limited to:

[0114] 1. Vehicle system.

[0115] For in-vehicle systems, lightweight machine learning models are usually the first choice for such systems because the computing and storage capabilities of the platform are limited and they usually require the integration of multiple intelligent technologies, such as voice recognition and control, and geolocation.

[0116] 2. Offline intelligent system on mobile phone.

[0117] At present, the intelligent systems on mobile phones usually upload and feedback through Internet transmission, which may cause privacy leakage, limited mobile phone signals, etc. A common solution is to deploy the intelligent systems on mobile phones offline. Due to the limited computing and storage capabilities of mobile phones, lightweight models are often required.

[0118] 3. Field Programmable Gate Array (FPGA) on-chip intelligent system.

[0119] As another important platform for edge computing, FPGA chips have been widely used in embedded devices, such as drones, smart homes, and wearable devices. FPGA chips often have the characteristics of low power consumption and low latency, so it is difficult to achieve ideal characteristics by directly deploying large-scale networks. Therefore, the demand for deploying lightweight network models on the FPGA side is also increasing.

[0120] 4. Network edge device inference scenario.

[0121] In various downstream businesses involving edge device reasoning, such as mini-program e-commerce product classification, recall, and sorting, and public account video content classification and retrieval, it is usually necessary to complete the model reasoning process in the shortest possible time. However, the reasoning time of large-scale network models is usually long and cannot meet the low-latency business feedback requirements.

[0122] With the continuous development of AI technology, in order to achieve more powerful functions, the parameter scale of network models is becoming larger and larger, and the model structure is becoming more and more complex. The difficulty of model deployment and application is also increasing. Therefore, the industry's demand for compression solutions for models with complex structures is also increasing.

[0123] By utilizing the method proposed in the subsequent embodiments of the present application, a complex structure network model having different types of network layers such as an embedding layer, an attention layer, and a feedforward layer can be compressed while ensuring the accuracy of the compressed model.

[0124] Figure 4 is a method flow chart of a model processing method according to an exemplary embodiment. The model processing method can be executed by a computer device, for example, the computer device can be the above-mentioned Figure 2 The network compression service device 240 shown, or the computer device may also be other computer devices, such as a server, or a personal computer, etc. The model processing method may include the following steps:

[0125] Step 410: Obtain model training data of the target model; the target model includes an embedding layer, and a plurality of feature processing blocks connected after the embedding layer, the feature processing block includes one or more attention layers, and one or more feed-forward layers.

[0126] The target model may be a BERT (Bidirectional Encoder Representations from Transformers)-based model, also known as a BERT base model. For example, the target model may include an embedding layer, and a plurality of feature processing blocks that are sequentially connected after the embedding layer. Alternatively, the plurality of feature processing blocks in the target model may be connected in other ways, for example, the plurality of feature processing blocks may be connected in a cross manner.

[0127] Alternatively, the target model may be a model other than the BERT base model, which meets the structural requirement of an embedding layer followed by multiple feature processing blocks.

[0128] Optionally, the target model may further include an embedding layer and other network layers other than the multiple feature processing blocks connected after the embedding layer. For example, the target model may also include a task module corresponding to the task objective. For example, the target model may also include a decoding branch, which may be connected behind multiple feature processing blocks. For example, the decoding branch may be connected behind the last feature processing block among the multiple feature processing blocks, which may be processed according to the features output by the feature processing block to output the prediction result of the target model.

[0129] The target model can be a machine learning model that has been pre-trained to convergence. For example, taking the BERT base model as an example, the initial target model can be obtained using the pre-training-fine-tuning paradigm. Taking the text classification model as an example, the BERT base model pre-trained on the open source Chinese corpus is fine-tuned on the downstream task dataset. The evaluation indicator of the classification task uses the F1 value, and the F1 value is calculated as follows. The fine-tuned model with the best evaluation indicator is taken as the compressed base model.

[0130] F1=2*precision*recall / (precision+recall)=2*TP / (2*TP+FP+FN) (1)

[0131] Please refer to Figure 5 , which shows a schematic diagram of an evaluation index involved in this application, such as Figure 5 As shown, it shows the definition and calculation method of TP / FP / FN and other values ​​in the above formula (1).

[0132] Among them, the above-mentioned model training data can be data corresponding to the task objectives of the target model; for example, in the field of network applications, the above-mentioned model training data can be text samples, such as text samples corresponding to various network contents. Optionally, the model training data can also include annotation information corresponding to the text sample, and the annotation information can be obtained by manual pre-annotation. For example, the annotation information can be obtained by manual annotation, and the text sample is the target result after being processed by the target model; for example, in the network content push scenario, the annotation information can indicate whether the network content is recommended to a certain user object.

[0133] For another example, in an image-based semantic parsing scenario, the model training data may be an image sample. Optionally, the model training data may also include annotation information corresponding to the image sample. The annotation information may be manually pre-annotated. For example, the annotation information may be the semantic text of the image sample obtained by manual annotation.

[0134] For another example, in an audio-based semantic analysis / speech recognition scenario, the model training data may be an audio sample. Optionally, the model training data may also include annotation information corresponding to the audio sample, which may be manually pre-annotated. For example, the annotation information may be manually annotated semantic text or speech text of the audio sample.

[0135] Step 420: Prune at least one of the attention layers and feed-forward layers in the multiple feature processing blocks based on the model training data.

[0136] Among them, the above-mentioned pruning of at least one of the attention layers and feedforward layers in multiple feature processing blocks based on model training data may refer to processing the above-mentioned model training data through the target model, performing reverse gradient calculation and updating on the weight parameters of the target network according to the processing results of the model training data and pre-set annotation information, determining the importance of each weight parameter in at least one network layer among the attention layer and the feedforward layer according to the results of the reverse gradient calculation and updating, and pruning the weight parameters with low importance.

[0137] For example, when the above-mentioned model training data is a text sample, the computer device can process the text sample through the target model and output the prediction result of the text sample. For example, for the text sample corresponding to the network content, the matching result between the text sample and the target user is output, and the loss function value is calculated based on the matching result and the annotation information. The weight parameters in the target model are reversely gradient updated through the loss function value. In this process, the importance of each weight parameter of the attention layer or the feedforward layer can be determined. According to the importance of each weight parameter of the attention layer or the feedforward layer, the weight parameters at the end of the importance in the attention layer or the feedforward layer are pruned. For example, the weight parameters at the end of the importance in the attention layer or the feedforward layer are set to 0 and fixed (that is, they are no longer updated in subsequent training processes).

[0138] For another example, when the above-mentioned model is training an image sample of data, the computer device can process the image sample through the target model, output the prediction result of the image sample, for example, output the semantic text of the image sample, and calculate the loss function value based on the matching result and the annotation information. The weight parameters in the target model are reversely updated with the loss function value. In this process, the importance of each weight parameter of the attention layer or the feedforward layer can be determined. According to the importance of each weight parameter of the attention layer or the feedforward layer, the weight parameters at the end of the importance in the attention layer or the feedforward layer are pruned.

[0139] For another example, when the above-mentioned model trains the audio sample of the data, the computer device can process the audio sample through the target model, and output the prediction result of the audio sample, for example, output the semantic text / speech text of the audio sample, and calculate the loss function value based on the matching result and the annotation information, and perform reverse gradient update on the weight parameters in the target model through the loss function value. In this process, the importance of each weight parameter of the attention layer or the feedforward layer can be determined, and according to the importance of each weight parameter of the attention layer or the feedforward layer, the weight parameters at the end of the importance in the attention layer or the feedforward layer can be pruned.

[0140] Step 430: After pruning at least one of the attention layer and the feed-forward layer in the multiple feature processing blocks, pruning the embedding layer is performed.

[0141] The pruning of the embedding layer may include pruning the embedding of some words in the embedding layer, or may include reducing the weight dimension of each word in the embedding layer.

[0142] In an embodiment of the present application, for a model structure in which an embedding layer is followed by multiple feature processing blocks, since the model simultaneously has multiple network layers (including at least an embedding layer, an attention layer, and a feedforward layer), different types of network layers may be suitable for different pruning algorithms. Therefore, it is necessary to execute the pruning processes of different network layers separately to ensure the accuracy of model pruning. Furthermore, since pruning the embedding layer will affect all weight parameters of the subsequent multiple feature processing blocks, the embodiment of the present application chooses to first prune at least one of the attention layer and the feedforward layer in the multiple feature processing blocks, and then prune the embedding layer, so as to ensure the accuracy and efficiency of pruning.

[0143] In a possible implementation, the method further includes:

[0144] After pruning the attention layer, feedforward layer, or embedding layer, the remaining model parameters in the target model are trained and updated based on the model training data until the target model converges;

[0145] The accuracy of the target model is restored by distillation learning. In the scheme shown in the embodiment of the present application, after pruning each network layer, in order to ensure the accuracy of the subsequent pruning process and the accuracy of the model after all pruning is completed, the computer device can train the remaining model parameters in the target model after pruning each network layer to make the model stable, and then perform distillation learning. The core idea of ​​distillation learning is to transfer the knowledge of the teacher model to the student model. The teacher model is usually trained with large-scale data sets, so it is more comprehensive and accurate in learning the feature representation of each category. Through distillation learning, these capabilities can be passed to the student model, that is, the student model still retains the experience and generalization ability of the teacher model on the basis of reducing the parameters and calculation amount of the model, so the compressed model can achieve the purpose of accuracy recovery by performing distillation learning.

[0146] This solution uses distillation to restore the accuracy of the compressed model, which can also be skipped directly or replaced by fine-tuning.

[0147] To summarize, the scheme shown in the embodiment of the present application, for a target model including an embedding layer, and multiple feature processing blocks connected after the embedding layer, the feature processing blocks include one or more attention layers, and one or more feed-forward layers, firstly, at least one of the attention layer and the feed-forward layer is pruned, and then the embedding layer is pruned. The pruning process of different network layers can be executed separately to ensure the accuracy of model pruning. At the same time, the pruning order of first the attention layer and the feed-forward layer, and then the embedding layer is adopted to ensure the accuracy and efficiency of pruning.

[0148] In addition, by executing the pruning processes of the attention layer, feedforward layer, and embedding layer separately, different pruning schemes can be used for different network layers in the model, thereby increasing the flexibility of pruning complex models and further improving the accuracy of model pruning.

[0149] based on Figure 4 The embodiment shown, referring to Figure 6 , which is a method flow chart of a model processing method according to an exemplary embodiment. Figure 6 As shown above Figure 4 Step 420 in the illustrated embodiment may be implemented as step 420a and step 420b.

[0150] Step 420a: Prune the attention layers in multiple feature processing blocks based on the model training data.

[0151] In a possible implementation, the above pruning of the attention layers in the multiple feature processing blocks based on the model training data includes:

[0152] Iterate through the following steps:

[0153] Based on the model training data, the forward and reverse gradient calculations are performed on the weights of the target model to obtain the updated values ​​of each weight parameter in the weight matrix of the attention layer. The weight matrix contains multiple channels, and each channel contains multiple weight parameters.

[0154] First, the input data is passed through the model to the attention layer. In the attention layer, the attention score is calculated. Using the attention score, the Softmax function can be used to convert the attention score into the attention weight parameter, calculate the weight matrix, and then use the loss function to calculate the loss, such as the cross entropy loss. Finally, the gradient is propagated back to the weight matrix of the attention layer to obtain the updated value of the attention weight parameter.

[0155] Arrange the channels of the attention layer's weight matrix in descending order according to the absolute value of the weight parameter.

[0156] For each channel of the weight matrix of the attention layer, the weight parameter of at least one channel arranged at the end is pruned.

[0157] The above-mentioned pruning of the weight parameters of at least one channel arranged later can be performed according to the pruning ratio, such as retaining 90% of the original number of channels, or a threshold can be preset, and the sum of the absolute values ​​of the weight parameters of each channel is compared with the threshold, and the weight parameters in the channels less than the threshold are set to zero, and these channels are no longer updated. During the training process, the computer device can iteratively perform the above steps until the ratio of the number of channels of the remaining weight parameters in the attention layer to the number of channels of the original weight parameters is not higher than the preset sparsity (such as 50%), that is, 50% of the weight parameters are pruned.

[0158] In an embodiment of the present application, unimportant channels that contribute little to the output of the attention layer are pruned, that is, the weight parameters in these channels are set to zero and are no longer stored or calculated, which means that the attention layer has less data to process, reducing the computational burden of the attention layer.

[0159] In a possible implementation, the channels of the weight matrix of the attention layer are arranged in descending order according to the absolute values ​​of the weight parameters, including:

[0160] Get the sum of the absolute values ​​of the weight parameters contained in each channel of the weight matrix of the attention layer;

[0161] Arrange the channels of the attention layer's weight matrix in descending order according to the sum of the absolute values ​​of the weight parameters they contain.

[0162] In the embodiment of the present application, because the weight parameter can be positive or negative, the positive and negative represent the direction of the weight parameter, and the absolute value of the number represents the strength of the weight parameter, that is, the contribution of the weight parameter to the model. For the situation where the contribution of negative and positive values ​​to the model is similar, the absolute value of the weight parameter and the strength of the weight parameter are used as the basis for pruning. The weight parameter with a small absolute value, that is, the weight parameter with a small contribution to the model is discarded. The absolute value is used as the basis to improve the accuracy of pruning. On the other hand, the sum of the absolute values ​​of the weight parameters corresponding to each channel is sorted in order from large to small. Subsequently, pruning is performed according to the channel. There is no need to traverse and judge each weight parameter, which saves the time of model pruning operations and greatly improves the efficiency of pruning.

[0163] In addition, in addition to expressing the importance of the channel by the sum of the absolute values ​​of the weight parameters contained in each channel, the computer device can also express the importance of the channel through other data, for example, expressing the importance of the channel by the average value of the absolute values ​​of the weight parameters contained in each channel, and so on.

[0164] In the scheme shown in the embodiment of the present application, since the attention layer is implemented by calculating the attention mechanism of the input vector through three linear layers, the size of the attention value affects the output result of this layer, so the absolute magnitude can be used to measure the importance of the parameter.

[0165] During the training process, when the weight parameter > 0 and the gradient < 0, or the parameter < 0 and the gradient > 0, it indicates that the parameter tends to be 0. When the parameter is 0, it indicates that the parameter is less important. According to the normalized parameter importance index, the top k parameters of unimportant parameters can be pruned. Assume that the required sparsity is atten_sparsity (the default is 0.5, and the calculation method is the number of model parameters after pruning / the number of model parameters before pruning). The detailed steps are as follows:

[0166] (1) Perform forward and reverse gradient calculations on the current epoch data set;

[0167] (2) Calculate the updated values ​​of each weight in the attention layer based on the gradient;

[0168] (3) Aggregate and sort the weight values ​​according to the channel dimension;

[0169] (4) The k channels with the lowest weight values ​​after sorting are masked and not included in the subsequent gradient calculation;

[0170] (5) Repeat steps 1-4 until the parameter quantity meets the sparsity requirement;

[0171] (6) Continue training remaining_step batches of data to stabilize the model effect.

[0172] The pruned model is distilled layer by layer to quickly restore accuracy. The schematic diagram of the layer by layer distillation scheme can be shown as follows: Figure 7 As shown, in Figure 7 In the layer-by-layer distillation process, the same model training data (such as input text) is input into the teacher model on the left and the pruned target model on the right. For the output parameters of each network layer 701 in the target model, the output parameters of the network layer 702 corresponding to the network layer 701 in the teacher model are obtained respectively. The weight parameters of the pruned target model are updated by the difference between the output parameters of the network layer 701 and the output parameters of the network layer 702. Optionally, during the distillation process, the weight parameters of the pruned target model can also be updated by the difference between the prediction result 703 of the target model and the prediction result 704 of the teacher model. The loss function between each layer of the distillation adopts the KL divergence, which can be used to measure the degree of difference between two distributions. The smaller the difference between the two, the smaller the KL divergence, and vice versa. When the two distributions are consistent, their KL divergence is 0. The calculation method is shown in formula (2):

[0173]

[0174] Among them, p(x)∈P represents the output embedding of the student model in the distilled layer, q(x)∈Q represents the output embedding of the teacher model in the layer to be distilled, and the loss function aims to minimize the difference between the teacher model and the student model at each layer.

[0175] For the compression of the attention layer, this scheme uses absolute magnitude as the evaluation indicator to measure the importance of parameters. It can also directly use L1 / L2 norm for pruning, that is, replace the absolute value of the above weight parameter with L1 / L2 norm.

[0176] Step 420b: After the pruning process of the attention layers in the multiple feature processing blocks is completed, the feedforward layers in the multiple feature processing blocks are pruned based on the model training data.

[0177] In a possible implementation, the pruning process of the feedforward layers in the plurality of feature processing blocks based on the model training data includes:

[0178] Iterate through the following steps:

[0179] Based on the model training data, forward reverse gradient calculation is performed on the weight of the target model to obtain the change amount of each weight parameter in the weight matrix of the feedforward layer on the gradient, the weight matrix includes multiple channels, and each channel includes multiple weight parameters; wherein the change amount of each weight parameter in the weight matrix of the feedforward layer on the gradient may refer to the amplitude of the gradient change of each weight parameter in the weight matrix of the feedforward layer during the reverse gradient calculation process;

[0180] Arrange the channels of the weight matrix of the feedforward layer in descending order according to the amount of change in the gradient of the weight parameter;

[0181] For each channel of the weight matrix of the feedforward layer, a weight parameter of at least one channel arranged at the end is pruned.

[0182] In an embodiment of the present application, the feedforward layer is pruned according to the amount of change in the gradient of the weight parameters. A small gradient change means that these weight parameters have not experienced drastic fluctuations during the training process, indicating that these weight parameters are close to the local minimum for minimizing the loss function, that is, these weight parameters contribute relatively limitedly to the performance improvement of the model, which means that even if a slight adjustment is made to the weight parameters, the performance of the model will not change much. Then, this pruning method is to no longer store and calculate these unimportant weight parameters that contribute little to the model output, so the feedforward layer has less data to calculate and store, thereby improving the processing speed of the feedforward layer.

[0183] In a possible implementation, the channels of the weight matrix of the feedforward layer are arranged in descending order according to the amount of change of the weight parameter on the gradient, including:

[0184] Get the sum of the absolute values ​​of the gradient changes of the weight parameters contained in each channel of the weight matrix of the feedforward layer;

[0185] Arrange the channels of the weight matrix of the feedforward layer in descending order according to the sum of the absolute values ​​of the gradient changes of the weight parameters they contain.

[0186] In the embodiment of the present application, because the amount of change in the gradient of the weight parameter can be positive or negative, the positive and negative represent the direction of the gradient change, and the absolute value of the number represents the amplitude of the gradient change, that is, the contribution of the weight parameter to the model. For the case where the contribution of negative values ​​to the model is similar to that of positive values, the absolute value of the weight parameter is used as the basis for pruning, and the absolute value of the gradient change is small, that is, the weight parameter with a small contribution to the model is discarded. The absolute value is used as the basis to improve the accuracy of pruning. On the other hand, the sum of the absolute values ​​of the gradient changes corresponding to each channel is sorted in order from large to small. Subsequently, pruning is performed according to the channel, and there is no need to traverse and judge each weight parameter, which saves the time of model pruning operations and greatly improves the efficiency of pruning.

[0187] In addition, in addition to expressing the importance of a channel by the sum of the absolute values ​​of the gradient changes of the weight parameters contained in each channel, a computer device can also express the importance of a channel by other data, for example, expressing the importance of a channel by the average value of the gradient changes of the weight parameters contained in each channel, and so on.

[0188] In one possible implementation, the change in gradient of each weight parameter in the weight matrix of the feedforward layer is determined by the trace of the Hessian matrix of the weight parameter and the change in the weight parameter before and after the forward reverse gradient calculation.

[0189] In an embodiment of the present application, a computer device can calculate the gradient change of the weight parameters before and after the reverse gradient calculation is updated through the Hessian matrix of the weight parameters, providing a solution that can accurately measure the gradient change of the weight parameters before and after the reverse gradient calculation is updated, thereby ensuring the accuracy of subsequent pruning.

[0190] The feedforward layer calculates the output of the attention layer. Directly using the magnitude method similar to the attention layer does not work well. The importance of parameters is measured based on the changes in the module on the sample gradient.

[0191] Assume that the loss function of the training process is as shown in formula (3):

[0192]

[0193] Where w represents the weight parameter, x represents the model input, y represents the corresponding label, and N is the number of samples. After the weight is pruned, Δw represents the change in the weight, and ΔL is the perturbation of the loss function after pruning. When the original model has been trained and converged to a local optimum, the model gradient is 0 (VwL(w) = 0), and the Hessian matrix of the weight is a semi-positive matrix. The perturbation of the loss function can be expressed in the form of a power series as shown in formula (4):

[0194]

[0195] Where gT is the Jacobian matrix of the weight matrix w, and H is the Hessian matrix. Since the gradient is 0 at this time, the first term in the formula is 0, and the last term is a high-order infinitesimal of Δw, which can be ignored numerically compared to the second term. The Hessian matrix in the second term can be expanded to be expressed as the following formula (5):

[0196]

[0197] stΔw p +w p =0 (6)

[0198] Where the subscript l represents the retained weight, p represents the pruned weight, Δwp and Δwl represent the corresponding perturbations respectively; where H represents the Hessian matrix, and the subscript of H represents the weight parameters that make up the Hessian matrix. For example, the above H p,p refers to the Hessian matrix composed of p and p, H l,l refers to the Hessian matrix composed of l and l, H p,l and H l,pis the Hessian matrix composed of p and l, and l and p. At the same time, this formula has the constraint condition of the above formula (6), that is, the change of the pruned weight, added to the value of the pruned weight, approaches 0. The Lagrange number multiplication method is used to solve the problem with constraints, as shown in formula (7):

[0199]

[0200] Take the partial derivative of this formula and set the partial derivative value to 0, as shown in formula (8):

[0201]

[0202] That is, as shown in formula (9):

[0203]

[0204] This formula is equivalent to formula (10), as shown below:

[0205] H p,p Δw p +H p,l Δw l +λ=0 (10)

[0206] H l,p Δw p +H l,l Δw l =0 (11)

[0207] Δw p +w p =0 Substituting into formula (11), we can get formula (12) and formula (13) as follows:

[0208] -H l,p w p +H l,l Δw l =0 (12)

[0209] Δw l =H l,l -1 H l,p w p (13)

[0210] Substituting equation (12) and equation (13) into equation (5), we can obtain equation (14):

[0211]

[0212] Ignore all cross terms, that is, the Hessian matrix containing both p and l, because we believe that each parameter is independent of each other, the diagonal matrix Hp,p(Diag(H p,p))'s trace(the trace of the matrix(Trace(H p,p )) is the sum of its eigenvalues, and also the sum of the diagonal values ​​of the diagonal matrix. The average of) can reflect the disturbance of loss after cutting off some parameters, that is, formula (15):

[0213]

[0214] According to H p,p The importance of the final weight is obtained by taking the average of the trace and the square of Δwp, that is, the change in loss after trimming the weight. Those weight parameters with small changes are considered to be suitable for trimming.

[0215] Assume the required sparsity is ffn_sparsity (for example, the default is 0.5, calculated as the number of model parameters after pruning / the number of model parameters before pruning). The detailed steps for pruning the FFN layer can be as follows:

[0216] (1) Perform forward and reverse gradient calculations on the current epoch data set;

[0217] (2) Calculate the importance of each weight according to formula (15);

[0218] (3) Aggregate and sort the weight importance values ​​according to the channel dimension;

[0219] (4) The k channels with the lowest weight values ​​after sorting are masked and not included in the subsequent gradient calculation;

[0220] (5) Repeat steps 1-4 until the parameter quantity meets the sparsity requirement;

[0221] (6) Continue training remaining_step batches to stabilize the model effect;

[0222] The cropped model is distilled layer by layer to restore its accuracy, and the restored model is kept for later use.

[0223] For the compression of the FFN layer, this scheme uses a gradient-based indicator to measure the importance of parameters. It can also use a magnitude indicator similar to the attention layer, that is, replacing the above-mentioned gradient change with a weight parameter.

[0224] In an embodiment of the present application, the trace of the Hessian matrix of the weight parameter can reflect the curvature of the weight parameter near the loss function curve. The smaller the curvature, the smaller the influence of the weight parameter on the model performance, that is, the smaller the contribution value. The forward and reverse gradient changes can be used to measure the changes of the weight parameters in forward and backward propagation, so as to determine the sensitivity of the weight parameters. That is to say, the lower the sensitivity of the weight parameter, the smaller the impact on the model output and the smaller the contribution value. Through the combined use of these two metrics, the influence of the weight parameter on the model can be captured more accurately, thereby improving the accuracy of pruning.

[0225] based on Figure 4 or Figure 6 The embodiment shown, referring to Figure 8 , which is a method flow chart of a model processing method according to an exemplary embodiment. Figure 8 As shown, the above step 430 can be implemented as step 430a, step 430b and step 430c.

[0226] Step 430a: Delete the embeddings corresponding to the low-frequency words in the vocabulary of the embedding layer.

[0227] Among them, the above-mentioned low-frequency words are determined according to the business scenarios of the application of the target model, and the low-frequency words corresponding to different business scenarios are different; among them, the low-frequency words corresponding to each business scenario can be pre-set / determined by the developer.

[0228] In another possible implementation, the low-frequency word is at least one word in the vocabulary that is arranged in descending order of appearance frequency.

[0229] Step 430b: Perform dimensionality reduction processing on the embedding layer through principal component analysis.

[0230] Step 430c: Decompose the embedding layer after dimensionality reduction by means of singular value decomposition (SVD) to obtain a pruned embedding layer.

[0231] The embedding layer represents the distributed word representation of the model input token. Pruning this layer should make the dimension of the embedding satisfy the semantic representation, so absolute magnitude or gradient method cannot be used for pruning.

[0232] Assume that the dimension of the embedding layer is M*N, where M is the vocabulary size and N is the dimension of the weight matrix. The following three steps are used to compress the embedding layer:

[0233] (1) Reduce the low-frequency words in the vocabulary and delete the corresponding embeddings to reduce the size of M;

[0234] (2) PCA is used to reduce the dimension of embedding and reduce the size of N;

[0235] (3) SVD is used to decompose the reduced-dimensional embedding to further reduce the number of parameters.

[0236] Among them, PCA (Principal Component Analysis), also known as principal component analysis, is the most widely used data dimensionality reduction algorithm. The main idea of ​​PCA is to map n-dimensional features to k-dimensional features. This k-dimensional feature is a new orthogonal feature also called principal component, which is a k-dimensional feature reconstructed on the basis of the original n-dimensional feature. The work of PCA is to sequentially find a set of mutually orthogonal coordinate axes from the original space. The selection of new coordinate axes is closely related to the data itself. Among them, the first new coordinate axis is selected in the direction with the largest variance in the original data, the second new coordinate axis is selected in the plane orthogonal to the first coordinate axis to make the variance the largest, and the third axis is the plane orthogonal to the first and second axes. By analogy, n such coordinate axes can be obtained. In this way, most of the variance of the new coordinate axes obtained at this time is contained in the first k coordinate axes, and the variance contained in the latter coordinate axes is almost 0. Therefore, the remaining coordinate axes can be ignored, and only the first k coordinate axes containing most of the variance are retained. In fact, this is equivalent to retaining only the dimensional features that contain most of the variance, while ignoring the feature dimensions that contain almost zero variance, thus achieving dimensionality reduction of data features. Fig. 9 shown.

[0237] Suppose there are m pieces of n-dimensional data. The algorithm steps of PCA are as follows:

[0238] (1) Organize the original data into an n-row and m-column matrix X;

[0239] (2) Zero-mean each row of X, that is, subtract the mean of this row;

[0240] (3) Find the covariance matrix C = 1 / m XXT;

[0241] (4) Find the eigenvalues ​​and corresponding eigenvectors of the covariance matrix;

[0242] (5) Arrange the eigenvectors into a matrix row by row from top to bottom according to the corresponding eigenvalues, and take the first k rows to form the matrix P;

[0243] (6)Y=PX is the data after dimension reduction to k dimensions.

[0244] Please refer to Fig.10 , which shows a schematic diagram of the SVD decomposition involved in this application, such as Fig.10As shown, SVD decomposition decomposes any matrix A into the product of an orthogonal matrix U, a diagonal matrix ∑, and another orthogonal matrix VT. The diagonal elements of a diagonal matrix are called the singular values ​​of the matrix. It can be proved that the singular values ​​are always greater than or equal to 0. When the singular values ​​of the diagonal matrix are arranged from large to small, the SVD decomposition is unique, as shown in Fig.10 As shown on the left. The distribution of singular values ​​is usually very uneven. In many cases, the sum of the first 10% or even 1% of the singular values ​​accounts for more than 99% of the sum of all singular values. Therefore, the matrix can be approximately described by the first r large singular values, as shown in Fig.10 As shown on the right. Assuming m = 10000, n = 8000, the original storage matrix A needs to store m*n = 80 million floating point numbers. If the singular value decomposition finds that the first 100 singular values ​​contribute 99% of the sum of the singular values, then it is possible to approximately retain only these 100 singular values ​​and the corresponding left and right singular vectors, then only 100+100×10000+100×8000=1.8001 million numbers need to be retained, which is less than 2.3% of the original. This patent uses SVD to decompose the weight matrices of the convolutional layer and the fully connected layer in the model into 3 small convolutions to achieve model compression. Optionally, the compressed model can be further fine-tuned on the data set to restore performance.

[0245] Assume that the matrix to be decomposed is A, and the SVD decomposition steps are as follows:

[0246] (1) Calculate AT*A;

[0247] (2) Calculate the eigenvector and eigenvalue of AT*A;

[0248] (3) Arrange the eigenvalues ​​in descending order, select the top k largest eigenvalues ​​and their corresponding eigenvectors to form the matrix V;

[0249] (4) Take the square root of the selected non-zero eigenvalues ​​and fill the diagonal of ∑ in the order of the eigenvectors in V;

[0250] (5)U can be obtained by solving AV=U∑.

[0251] The output dimension of the embedding layer changes after compression, so the adaptive distillation method is used to distill the compressed decomposed model again to restore the accuracy. At this time, an intermediate layer with a dimension of I needs to be added between the corresponding module layers of the teacher model and the student model during distillation to achieve the adaptive distillation. The loss function of the distillation still uses KL divergence, as shown in formula (2). Please refer to Fig.11 , which shows a schematic diagram of the adaptive distillation involved in this embodiment, Fig.11In the adaptive distillation process, the same model training data (such as input text) is respectively input into the teacher model on the left and the pruned target model on the right. For the output parameters of each network layer 1101 in the target model, the output parameters of the network layer 1102 corresponding to the network layer 1101 in the teacher model are respectively obtained. The output parameters of the network layer 1102 are reduced in dimension through the intermediate layer 1103. The dimension of the output parameters of the network layer 1102 after dimension reduction is the same as the dimension of the output parameters of the network layer 1102. The weight parameters of the pruned target model are updated through the difference between the output parameters of the network layer 1101 and the output parameters of the network layer 1102 after dimension reduction. Optionally, during the distillation process, the weight parameters of the pruned target model can also be updated through the difference between the prediction result 1105 of the target model and the prediction result 1104 of the teacher model.

[0252] For the compression of the embedding layer, this solution adopts the low-rank decomposition + PCA method. It can also be achieved by directly training a lower-dimensional embedding layer to achieve compression in the real dimension.

[0253] In the embodiment of the present application, by deleting low-frequency words to reduce the dimension, it means that the embedding layer needs less data to be trained and the calculation time is shorter, thereby making the training speed faster. On the other hand, the embedding layer matrix is ​​decomposed into three matrices using singular value decomposition. By retaining the main singular values ​​and their corresponding vectors, dimensionality reduction and feature extraction can be achieved faster, thereby improving the training efficiency of the embedding layer.

[0254] based on Figure 4 , Figure 6 or Figure 8 The embodiment shown, referring to Fig.12 , which is a method flow chart of a model processing method according to an exemplary embodiment. Fig.12 As shown, the above method may further include step 440.

[0255] Step 440: Quantify and store the parameters in the target model.

[0256] In a possible implementation, the parameters in the target model are quantized and stored in FP16 or int8 format.

[0257] For example, the parameters in the target model are quantized and stored in FP16 format. After the model is trained, the parameters in the model are converted from floating point format (usually 32-bit floating point numbers) to FP16 format using the built-in tools or functions of the deep learning framework, and the converted parameters are saved in the model file in a custom model file format, such as HDF5 or ONNX.

[0258] In an embodiment of the present application, for the above-mentioned pruned and distilled restored model, in order to further compress the model size, FP16 or int8 quantization can be used to store model parameters. When FP16 storage is used, that is, the FP32 floating-point parameters in the original model are saved in the FP16 half-precision format, the size of the original model can be reduced by half. When int8 quantization is used, that is, fixed-point integers between -128 and 127 are used to save the scaled FP32 floating-point parameters, the size of the original model can be reduced to 1 / 4 of the size.

[0259] Taking the target model as an example, the evaluation results of the compression of the Bert Base model on the public text classification dataset can be shown in Table 1. For the original large model with 109.5M parameters, a compression rate of 84.32% can be achieved after multi-stage hybrid compression, while the model accuracy only loses 3.6%, from 84.95% to 81.35%.

[0260] Table 1

[0261] Parameter quantity / M Compression Accuracy Original Model 109.5 0.0％ 84.95％ Attention Pruning 46.24 57.76％ 84.42％ FFN Pruning 31.8 70.95％ 83.79％ Embedding pruning 17.17 84.32％ 81.35％

[0262] The solution shown in the above embodiment of the present application relates to the technical field of neural network model compression, and more specifically, to a multi-stage model compression and distillation method for a transformer structure. Please refer to Fig.13 , which shows a process framework diagram of a model pruning involved in the present application, wherein the model at least includes multiple multi-head attention layers 1301, multiple feedforward network layers 1302, and multiple embedding layers 1303 before the multi-head attention layers 1301 and the multiple feedforward network layers 1302; taking the model as a text processing model as an example, the model is a machine learning model obtained by pre-training and fine-tuning after removing and simplifying the network layer of the teacher model used for text processing, such as Fig.13 As shown in Figure 1, the model pruning is divided into three stages that are executed successively:

[0263] Stage 1: Attention layer pruning stage.

[0264] Step S1-1: The model is subjected to multiple forward and reverse gradient calculations and updates through the model training data (text sample 1304). After each update, an importance evaluation method based on the magnitude of weight parameters is adopted, and the channel is used as the pruning granularity to perform structured pruning of parameters on the multi-head attention layer 1301 in the model.

[0265] Step S1-2: Continue to perform multiple forward and reverse gradient calculations and updates on the model through model training data to make the model stable.

[0266] Step S1-3: Using the teacher model 1305, the accuracy of the model is restored by layer-by-layer distillation.

[0267] Phase 2: Feedforward network layer pruning phase.

[0268] Step S2-1: The model is subjected to multiple forward and reverse gradient calculations and updates through model training data. After each update, an importance evaluation method based on gradient changes is used, with channels as the pruning granularity, to perform structured parameter pruning on the feedforward network layer 1302 in the model.

[0269] Step S2-2: Continue to perform multiple forward and reverse gradient calculations and updates on the model through model training data to make the model stable.

[0270] Step S2-3: Restore the accuracy of the model by layer-by-layer distillation through the teacher model.

[0271] Phase 3: Embedding layer pruning phase.

[0272] Step S3-1: Use PCA dimensionality reduction combined with low-rank decomposition to reduce the weight parameters of the embedding layer 1303.

[0273] Step S3-2: Recover the accuracy of the embedding layer 1303 using the teacher model using an adaptive distillation method.

[0274] For the trained transformer model to be compressed, different methods are used to compress the module parameters of the embedding layer, multi-head attention layer and feedforward network layer respectively. At the same time, distillation learning is performed immediately after compression in each stage to restore the model accuracy and minimize the loss of model accuracy.

[0275] The scheme shown in this application proposes that the order of multi-stage compression and the process of multiple precision recovery are crucial to the compression effect and accuracy of the model. At the same time, this application proposes a set of feasible compression and distillation schemes for the transformer module.

[0276] Please refer to Fig.14 , which shows a block diagram of a model processing device provided by an exemplary embodiment of the present application. The model processing device can be implemented as all or part of a computer device by hardware or a combination of hardware and software to achieve the above Figure 5 , Figure 7 , Fig. 9 or Fig.13 All or part of the steps in the embodiments shown. Fig.14 As shown, the model processing device includes:

[0277] A data acquisition module 1401 is used to acquire model training data of a target model; the target model includes an embedding layer, and a plurality of feature processing blocks connected after the embedding layer, the feature processing blocks include one or more attention layers, and one or more feed-forward layers;

[0278] A first pruning module 1402, configured to prune at least one of the attention layer and the feedforward layer in the plurality of feature processing blocks based on the model training data;

[0279] The second pruning module 1403 is used to prune the embedding layer after pruning at least one of the attention layer and the feedforward layer in the multiple feature processing blocks.

[0280] In a possible implementation, the first pruning module 1402 is used to:

[0281] Based on the model training data, pruning the attention layers in the plurality of feature processing blocks;

[0282] After the pruning process of the attention layers in the multiple feature processing blocks is completed, the pruning process of the feedforward layers in the multiple feature processing blocks is performed based on the model training data.

[0283] In a possible implementation, the first pruning module 1402 is configured to iteratively perform the following steps:

[0284] Based on the model training data, forward reverse gradient calculation is performed on the weight of the target model to obtain updated values ​​of each weight parameter in the weight matrix of the attention layer, wherein the weight matrix includes multiple channels, and each channel includes multiple weight parameters;

[0285] Arrange the channels of the weight matrix of the attention layer in descending order according to the absolute values ​​of the weight parameters;

[0286] For each channel of the weight matrix of the attention layer, a weight parameter of at least one channel arranged at the end is pruned.

[0287] In a possible implementation, the first pruning module 1402 is used to:

[0288] Obtain the sum of the absolute values ​​of the weight parameters contained in each channel of the weight matrix of the attention layer;

[0289] Arrange the channels of the weight matrix of the attention layer in descending order according to the sum of the absolute values ​​of the weight parameters they contain.

[0290] In a possible implementation, the first pruning module 1402 is configured to iteratively perform the following steps:

[0291] Based on the model training data, forward reverse gradient calculation is performed on the weight of the target model to obtain the gradient change of each weight parameter in the weight matrix of the feedforward layer, wherein the weight matrix includes multiple channels, and each channel includes multiple weight parameters;

[0292] Arrange the channels of the weight matrix of the feedforward layer in descending order according to the amount of change of the weight parameter on the gradient;

[0293] A pruning process is performed on the weight parameters of at least one channel arranged at the end in each channel of the weight matrix of the feed-forward layer.

[0294] In a possible implementation, the first pruning module 1402 is used to:

[0295] Obtaining the sum of the absolute values ​​of the gradient changes of the weight parameters contained in each channel of the weight matrix of the feedforward layer;

[0296] Arrange the channels of the weight matrix of the feedforward layer in descending order according to the sum of the absolute values ​​of the gradient changes of the weight parameters contained in each channel.

[0297] In one possible implementation, the change in gradient of each weight parameter in the weight matrix of the feedforward layer is determined by the trace of the Hessian matrix of the weight parameter and the change of the weight parameter before and after the forward reverse gradient calculation.

[0298] In a possible implementation, the second pruning module 1403 is used to:

[0299] Deleting embeddings corresponding to low-frequency words in the vocabulary of the embedding layer; the low-frequency words are at least one word in the vocabulary that is arranged in descending order of frequency of occurrence;

[0300] Performing dimensionality reduction processing on the embedding layer by principal component analysis;

[0301] The embedding layer after dimensionality reduction is decomposed by singular value decomposition (SVD) to obtain the embedding layer after pruned.

[0302] In a possible implementation manner, the device further includes:

[0303] A training module, configured to, after pruning the attention layer, the feedforward layer or the embedding layer, train and update the remaining model parameters in the target model based on the model training data until the target model converges;

[0304] The distillation module is used to restore the accuracy of the target model through distillation learning.

[0305] In a possible implementation manner, the device further includes:

[0306] The quantization module is used to quantize and store the parameters in the target model.

[0307] In a possible implementation manner, the quantizing and storing the parameters in the target model includes:

[0308] The parameters in the target model are quantized and stored in FP16 or int8 format.

[0309] Fig.15 1 is a schematic diagram of a computer device according to an exemplary embodiment. The computer device 1500 includes a central processing unit (CPU) 1501, a system memory 1504 including a random access memory (RAM) 1502 and a read-only memory (ROM) 1503, and a system bus 1505 connecting the system memory 1504 and the central processing unit 1501. The computer device 1500 also includes a basic input / output system (I / O system) 1506 for helping to transmit information between various devices in the computer, and a large-capacity storage device 1507 for storing an operating system 1513, an application program 1514 and other program modules 1515.

[0310] The basic input / output system 1506 includes a display 1508 for displaying information and an input device 1509 such as a mouse and a keyboard for user inputting information. The display 1508 and the input device 1509 are connected to the central processing unit 1501 through an input / output controller 1510 connected to the system bus 1505. The basic input / output system 1506 may also include an input / output controller 1510 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1510 also provides output to a display screen, a printer, or other types of output devices.

[0311] The mass storage device 1507 is connected to the central processing unit 1501 through a mass storage controller (not shown) connected to the system bus 1505. The mass storage device 1507 and its associated computer readable medium provide non-volatile storage for the computer device 1500. That is, the mass storage device 1507 may include a computer readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0312] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid-state storage technologies, CD-ROM, DVD (Digital Video Disc) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media is not limited to the above. The above-mentioned system memory 1504 and mass storage device 1507 can be collectively referred to as memory.

[0313] The computer device 1500 can be connected to the Internet or other network devices through a network interface unit 1511 connected to the system bus 1505 .

[0314] The memory also includes one or more programs, which are stored in the memory. The CPU 1501 implements the one or more programs by executing the one or more programs. Figure 4 , Figure 6 , Figure 8 or Fig.12 All or part of the steps in the method shown.

[0315] In an exemplary embodiment, a chip is also provided, which includes a programmable logic circuit and / or program instructions. When the chip runs on a computer device, it is used to implement all or part of the steps of the method shown in the above embodiments of the present application.

[0316] In an exemplary embodiment, a computer program product is also provided, the computer program product includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor reads and executes the computer instructions from the computer-readable storage medium to implement all or part of the steps of the method shown in the above-mentioned various embodiments of the present application.

[0317] In an exemplary embodiment, a computer-readable storage medium is also provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement all or part of the steps of the method shown in the above-mentioned embodiments of the present application.

[0318] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0319] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0320] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A model processing method, characterized in that: The method comprises: Obtain model training data of a target model; the target model includes an embedding layer, and a plurality of feature processing blocks connected after the embedding layer, the feature processing blocks including one or more attention layers, and one or more feed-forward layers; Based on the model training data, pruning at least one of the attention layer and the feedforward layer in the plurality of feature processing blocks; After pruning the attention layer and at least one of the feed-forward layers in the multiple feature processing blocks, pruning is performed on the embedding layer.

2. The method according to claim 1, characterized in that The step of pruning at least one of the attention layer and the feedforward layer in the plurality of feature processing blocks based on the model training data comprises: Based on the model training data, pruning the attention layers in the plurality of feature processing blocks; After the pruning process of the attention layers in the multiple feature processing blocks is completed, the pruning process of the feedforward layers in the multiple feature processing blocks is performed based on the model training data.

3. The method according to claim 2, characterized in that The pruning of the attention layers in the plurality of feature processing blocks based on the model training data comprises: Iterate through the following steps: Based on the model training data, forward reverse gradient calculation is performed on the weight of the target model to obtain updated values ​​of each weight parameter in the weight matrix of the attention layer, wherein the weight matrix includes multiple channels, and each channel includes multiple weight parameters; Arrange the channels of the weight matrix of the attention layer in descending order according to the absolute values ​​of the weight parameters; For each channel of the weight matrix of the attention layer, a weight parameter of at least one channel arranged at the end is pruned.

4. The method according to claim 3, characterized in that: The method of arranging the channels of the weight matrix of the attention layer in descending order according to the absolute values ​​of the weight parameters includes: Obtain the sum of the absolute values ​​of the weight parameters contained in each channel of the weight matrix of the attention layer; Arrange the channels of the weight matrix of the attention layer in descending order according to the sum of the absolute values ​​of the weight parameters they contain.

5. The method according to claim 2, characterized in that: The pruning of the feedforward layers in the plurality of feature processing blocks based on the model training data comprises: Iterate through the following steps: Based on the model training data, forward reverse gradient calculation is performed on the weight of the target model to obtain the gradient change of each weight parameter in the weight matrix of the feedforward layer, wherein the weight matrix includes multiple channels, and each channel includes multiple weight parameters; Arrange the channels of the weight matrix of the feedforward layer in descending order according to the amount of change of the weight parameter on the gradient; A pruning process is performed on the weight parameters of at least one channel arranged at the end in each channel of the weight matrix of the feed-forward layer.

6. The method according to claim 5, characterized in that The method of arranging the channels of the weight matrix of the feedforward layer in descending order according to the amount of change of the weight parameter on the gradient includes: Obtaining the sum of the absolute values ​​of the gradient changes of the weight parameters contained in each channel of the weight matrix of the feedforward layer; Arrange the channels of the weight matrix of the feedforward layer in descending order according to the sum of the absolute values ​​of the gradient changes of the weight parameters contained in each channel.

7. The method according to claim 5, characterized in that The amount of change in gradient of each weight parameter in the weight matrix of the feedforward layer is determined by the trace of the Hessian matrix of the weight parameter and the amount of change of the weight parameter before and after the forward reverse gradient calculation.

8. The method according to any one of claims 1 to 7, characterized in that: The pruning of the embedding layer includes: Deleting embeddings corresponding to low-frequency words in the vocabulary of the embedding layer; Performing dimensionality reduction processing on the embedding layer by principal component analysis; The embedding layer after dimensionality reduction is decomposed by singular value decomposition (SVD) to obtain the embedding layer after pruned.

9. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: After pruning the attention layer, the feedforward layer or the embedding layer, training and updating the remaining model parameters in the target model based on the model training data until the target model converges; The accuracy of the target model is restored by distillation learning.

10. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The parameters in the target model are quantified and stored.

11. The method according to claim 10, characterized in that The quantizing and storing the parameters in the target model includes: The parameters in the target model are quantized and stored in FP16 or int8 format.

12. A model processing device, characterized in that: The device comprises: A data acquisition module, used to acquire model training data of a target model; the target model includes an embedding layer, and a plurality of feature processing blocks connected after the embedding layer, wherein the feature processing blocks include one or more attention layers, and one or more feed-forward layers; A first pruning module, configured to prune at least one of the attention layer and the feedforward layer in the plurality of feature processing blocks based on the model training data; The second pruning module is used to prune the embedding layer after pruning at least one of the attention layer and the feedforward layer in the multiple feature processing blocks.

13. A computer device, characterized in that: The computer device includes a processor and a memory, wherein instructions are stored in the memory, and the instructions are executed by the processor to implement the model processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: The storage medium stores instructions, and the instructions are executed by a processor of a computer device to implement the model processing method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the model processing method according to any one of claims 1 to 11.

Images