Method and apparatus for generative pre-training of language models by quantization compression

Abstract

A method performed by a processing system for quantifying a neural network model is provided. The method includes determining a scaling factor based on a distribution of weights associated with the neural network model; determining a quantized weight based on the scaling factor and the weight associated with the distribution; determining the training loss of the neural network model based on the quantized weight during the training period of the neural network model; and determining an updated scaling factor of the neural network model based on the gradient of the training loss.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims the benefit of U.S. Patent Application No. 17945978, filed by Lu Hou on September 15, 2022, entitled "Method and Device for Compressing Generative Pre-trained Language Models via Quantization," which is incorporated herein by reference. Technical Field

[0002] The present invention relates generally to machine learning techniques and, more particularly, to processing data using neural network techniques. Background Art

[0003] The framework of pre-training and fine-tuning on large-scale language corpora has been widely used in natural language understanding and generation tasks. Common pre-training objectives in natural language include masked language modeling (MLM) and causal language modeling (CLM). For example, bidirectional encoder representations from transformers (BERT) is an exemplary model of MLM. Generative pre-trained transformer is an exemplary model of CLM. MLM predicts the masked word based on other words in the sentence, which is bidirectional in nature because the masked token "sees" its left and right tokens. Therefore, MLM understands the sequence as a whole and achieves remarkable performance in downstream natural language understanding tasks. For CLM, such as generative pre-training Transformer (GPT), it predicts each token in order from left to right. Due to its unidirectional nature, GPT is suitable for generation tasks, whether zero-shot or fully fine-tuned.

[0004] Transformer-based generative pre-trained language models (“PLMs”) have demonstrated strong multi-task and few-shot learning capabilities and have achieved remarkable performance on a variety of tasks. However, due to the large number of parameters and token-by-token generation process in these models, Transformer-based PLMs tend to incur high overhead in terms of computational complexity and memory bandwidth. Many methods have been proposed to compress PLMs, but most of them focus on comprehension tasks, such as sentence classification using BERT. Recent studies have attempted to compress GPT-2 using tensor decomposition and knowledge distillation, but achieved much less compression than BERT.

[0005] However, the underlying difficulties in compressing generative language models remain unclear. Therefore, it is necessary to develop a technique that provides solutions to improve model compression in natural language learning applications. Summary of the invention

[0006] In an exemplary embodiment, the present invention provides a method for quantizing a neural network model performed by a processing system. The method includes: (a) determining a scaling factor based on a distribution of weights associated with the neural network model; (b) determining a quantized weight based on the scaling factor and the weight associated with the distribution; (c) determining a training loss of the neural network model based on the quantized weight during training of the neural network model; (d) determining an updated scaling factor of the neural network model based on a gradient of the training loss.

[0007] In another exemplary embodiment, the method further comprises: determining an average weight size based on the distribution of the weights; determining a clipping factor as a product of the scaling factor and the average weight size. Determining the quantized weights based on the weights in the distribution and the scaling factor is based on the clipping factor.

[0008] In another exemplary embodiment, the method further comprises: determining the average weight magnitude by an L1 norm function of the weight associated with the distribution.

[0009] In another exemplary embodiment, the weights associated with the distribution are divided into a plurality of value ranges based on the scaling factor. The method further comprises: calculating a gradient contribution of each of the weights associated with the distribution based on the value range that the corresponding weight falls into; and calculating the gradient of the training loss by aggregating the gradient contributions from the weights associated with the distribution.

[0010] In another exemplary embodiment, the method further comprises: setting an initial value of the scaling factor to 1.

[0011] In another exemplary embodiment, the method further comprises: determining an initial value of the scaling factor based on an initial value of the weight in the neural network model.

[0012] In another exemplary embodiment, the method further comprises: determining an optimized scaling factor for the task by performing multiple iterations of (a) to (d).

[0013] In another exemplary embodiment, the updated scaling factor is associated with a weight matrix in a plurality of weight matrices in the neural network model. The method further includes determining an updated scaling factor for each of the other weight matrices in the plurality of weight matrices in the neural network model by performing (a) to (d) on the corresponding weight matrix.

[0014] In another exemplary embodiment, the method further includes: applying the updated scaling factor to the neural network model; determining the quantized weights associated with the updated scaling factor as learnable weights in the neural network model; and updating the neural network model by updating the learnable weights.

[0015] In another exemplary embodiment, the neural network model having the quantized weights associated with the updated scaling factor is included in a student network. The student network is trained using a teacher network. The method further includes: obtaining a first set of token representations by the student network and a second set of token representations by the teacher network based on an input sequence; determining a first loss based on a pairwise comparison between a first token in the first set of token representations and a second token in the second set of token representations; determining a third loss during training of the student network based on the first loss; and updating the student network based on the third loss.

[0016] In another exemplary embodiment, the first loss includes a student-to-teacher loss and a teacher-to-student loss.

[0017] In another exemplary embodiment, the method further includes: obtaining a first logit set and a second logit set from the student network based on the input sequence; determining a second loss based on a pairwise comparison between each first logit in the first logit set and a corresponding second logit in the second logit set; and determining the third loss based on the first loss and the second loss.

[0018] In another exemplary embodiment, determining the third loss based on the first loss and the second loss further comprises: determining the third loss by aggregating the first loss and the second loss with an adjustable factor.

[0019] In another exemplary embodiment, the present invention provides a system for quantizing a neural network model. The system includes one or more processors and a non-transitory computer-readable medium having computer-executable instructions stored therein. The computer-executable instructions, when executed by one or more processors, facilitate the one or more processors to: (a) determine a scaling factor based on a distribution of weights associated with the neural network model; (b) determine a quantized weight based on the scaling factor and the weight associated with the distribution; (c) determine a training loss of the neural network model based on the quantized weights during training of the neural network model; (d) determine an updated scaling factor of the neural network model based on a gradient of the training loss.

[0020] In another exemplary embodiment, the one or more processors further facilitate: determining an average weight magnitude based on the distribution of the weights; determining a clipping factor as a product of the scaling factor and the average weight magnitude. Determining the quantized weights based on the weights in the distribution and the scaling factor is based on the clipping factor.

[0021] In another exemplary embodiment, the weights associated with the distribution are divided into a plurality of value ranges based on the scaling factor. The one or more processors further facilitate: calculating a gradient contribution of each of the weights associated with the distribution based on the value range within which the corresponding weight falls; and calculating a gradient of the training loss by aggregating the gradient contributions from the weights associated with the distribution.

[0022] In another exemplary embodiment, the updated scaling factor is associated with a weight matrix in a plurality of weight matrices in the neural network model. The one or more processors further facilitate determining an updated scaling factor for each of the other weight matrices in the plurality of weight matrices in the neural network model by performing (a) to (d) on the corresponding weight matrix.

[0023] In another exemplary embodiment, the one or more processors further facilitate: applying the updated scaling factor to the neural network model; determining the quantized weights associated with the updated scaling factor as learnable weights in the neural network model; and updating the neural network model by updating the learnable weights.

[0024] In another exemplary embodiment, the neural network model having the quantized weights associated with the updated scaling factor is included in a student network. The student network is trained using a teacher network. The one or more processors also facilitate: based on an input sequence, obtaining a first set of token representations by the student network and a second set of token representations by the teacher network; determining a first loss based on a pairwise comparison between a first token in the first set of token representations and a second token in the second set of token representations; based on the input sequence, obtaining a first set of logits from the student network and a second set of logits from the teacher network; determining a second loss based on a pairwise comparison between each first logit in the first set of logits and a corresponding second logit in the second set of logits; determining a third loss during training of the student network based on the first loss and the second loss; and updating the student network based on the third loss.

[0025] In yet another exemplary embodiment, the present invention provides a non-transitory computer-readable medium storing processor-executable instructions for quantizing a neural network model. The computer-executable instructions, when executed by one or more processors, cause the one or more processors to facilitate: (a) determining a scaling factor based on a distribution of weights associated with the neural network model; (b) determining quantized weights based on the scaling factor and the weights associated with the distribution; (c) determining a training loss of the neural network model based on the quantized weights during training of the neural network model; (d) determining an updated scaling factor of the neural network model based on a gradient of the training loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The system and method for data processing are described in detail below with reference to the accompanying drawings, wherein:

[0027] Figure 1A An exemplary network environment provided by one or more examples of the present invention is shown;

[0028] Figure 1B An exemplary computer system provided by one or more examples of the present invention is shown;

[0029] Figure 2A and Figure 2B is a diagram showing an exemplary weight distribution based on weights provided by a 12-layer full-precision GPT-2 model;

[0030] Figure 3 is an exemplary process of data processing provided by one or more examples of the present invention;

[0031] Figure 4 An exemplary training workflow provided by one or more examples of the present invention is shown;

[0032] Figure 5 It is an exemplary process of data processing provided by one or more examples of the present invention. DETAILED DESCRIPTION

[0033] Disclosed are systems and methods related to quantization-based compression techniques, which provide a scheme for compressing a generative language model with improved performance (e.g., an increased compression ratio). The quantization-based compression techniques in the present invention can be used to obtain quantized / compressed models, which can be widely used in various tasks, such as dialogue systems, question-answering, poetry generation, code generation, and image generation in various products. Examples of products include, but are not limited to, cloud systems, terminal devices, autonomous driving vehicles, and the like. The technology based on the present invention can achieve model compression, thereby greatly improving the storage capacity required to carry large models and the inference speed of the model. Therefore, the technology of the present invention can greatly save the budget for providing cloud services based on generative language models.

[0034] Figure 1A An exemplary network environment 100 provided by one or more examples of the present invention is shown. Machine learning techniques for implementing various embodiments disclosed herein may occur in the exemplary network environment 100. A network environment suitable for implementing embodiments of the present invention may include one or more client devices 120, servers 130, and / or other device types.

[0035] The components of the network environment can communicate with each other through one or more networks 110, which can be wired, wireless, or both. As examples, the network 110 can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (e.g., the Internet), and / or one or more private networks. In the case where the network includes a wireless communication network, components such as base stations, communication towers, access points, or other components can provide wireless connections.

[0036] Compatible network environments may include one or more peer-to-peer network environments (in which case, the network environment may not include a server) and one or more client-server network environments (in which case, the network environment may include one or more servers). In a peer-to-peer network environment, the functionality described herein with respect to one or more servers may be implemented in any number of client devices.

[0037] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented in one or more servers, the one or more servers may include one or more core network servers and / or edge servers. The framework layer may include a framework for supporting software at the software layer and / or one or more applications at the application layer. The software or one or more applications may include web-based service software or applications, respectively. In an embodiment, one or more client devices may use web-based service software or applications (e.g., by accessing the service software and / or application via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open source software web application framework, such as a distributed file system that may be used for large-scale data processing (e.g., "big data").

[0038] A cloud-based network environment may provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across a state, region, country, world, etc.). A cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0039] One or more client devices 120 may include Figure 1B The client device 120 may be implemented as a personal computer (PC), a laptop, a mobile device, a smartphone, a tablet, a virtual reality headset, a video player, a video camera, a vehicle, a virtual machine, a drone, a robot, a handheld communication device, an in-vehicle computer system, an embedded system controller, a workstation, an edge device, any combination of these depicted devices, or any other suitable device.

[0040] Figure 1B 1 is a block diagram of an exemplary computer system 150 for implementing various functions provided by one or more embodiments of the present invention. In some examples, the computer system 150 may be configured as follows: Figure 1AThe network environment 100 is shown as being implemented in a client device 120 or a server 130. One or more computing systems 150, one or more client devices 120, one or more servers 130, or a combination thereof may form a processing system for executing the processes of the present invention.

[0041] like Figure 1B As shown, the computer system 150 may include one or more processors 160, a communication interface 170, a memory 180, and a display 190. The one or more processors 160 may be used to perform operations according to instructions stored in the memory 180. The one or more processors 160 may include any suitable type of general or special purpose microprocessor (e.g., a CPU or GPU, respectively), a digital signal processor, a microcontroller, etc. The memory 180 may be used to store computer readable instructions, which, when executed by the one or more processors 160, may cause the one or more processors 160 to perform various operations disclosed herein. The memory 180 may be any non-transitory type of mass storage, such as volatile or non-volatile, magnetic, semiconductor-based, tape-based, optical, removable, non-removable or other type of storage device or tangible computer readable medium, including but not limited to read-only memory (ROM), flash memory, dynamic random access memory (RAM) and / or static RAM.

[0042] Communication interface 170 can be used to connect computer system 150 to other devices or systems (such as Figure 1A The communication interface 170 may be used to transmit information between the client device 120 and / or the server 130 shown in the figure. For example, the communication interface 170 may include an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection. As another example, the communication interface 170 may include a local area network (LAN) card to provide a data communication connection to a compatible LAN. As another example, the communication interface 170 may include a high-speed network adapter, such as a fiber optic network adapter, a 10G Ethernet adapter, etc. A wireless link may also be implemented through the communication interface 170. In this implementation, the communication interface 170 may send and receive electrical signals, electromagnetic signals, or optical signals through the network, which carry digital data streams representing various types of information. The network may typically include a cellular communication network, a wireless local area network (WLAN), a wide area network (WAN), etc.

[0043] The communication interface 170 may also include various I / O devices, such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, a camera, a biosensor, etc. A user may input data to the computer system 150 (eg, a terminal device) through the communication interface 170 .

[0044] Display 190 may be integrated as part of computer system 150, or may be provided as a separate device communicatively coupled to computer system 150. Display 190 may include a display device such as a liquid crystal display (LCD), a light emitting diode display (LED), a plasma display, or any other type of display, and provide a graphical user interface (GUI) for user input and data depiction presented on the display. In some embodiments, display 190 may be integrated as part of communication interface 170.

[0045] The model compression techniques disclosed herein can be used to deploy state-of-the-art deep networks in devices with low power consumption and resources without significantly compromising the accuracy of the language model. By compressing the size of the model and / or reducing the latency of the model, the model can have fewer parameters and require less RAM capacity. Latency reduction refers to the reduction in the time it takes the model to make predictions or inferences based on the model's inputs, thereby reducing energy consumption at runtime. Therefore, model compression can reduce CPU / GPU time, memory usage, and disk storage, making language models suitable for production that were previously too expensive, slow, or bulky.

[0046] Processes for model compression may include pruning, quantization, low-rank approximation and sparsification, knowledge distillation, and neural architecture search (NAS). Pruning involves removing connections between neurons or between all neurons, channels, or filters from a trained network, so that pruning compresses the model by reducing the number of weights. In general, quantization is the process of mapping values ​​from a large set to values ​​from a small set, meaning that the output includes a smaller range of possible values ​​than the input, ideally without losing too much information in the process. In other words, quantization reduces the size of the weights. The goal of low-rank approximation is to approximate the numerous redundant weights of a layer using a linear combination of fewer weights. Knowledge distillation is the idea of ​​transferring knowledge from a large trained model (or collection of models) to a smaller model for deployment, by training the smaller model to mimic the output of the larger model. In the most general sense, NAS is a search for a set of decisions that define the different components of a neural network, and it is a systematic, automated way to learn the best model architecture.

[0047] In the present invention, in contrast to traditional methods, a quantization method implemented using module-level dynamic scaling is used to reduce the sensitivity of the clipping factor to initialization, and is combined with improved gradient estimates of weights within the clipping range. The clipping factors learned by the quantization method of the present invention provide finer resolution for most weights. In some examples, the quantized model is further implemented by applying token-level contrastive distillation learning, which helps to alleviate the word collapse problem by making token representations more distinguishable. In some embodiments, token-level contrastive distillation learning may be superior to sequence-level contrastive distillation learning. Sequence-level knowledge distillation (SLKD) is a model compression technique that uses a large, accurate teacher model to train a smaller, under-parameterized student model.

[0048] To illustrate, the general formulation for quantization-aware training of compressed generative PLMs is described below. A language model is a probability distribution over sequences of words in a given language L. The vectorized full-precision weights are denoted as w. Given a sequence of length n (e.g., including n tokens in the sequence), the language model assigns a probability P(w1,…,w n ).

[0049] Quantization is the process of mapping full-precision values ​​to a smaller set of discrete finite values. For example, most deep network models trained on GPUs use the 32-bit floating point ("FP32") format as full precision, where data is represented as 32-bit floating point numbers. A clipping factor is associated with the range of the discrete finite set of values, for example, by defining the maximum absolute value in the discrete finite set of values. Quantization of deep learning networks is an important step used to help speed up inference and reduce memory and power consumption in embedded devices. The difference between an input value and its quantized value (e.g., rounding error) is called the quantization error. The device or algorithmic function that performs quantization is called a quantizer.

[0050] Quantization-aware training trains a quantized network (also called a quantized model) from scratch or from a trained full-precision network (e.g., PLM). Quantization can be performed in different ways. For example, binarization converts each weight value to one of "-1" or "+1". Ternaryization converts each weight value to one of "-1", "0", or "+1". In addition, b-bit quantization can be performed as linear quantization, quantizing each weight value to Where k = 2 b-1 Alternatively, the b-bit quantization can be performed as a logarithmic quantization, quantizing each weight value to Where k = 2 b-1 -1.

[0051] For the purpose of illustration, linear quantization is described in the present invention as an example. It should be understood that embodiments of the present invention can be easily extended to other types of quantization. When performing quantization-based compression on PLM, each forward pass first limits the weights by a positive limit factor α, and then quantizes the limited weights to b bits, as shown below: where Q(·) is a quantization function that maps each entry in clip(w,-α,α) to a set of uniform discrete values The closest quantized value and k = 2 b-1 -1. Therefore, based on w q Calculating training loss During backpropagation, due to the non-differentiability of Q(·), we can use the quantized weights The full-precision weight w is updated with the gradient of . For example, a straight-through estimator (STE) can be used to update the full-precision weight w based on the quantized weight w q The gradient of is converted to the gradient associated with the weight w. STE can be used to estimate the gradient of a function by ignoring the derivative of the threshold function and passing the incoming gradient as if the function was the identity function. The calculated gradient is used to update the learnable weights in the model.

[0052] It is expected that the ideal clipping factor takes into account most of the full-precision weights through clipping. In other words, the ideal clipping factor quantizes the optimal range of densely distributed data, thereby reducing the quantization error. Various techniques can be used to facilitate the quantization process. According to some embodiments, a learnable clipping factor can be used for quantization. For example, the parameterized clipping activation (PACT) technique parameterizes the clipping factor as an activation clipping parameter and trains a neural network to predict the optimal activation clipping parameter. The PACT technique typically learns the clipping factor α by gradient descent. The learned step size quantization (LSQ) technique learns the step size instead of a clipping factor, which requires careful initialization and gradient updates.

[0053] The experimental results show that there are different distributions on the weight values ​​embedded in the neural network model. For example, generative pre-trained Transformer 2 (GPT-2) is an open source artificial intelligence (AI) created by OPENAI. It is a large Transformer-based language model with 1.5 billion parameters, including various learnable weights. GPT-2 has a 12-layer architecture that includes various functional modules in each layer of the embedding model, such as multi-head attention modules, linear layers, etc. Each functional module is associated with one or more learnable weight matrices. For example, the multi-head attention module is associated with the output projection matrix w o Associated, the linear layer is associated with the matrix w g Associated.

[0054] Figure 2A and Figure 2B is a diagram showing an exemplary weight distribution based on the weights provided by a 12-layer full-precision GPT-2. Specifically, Figure 2A Figure 200 in Figure 200 shows the output projection matrix w in the multi-head attention module in the feed-forward network from the fourth layer of the 12-layer full-precision GPT-2 fitted by curve 210 o The distribution of , and the prediction of the limiting factor by the prior art (such as PACT 212) and the technology 214 disclosed in this article. Specifically, Figure 2B Figure 250 in FIG. 250 shows the weight matrix w in the second linear layer in the feed-forward network from the fourth layer of the 12-layer full-precision GPT-2 fitted by curve 220 g distribution, and prediction of the limiting factor by prior art (eg, PACT 222) and the technique 224 disclosed herein.

[0055] like Figure 2A and Figure 2B As shown in , the weight distribution may be highly skewed by outliers, which may cause difficulties in estimating the clipping factor α of the quantizer using existing methods such as PACT. As mentioned above, PACT learns the clipping factor α by gradient descent. For example, in PACT, the approximation of the gradient of the clipping factor α only depends on certain weights with an absolute value greater than α, i.e., |w|≥α. In other words, the solution provided by PACT ignores the influence of weights in the range [-α,α] and is heavily dependent on the initialization of α. As Figure 2B As shown, PACT has a larger clipping factor for quantitative prediction than the technique of the present invention. Figure 2AWe show that in more extreme cases, when PACT uses incorrect initialization and inaccurate gradient estimates for the clipping factors, PACT may predict clipping factors α that are too large to provide fine resolution for most weights within the clipping range. Coarse resolution may result in large quantization errors, i.e., large differences between input values ​​and quantized values. Furthermore, quantization errors may accumulate over time, such as during token-by-token sequential generation by CLM during inference, making this problem even worse.

[0056] In view of the changing weight distribution, the present invention provides a technical solution to alleviate the above-mentioned problems with the changing weight distribution. The technology disclosed in this article is called module-dependent dynamic scaling, which can be used to determine the scaling (related to the clipping factor) based on the statistical information of the weights of each module. In particular, the technology applies the following equation based on the new scaling factor γ and the average weight size To obtain the clipping factor α: Among them, ||·||1 is the L1 norm function, that is, it is used to obtain the sum of the magnitudes of each vector in the vector space, and n is the number of weights in the corresponding weight distribution. In some examples, the scaling factor γ can be initialized to 1, which not only simplifies the initialization, but also ensures that regardless of the diversity of the weight distribution, the initial clipping factor α does not deviate too much from the full-precision weight. In addition, compared with traditional methods, this technique provides a more accurate method to estimate the gradient of the scaling factor α by considering weights within the clipping range (i.e., |w|<α). In particular, the gradient with respect to γ ​​is calculated as the sum of the contributions from each weight, which is formulated as: The gradient contributed by each weight (i.e. ) is calculated as follows: in, is the total training loss and u i =clip(w i , -α, α). In one embodiment, the update of the clipping factor may be affected by both weights outside and inside [–α, α] because α controls the quantization error in two ranges. For example, a larger clipping factor may result in a smaller quantization error for weights outside [–α, α] and a larger quantization error for weights inside [-α, α]. The estimate of the gradient of the scaling factor γ in equations 3 and 4a to 4c above balances the weights outside and inside the range [–α, α]. In addition, the calculated scaling is less sensitive to changing weight distributions because the gradient of the scaling With the average weight Directly proportional.

[0057] Figure 3 300 is an exemplary process of data processing provided by one or more examples of the present invention. Process 300 may be performed by a processing system including a processing system such as Figure 1B One or more computer systems 150 shown in FIG. Figure 1A The depicted network environment 100 is implemented as one or more client devices 120, one or more servers 130, or a combination thereof. The process 300 can be performed alone or in combination with other processes in the present invention. Those skilled in the art will appreciate that the process 300 can be performed in any suitable environment, and the blocks in the process 300 can be performed in any suitable order.

[0058] At step 310 , the processing system determines a scaling factor based on a weight distribution associated with the neural network model.

[0059] In some examples, the neural network model can be a PLM having multiple functional modules. Each functional module can be associated with one or more weight matrices. The processing system can obtain the weight matrix provided by the PLM and determine the weight distribution associated with any functional module, thereby determining the scaling factor γ associated with the corresponding weight distribution. The scaling factor γ is called the module-level dynamic scaling of the corresponding functional module. In other examples, the processing system can apply Equation 2 to obtain the clipping factor α associated with the determined scaling factor γ.

[0060] In some variations, the neural network model may be a model to be trained from scratch. In this case, the processing system may determine an initial value for the scaling factor γ. For example, the scaling factor γ may be initialized to 1. Alternatively, the processing system may apply step 310 of process 300 based on the initial values ​​of the learnable weights in the model to obtain an initial value for the scaling factor γ.

[0061] At step 320, the processing system determines the quantized weights based on the weights in the distribution and the scaling factors. In particular, the processing system can apply Equation 2 to obtain the clipping factor α associated with the determined scaling factor γ. The processing system can then apply Equation 1 to the weights w in the model to obtain the quantized weights w q .

[0062] The determined clipping factor α can be combined with various techniques to facilitate the quantization of weights in the model. For illustration, in an exemplary transformer-based model of bidirectional encoder representations from transformers (BERT), the processing system can perform layer-by-layer quantization in the transformer layer and row-by-row quantization in the embedding layer, wherein each transformer layer and embedding layer may include one or more functional modules in the model. In particular, the former is to apply a clipping factor to the weights in each weight matrix for all weight matrices in the transformer layer (e.g., associated with one or more functional modules), and the latter is to apply a clipping factor to each word embedding in the corresponding embedding layer. In some cases, the processing system may use asymmetric uniform quantization for activations after self-attention (layer) and Gaussian error linear unit (GeLU) functions, the elements of which (e.g., weights) are mostly positive, while symmetric uniform quantization is used for other activations. In some variants, the processing system may not quantize the layer normalization layer, skip connection, and bias in the model because the computational overhead of these functional modules in the model is small.

[0063] At step 330, the processing system determines the training loss of the model based on the quantized weights during model training. For example, the processing system may determine the training loss of the model based on the quantized weights w during forward propagation. q Calculating training loss During back-propagation, the gradient with respect to the scaling factor γ can be calculated by applying Equations 3 and 4a to 4c This gradient can be used to update the corresponding scaling factor in the model. In some cases, the gradient with respect to the quantized weights It can be used to update the weights w in the model. In some examples, STE can be used to update the weights w based on the quantized q The gradient of is transformed into the gradient associated with the weight w. The calculated gradient associated with the weight can be used to update the learnable weights in the model.

[0064] At step 340, the processing system determines an updated scaling factor based on the gradient of the training loss. In some variations, the updated learnable weights in the model may be updated in the model during training, resulting in a change in the weight distribution. The processing system may recalculate the weight distribution in the model to determine an updated scaling factor based on the updated weights. The processing system may dynamically scale the clipping factor α based on statistical information of the weights of various modules in the model during training.

[0065] The processing system can run multiple iterations of process 300 to repeatedly perform some or all steps to train the model implemented on the processing system so that the processing system can determine the optimal scaling factor and the corresponding optimal clipping factor for the functional modules included in the model. Thus, the processing system can obtain a trained model optimized for a specific task (e.g., a downstream task). The processing system can use the trained model during inference.

[0066] As mentioned above, the compression ratio achieved by traditional quantization methods can be much smaller than that of BERT. In natural language processing (NLP), word embedding is a term used for the representation of words for text analysis, usually in the form of real-valued vectors that encode the meaning of the words so that words that are closer in the vector space are expected to be similar in meaning. Word embeddings can be obtained using a set of language modeling and feature learning techniques, in which words or phrases in the vocabulary are mapped to real vectors. The word embeddings of the full-precision model can be scattered in the vector space and are therefore distinguishable. One difficulty in quantizing generative PLMs is that the learned word embeddings may tend to be homogeneous and indistinguishable due to the reduced capacity caused by quantization. Empirical results show that the word embeddings learned by models applying common quantization methods (such as PACT and LSQ) may become homogeneous. For example, the learned word embeddings may become clustered in the vector space and therefore have low distinguishability. A direct consequence of homogeneous word embeddings is that the more homogeneous the word embeddings of the quantized model are, the less dependencies between different tokens. Furthermore, the weight distributions vary widely across modules and / or across transformer layers. These issues can be further exacerbated by the nature of sequential left-to-right predictions in generative PLMs, as quantization errors accumulate over time. For example, GPT computes each token in a left-to-right order, so quantization errors incurred in previous tokens are propagated to subsequent tokens, making the learning signal noisier over time and ultimately resulting in less informative word embeddings.

[0067] In another example, a quantization method called token-level contrastive distillation can be implemented in the quantized model to alleviate the problem of homogeneous word embeddings in the quantized model. In the present invention, sequence-level operations refer to operations performed on the input (i.e., sequence) of the neural network, and token-level operations refer to operations performed on tokens representing elements in the input (e.g., fragments of the input). A token is an instance of a sequence of characters (e.g., related to a word) in a particular document that are grouped together as useful semantic units for processing. Token-level contrastive distillation compares tokens rather than sequences (i.e., input sequences) to learn distinguishable representations for each token.

[0068] Figure 4 An exemplary training process 400 provided by one or more examples of the present invention is shown. Some or all steps of process 400 may be performed by a processing system including, for example, Figure 1B One or more computer systems 150 shown in FIG. Figure 1A The depicted network environment 100 is implemented as one or more client devices 120, one or more servers 130, or a combination thereof. The process 400 can be performed alone or in combination with other processes in the present invention. Those skilled in the art will appreciate that the process 400 can be performed in any suitable environment and in any suitable order.

[0069] As an example, a processing system may use process 400 to train a quantized model that may be compressed based on a full-precision model by applying a quantization method. For example, a scaling factor determined using process 300 may be applied to quantize learnable weights in a full-precision model to obtain a quantized model. According to an embodiment of process 400, the quantized model in training may utilize the results generated by the corresponding full-precision model to improve learning performance by performing the process demonstrated in process 400 each time iteratively.

[0070] Box 410 shows an exemplary sequence (eg, text) to be processed by the quantized model. The sequence can be parameterized into n tokens t1, t2, ..., t n , to obtain an input token sequence, as shown in box 420. The input sequence is then processed by two networks, which can operate concurrently. One network is called a teacher network, as shown in box 430, and the other network is called a student network, as shown in box 440. The teacher network may include a full-precision model associated with the quantized model. For example, the quantized model may be derived from the full-precision model. In some embodiments, the teacher network may also be a trained model on the same task where the quantized model is trained. As shown in box 430, the teacher network may include a series of layers, including an embedding layer 432, multiple transformer layers 434 to 436, and an embedding layer 438. The student network may include a quantized model to be trained. The quantized model may be similar to the model architecture in the teacher network, for example, by having the same number / type of layers. In this example, the student network has the same number of layers as the teacher network. In particular, as shown in box 440, the student network also includes a series of layers, including an embedding layer 442, multiple transformer layers 444 to 446, and an embedding layer 448. The outputs of the teacher network and the student network may be compared as shown in blocks 460 and 470 .

[0071] In this example, the superscripts s and t denote the student network and the teacher network, respectively. The length n input sequence consisting of n tokens is denoted as (t1, t2, …, t n ). For the i-th token t i , linearly projecting the hidden states of the last transformer layers 446 and 436 from the student network and the teacher network to the token representation like Figure 4 As shown, in box 440, the token memory (V b ) 450 can be coupled to the student network to store momentum token representations from the quantized network. In some examples, the token memory 450 will Save As For each token of the quantized model, the representation of the same token from the full-precision teacher network can be defined as positive (or positive sample), while the representation of other tokens in the same input sequence is defined as negative (or negative sample). i Represents a union, which includes the i-th token i The positive indices i are the ones sampled and the negative indices are the ones sampled.

[0072] The token representations from the student network and the teacher network can be compared by applying token-level contrastive distillation as shown in box 460. The goal of token-level contrastive distillation is to find the deviation of the embedding learning between the student network and the teacher network, thereby guiding the quantized model to pull tokens together with the tokens associated with these tokens in the vector space generated by the full-precision model, while pushing away other tokens in the input sequence. A legend 480 is provided to illustrate the operations associated with "pulling together" and "pull away" between token representations. The token-level student-to-teacher contrastive distillation loss is denoted as 462, which can be formulated as follows for a sequence of length n: in, Compute the cosine similarity of vectors x and y, τ is a fixed temperature parameter. In some embodiments, the memory bank can be used to store the momentum token representation of the teacher network and calculate the teacher-to-student contrastive loss To this end, the final contrast loss can be an aggregation of the contrastive losses from both sides, for example, by applying the following equation: In some embodiments, the contrastive losses from both sides may be aggregated by applying any suitable weights to compute the final contrastive loss In some examples, when computing the contrastive distillation loss, the representations of negative samples can be loaded from the corresponding memory bank by applying a low-cost indexing operation.

[0073] In some examples, the token t in the corresponding memory bank (e.g., 450) may be updated based on a moving average of the token representations from the student network. i The moving average is called a smoothing operation and is formulated as: Among them, m∈[0,1) is the momentum coefficient that controls the smoothness of the token representation.

[0074] Referring to box 470, another distillation loss can be calculated based on the logits provided by the teacher network and the student network. The logit in the language model refers to the vector of raw (i.e., non-normalized) predictions generated by the classification model. For the i-th token t i , the logits of the quantized network and the full-precision network are expressed as Where |V| is the vocabulary size. Vocabulary refers to the set of unique words used in a text corpus. The logits provided by the quantized or full-precision network are associated with the tokens generated by the corresponding network.

[0075] As shown in block 470, the logits provided by the student network and the teacher network may be compared to obtain a representation of The distillation loss is 472. In this example, each logit provided by the student network is compared to the corresponding logit in the logit provided by the teacher network. 472 can be calculated by applying the soft cross entropy loss function as follows: In this way, the total training loss It can be formulated as follows: Where λ is the balancing factor, which is set to 0.1 by default and can be adjusted via user input. and contrast loss It provides information from different perspectives, so that the student network can obtain comprehensive information from the teacher network, thereby improving the training performance of the student network. The student network can learn to obtain tokens similar to those generated by the teacher network by imitating each token based on the corresponding token generated by the teacher network. With the knowledge of the teacher network, the student network can not only learn to obtain tokens that are closer to the corresponding tokens (i.e., positive values) generated by the teacher network in the vector space, but can also further distinguish the obtained tokens from other tokens (i.e., negative values) generated by the teacher network in the vector space.

[0076] Figure 5 500 is an exemplary process of data processing provided by one or more examples of the present invention. Process 500 may be performed by a processing system including a processing system such as Figure 1B One or more computer systems 150 shown in FIG. Figure 1A The depicted network environment 100 is implemented as one or more client devices 120, one or more servers 130, or a combination thereof. The process 500 can be performed alone or in combination with other processes in the present invention. Those skilled in the art will appreciate that the process 500 can be performed in any suitable environment and in any suitable order.

[0077] In such Figure 5 In the exemplary process 500 shown, the processing system may use Figure 4 The workflow 400 shown is designed to implement the workflow. In the implemented workflow, the student network may include a quantized model, which may be quantized by applying the module-dependent dynamic scaling method of the process 300.

[0078] At step 510, the processing system determines a first set of token representations from the student network and a second set of token representations from the teacher network based on the input sequence. As shown in workflow 400, the input sequence may include a token sequence. The input sequence may be processed by the student network and the teacher network in two branches.

[0079] At step 520, the processing system determines a first loss based on a pairwise comparison between a first token representation in the first set of token representations and a second token representation in the second set of token representations. The processing system can calculate the first loss based on a token-level contrast distillation loss method as shown in workflow 400. In some examples, the processing system can calculate the first loss by applying Equation 5 and / or Equation 6.

[0080] At step 530 , the processing system determines a first set of logits from the student network and a second set of logits from the teacher network based on the input sequence.

[0081] At step 540, the processing system determines a second loss based on a pairwise comparison between each first logit in the first logit set and a corresponding second logit in the second logit set. The processing system can refer to block 470 in workflow 400 to compare the first logit set and the second logit set. In some examples, the processing system can calculate the second loss by applying Equation 8.

[0082] At step 550, the processing system determines a third loss based on the first loss and the second loss. In some variations, the processing system can aggregate the first loss and the second loss by applying appropriate weights. For example, the processing system can calculate the third loss by applying Equation 9.

[0083] At step 550, the processing system updates the student network based on the third loss. For example, the processing system can calculate the gradient of the training loss (e.g., the third loss) with respect to the quantized weights in the quantized model (i.e., the student network), which can be used to calculate the updated weights in the model during backpropagation, thereby updating the weights in the model.

[0084] In some examples, the processing system may perform all or part of the steps in process 500 through multiple iterations so that the quantization model can be fine-tuned for a specific task.

[0085] Other details and advantages related to exemplary embodiments of the present invention are discussed in “Compression of Generative Pre-trained Language Models via Quantization” by Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong (2022) (available at arxiv.org / abs / 2203.10705), the entire contents of which are incorporated herein by reference.

[0086] It should be noted that the technology described herein may be embodied in executable instructions stored in a computer-readable medium for use by a processor-based instruction execution machine, system, device or equipment or in combination with a processor-based instruction execution machine, system, device or equipment. Those skilled in the art will appreciate that for some embodiments, various types of computer-readable media may be included to store data. As used herein, "computer-readable media" includes one or more of any suitable media for storing executable instructions of a computer program, so that an instruction execution machine, system, device or equipment can read (or obtain) instructions from a computer-readable medium and execute instructions for executing the described embodiments. Suitable storage formats include one or more of electronic formats, magnetic formats, optical formats, and electromagnetic formats. A non-exhaustive list of conventional exemplary computer-readable media includes: portable computer floppy disks; random-access memory (RAM); read-only memory (ROM); erasable programmable read only memory (EPROM); flash memory devices; and optical storage devices, including portable compact discs (CDs), portable digital video discs (DVDs), and the like.

[0087] It should be understood that the arrangement of the components shown in the drawings is for illustrative purposes, and other arrangements may also exist. For example, one or more elements described herein may be implemented as electronic hardware components in whole or in part. Other elements may be implemented in software, hardware, or a combination of software and hardware. In addition, some or all of these other elements may be combined, some elements may be completely omitted, and additional components may be added while still achieving the functions described herein. Therefore, the subject matter described herein may be embodied in many different variations, and all of these variations are contemplated to be within the scope of the claims.

[0088] In order to facilitate understanding of the subject matter described herein, many aspects are described according to the order of actions. Those skilled in the art will recognize that various actions can be performed by special circuits or circuit systems, by program instructions executed by one or more processors, or by a combination of the two. The description of any sequence of actions herein is not intended to imply that the specific order described for executing the sequence must be followed. All methods / processes described herein can be performed in any suitable order, unless otherwise specified herein or clearly contradictory to the context.

[0089] The use of the terms "one" and "the" and similar references in the context of describing the subject matter (particularly in the context of the appended claims) should be interpreted as covering the singular and the plural, unless otherwise specified herein or clearly contradicted by the context. The use of the term "at least one" followed by a list of one or more items (e.g., "at least one of A and B") should be interpreted as meaning an item selected from the listed items (A or B) or any combination of two or more items in the listed items (A and B), unless otherwise specified herein or clearly contradicted by the context. In addition, the foregoing description is for illustrative purposes only, not for limiting purposes, because the scope of protection sought is defined by the claims set forth below and any equivalents thereof. The use of any and all examples or exemplary language provided herein (e.g., "for example") is only to better illustrate the subject matter and does not constitute a limitation on the scope of the subject matter, unless otherwise requested for protection. The use of the term "based on" and other similar phrases in the claims and written descriptions indicates the conditions for producing a result, and is not intended to exclude any other conditions for producing the result. Any language in the specification should not be interpreted as indicating that any non-claimed element is essential for implementing the claimed invention.

Claims

1. A computer-implemented method for quantizing a neural network model performed by a processing system, characterized in that: include: (a) determining a scaling factor based on a distribution of weights associated with the neural network model; (b) determining a quantized weight based on the scaling factor and the weight associated with the distribution; (c) determining a training loss of the neural network model based on the quantized weights during training of the neural network model; (d) determining an updated scaling factor of the neural network model based on the gradient of the training loss.

2. The method according to claim 1, characterized in that Also includes: Determining an average weight based on the distribution of the weights; determining a clipping factor as the product of the scaling factor and the average weight magnitude, Wherein, determining the quantized weight based on the weight in the distribution and the scaling factor is based on the clipping factor.

3. The method according to claim 2, characterized in that Also included is determining the average weight magnitude using an L1 norm function of the weights associated with the distribution.

4. The method according to claim 1, characterized in that: The weight associated with the distribution is divided into a plurality of value ranges based on the scaling factor, the method further comprising: calculating a gradient contribution for each of the weights associated with the distribution based on the range of values ​​within which the corresponding weight falls; The gradient of the training loss is calculated by aggregating the gradient contributions from the weights associated with the distribution.

5. The method according to claim 1, characterized in that Also includes: The initial value of the scaling factor is set to 1.

6. The method according to claim 1, characterized in that Also includes: Based on the initial values ​​of the weights in the neural network model, an initial value of the scaling factor is determined.

7. The method according to claim 1, characterized in that It also includes determining an optimized scaling factor for the task by performing multiple iterations of (a) to (d).

8. The method according to claim 1, characterized in that The updated scaling factor is associated with a weight matrix of a plurality of weight matrices in the neural network model, the method further comprising: By performing (a) to (d) on the corresponding weight matrix, an updated scaling factor is determined for each weight matrix in the other weight matrices in the plurality of weight matrices in the neural network model.

9. The method according to claim 8, characterized in that Also includes: applying the updated scaling factor to the neural network model; determining quantized weights associated with the updated scaling factors as learnable weights in the neural network model; The neural network model is updated by updating the learnable weights.

10. The method according to claim 9, characterized in that The neural network model having the quantized weights associated with the updated scaling factors is included in a student network, and the student network is trained using a teacher network, the method further comprising: Based on the input sequence, the student network obtains a first token representation set and the teacher network obtains a second token representation set; determining a first loss based on a pairwise comparison between a first token in the first token representation set and a second token in the second token representation set; determining a third loss during training of the student network based on the first loss; The student network is updated based on the third loss.

11. The method according to claim 10, characterized in that The first loss includes student-to-teacher loss and teacher-to-student loss.

12. The method according to claim 10, characterized in that Also includes: Based on the input sequence, obtaining a first logit set from the student network and a second logit set from the teacher network; determining a second loss based on pairwise comparisons between each first logit in the first set of logits and a corresponding second logit in the second set of logits; The third loss is determined based on the first loss and the second loss.

13. The method according to claim 12, characterized in that Determining the third loss based on the first loss and the second loss further includes determining the third loss by aggregating the first loss and the second loss with an adjustable factor.

14. A system for quantizing a neural network model, characterized in that: include: one or more processors; A non-transitory computer-readable medium having computer-executable instructions stored therein, which, when executed by one or more processors, cause the one or more processors to facilitate: (a) determining a scaling factor based on a distribution of weights associated with the neural network model; (b) determining a quantized weight based on the scaling factor and the weight associated with the distribution; (c) determining a training loss of the neural network model based on the quantized weights during training of the neural network model; (d) determining an updated scaling factor of the neural network model based on the gradient of the training loss.

15. The system according to claim 14, characterized in that The one or more processors further facilitate: Determining an average weight based on the distribution of the weights; determining a clipping factor as the product of the scaling factor and the average weight magnitude, Wherein, determining the quantized weight based on the weight in the distribution and the scaling factor is based on the clipping factor.

16. The system according to claim 14, characterized in that The weight associated with the distribution is divided into a plurality of value ranges based on the scaling factor, and the one or more processors further facilitate: calculating a gradient contribution for each of the weights associated with the distribution based on the range of values ​​within which the corresponding weight falls; The gradient of the training loss is calculated by aggregating the gradient contributions from the weights associated with the distribution.

17. The system according to claim 14, characterized in that The updated scaling factor is associated with a weight matrix of a plurality of weight matrices in the neural network model, wherein the one or more processors further facilitate: By performing (a) to (d) on the corresponding weight matrix, an updated scaling factor is determined for each weight matrix in the other weight matrices in the plurality of weight matrices in the neural network model.

18. The system according to claim 17, characterized in that The one or more processors further facilitate: applying the updated scaling factor to the neural network model; determining quantized weights associated with the updated scaling factors as learnable weights in the neural network model; The neural network model is updated by updating the learnable weights.

19. The system according to claim 18, characterized in that The neural network model having the quantized weights associated with the updated scaling factors is included in a student network, and the student network is trained using a teacher network, wherein the one or more processors further facilitate: Based on the input sequence, the student network obtains a first token representation set and the teacher network obtains a second token representation set; determining a first loss based on a pairwise comparison between a first token in the first token representation set and a second token in the second token representation set; Based on the input sequence, obtaining a first logit set from the student network and a second logit set from the teacher network; determining a second loss based on pairwise comparisons between each first logit in the first set of logits and a corresponding second logit in the second set of logits; determining a third loss during training of the student network based on the first loss and the second loss; The student network is updated based on the third loss.

20. A non-transitory computer readable medium, characterized in that The invention also includes computer executable instructions for quantizing a neural network model, which, when executed by one or more processors, cause the one or more processors to: (a) determining a scaling factor based on a distribution of weights associated with the neural network model; (b) determining a quantized weight based on the scaling factor and the weight associated with the distribution; (c) determining a training loss of the neural network model based on the quantized weights during training of the neural network model; (d) determining an updated scaling factor of the neural network model based on the gradient of the training loss.

Images