Method and apparatus for LLM quantization

Abstract

Broadly speaking, embodiments of the present techniques provide a method for quantizing a large language model, LLM, on and / or for a resource-constrained device, such as a smartphone. The method comprises defining a scaling vector which comprises a plurality of weight equalization parameters to equalize the set of weights and the set of activation tensors before quantization; defining range parameters which are representative of a minimum value for each activation tensor and a maximum value for each activation tensor; and obtaining a calibration dataset. The method also comprises jointly learning, using the calibration dataset, the weight equalization parameters and the range parameters by: applying the scaling vector to the set of weights and the set of activation tensors to obtain a scaled set of weights and a scaled set of activation tensors; applying weight quantization to the scaled weights of the LLM to obtain a quantized LLM having a set of quantized weights; and optimising the weight equalization parameters and the range parameters to minimise a difference between an output from the LLM and an output from the quantized LLM, while keeping the weights of the LLM and the weights of the quantized LLM fixed. The quantized weights for the quantized LLM are output to the second electronic device together with the optimised weight equalization parameters and the range parameters.

Description

Description Title of Invention :METHOD AND APPARATUS FOR LLM QUANTIZATIONTechnical Field

[0001] The present application generally relates to a method for large language model, LLM, quantization. In particular, the present application provides a method for quantizing an LLM on and / or for a resource-constrained device, such as a smartphone.Background Art

[0002] Large language models (LLMs) have markedly improved traditional language processing capabilities, paving the way for expansive applications in artificial intelligence. However, LLM deployment has high costs in terms of memory, computation, and energy, which can be prohibitive on edge devices like mobile phones due to the limited compute power and battery life available. Quantizing the models into low-bitwidth representations is a promising strategy to mitigate these costs.

[0003] Several effective quantization solutions have been developed, primarily for Graphic Processing Units (GPUs). These units are optimized for floating point computation but are hindered by memory access bottlenecks. Among these solutions, weight-only quantization converts model weights into low-bitwidth integers (e.g., 4-bit) while maintaining activations in the original 16-bit floating point format. On-the-fly weight dequantization is used at inference time to perform computations in floating-point. Weight-only quantization can preserve accuracy and reduce inference latency by avoiding memory access overheads. Unfortunately, peak-memory use, energy consumption, and latency can still be high. To address this, weight-activation quantization methods also quantize the activations and use efficient fixed-point operators, eliminating the need for on-the-fly dequantization. Yet, quantizing activations beyond 16 bits typically degrades accuracy, especially with static per- tensor statistics. Previous strategies have included quantizing activations for only certain expensive operations, e.g. matrix multiplication, or employing dynamic per-token quantization, which tends to be slow on GPUs and lacks robust hardware support on edge devices such as mobile phones.

[0004] Hardware support is another crucial practical aspect when considering on-device deployment of LLMs. For example, edge devices often have a dedicated compute unit, which is optimized for operations in a specific bitwidth (e.g. int8-int8 operations) and rely on private libraries, hence preventing the use of custom kernels. In contrast, GPUs (even edge device ones) are more flexible at the expense of lower efficiency. Thus, quantization approaches compatible with tensor compute units are particularly attractive in practice.

[0005] Notably, none of these methods provides lossless int8 activation quantization, thus failing to concurrently achieve compute efficiency, memory efficiency, and energy efficiency in on-device settings.

[0006] The applicant has therefore identified the need for a way to implement LLMs on constrained-resource devices.Summary of Invention

[0007] In a first approach of the present techniques, there is provided a computer-implemented method for quantizing, using a first electronic device, a large language model, LLM, for use on a second electronic device. The second electronic device typically has lower resources (e.g. less memory, lower computing power etc) than the first electronic device and may be considered a resource-constrained device. The LLM comprises a set of weights and a set ofactivation tensors. The method comprises: defining a scaling vector which comprises a plurality of weight equalization parameters to equalize the set of weights and the set of activation tensors before quantization; defining range parameters which are representative of a minimum value for each activation tensor and a maximum value for each activation tensor; obtaining a calibration dataset; jointly learning, using the calibration dataset, the weight equalization parameters and the range parameters by applying the scaling vector to the set of weights and the set of activation tensors to obtain a scaled set of weights and a scaled set of activation tensors; quantizing the scaled weights of the LLM to obtain a quantized LMM having a set of quantized weights and optimising the weight equalization parameters and the range parameters by minimising a difference between an output from the LLM and an output from the quantized LLM, while keeping the weights of the LLM fixed; and outputting, from the first electronic device to the second electronic device, the quantized weights for the quantized LLM together with the optimised weight equalization parameters and the optimised range parameters .

[0008] An LLM is a well-known type of computational model designed for natural language processing tasks. A full precision LLM is typically computer and memory intensive. The size of a large neural network may be reduced by quantizing the model, for example by modifying the precision of each parameter, in particular by converting weights of the model from higher precision data types to lower precision ones (e.g. 8-bit or 4-bit). A quantized LLM typically requires reduced memory and has accelerated inference. The present techniques propose a method of fine tuning the quantization process which may be carried out after the LLM has been trained to full precision using standard techniques. The calibration dataset which is used may be a limited training dataset (i.e. up to 1024 training sample) which is randomly sampled from a of text corpus from a general purpose dataset, e.g. Pile. The calibration dataset may thus be a training dataset which comprises inputs and outputs which can be used to train the weight equalization parameters and the range parameters.

[0009] Defining the scaling vector and the range parameters may comprise adding them to the diffusion ML model. For example, they may be added with initialized values and then the values are optimized by the joint learning. By adding and then training the range parameters and the weight equalization parameters, the quantized LLM may be more accurate. Optimising the range and the weight equalization parameters may be done using any suitable loss function between a first output from the unquantized (and frozen) LLM and a second output from the quantized model which has been scaled using the scaling vector. A suitable loss function may be a standard Mean Square Estimate, e.g. minimizing the 12 distance between the two outputs. To be specific, each of the first and second outputs may refer to the activation outputted from the last element (e.g. transformer block) in the LLM and quantized LLM. In other words, there is end-to-end optimisation not a layer-wise optimisation.

[0010] Moreover, the process uses integer-only quantization and avoids use of pertoken statistics. When outputting the quantized LLM only the quantized weights together with the weight equalization parameters and the range parameters need to be output to the second electronic device to be stored thereon. The activation tensors are generated at inference when using the LLM and need not be stored on the second electronic device. Thus, the quantization of the activation tensors only happens at interference. Sending only the quantized weights reduces the size of the LLM which is stored. The weight equalization parameters and the range parameters are typically small in size compared to the weights (quantized or unquantized). For example, the weight equalization parameters may be two 1-D vectors for each 2-D weight in the per-channel quantization for each tensor and the range parameters may be two scalars for each tensor. The weight equalization parameters and the range parameters may be further reduced in size by fusing them together in the inference stages.

[0011] The first electronic device may be a server or similar device with high resources (e.g. high memory, high processing power etc). The second electronic device may be a userdevice and may have lower resources than the first electronic device. The first and second devices may be located remotely (i.e. separately) from one another. In other words, more generally the present techniques may be considered to be a computer-implemented method for quantizing a large language model, LLM, for (and / or on) resource-constrained devices, the method comprising: applying per-tensor / channel weight quantization to weights of the LLM; and applying per-tensor activation quantization to activation tensors of the LLM. The method may further comprise jointly learning a range of the activation tensors of the LLM and the weight quantization to obtain learned ranges and learned weight quantization.

[0012] The LLM may be a neural network comprising a plurality of linear layers which involve a matrix multiplication between a 2-dimensional weight tensor (e.g. weight matrix) and a 1 -dimensional activation tensor (e.g. the input activation values for each layer). In this case, applying the scaling vector to the set of weights and the set of activation tensors to obtain a scaled set of weights and a scaled set of activation tensors may be defined mathematically, for each layer, asY = XW = (XS-1) • (SW) = XW where W is the weight matrix (or weight tensor for the layer) which is part of the set of weights and W E IRWXM, X is the activation tensor for the layer and is one of the set of activation tensors and X E , Y is the output Y E IRM, S E is the scaling vector, X is the scaled activation tensor for the layer (i.e. one of the scaled set of activation tensors), W is the scaled weight tensor for the layer (i.e. one of the scaled set of weights), N is the number of input channels and M is the number of output channel . Learning the scaling vector S (and hence learning the weight equalization parameters) means a scaling vector S which when applied to the weight matrix and activation tensor results in scaled weight matrix of each linear layer as well as scaled activation tensor of each linear layer which are easier to quantize. In other words, the scaling vector may act on both the set of weights and the set of activation tensors. It will be appreciated that the overall output of the scaled LLM model will depend both on the scaled set of weights and the scaled set of activation tensors.

[0013] Each activation tensor may have or be represented by a plurality of channels. A channel typically represents a specific feature map produced by applying filters to the input data. Channels organize the learned features along a specific dimension, allowing the network to capture diverse patterns and representations of the input data across different channels. For example, X may be used to denote the activation tensor and Xwhere N is the number of channels. The method comprises defining a scaling vector which may also be termed a smoothing vector. Defining the scaling vector may comprises defining, for each channel, a weight equalization parameter. Each weight equalization parameter may be initialized using a maximum value of an activation for the channel, a maximum value of a weight for the channel and a hyperparameter. The scaling vector may be initialized as S aswhere are the activation tensor values for channel i, Wj are the weight values for channel i and a is the hyperparameter. A hyperparameter may be defined as a parameter that can be set in order to define any configurable part of a model’s learning process.

[0014] Quantizing the scaled weights may also be termed applying weight quantization. Applying the weight quantization may comprise applying the weight quantization to consecutive layers of the LLM with linear components (i.e. to linear layers), while not applying weight quantization to non-linear components. The scaling vector may comprise weight equalization parameters for each layer of the LLM with linear components but not any parameters for the non-linear layers. This is because there are no trainable parameters in nonlinear layers. A linear layer may also be termed a fully-connected layer in which every input neuron in a first layer is connected to every output neuron in a consecutive layer.

[0015] Defining range parameters may comprise defining a scale parameter and an offset parameter, wherein the scale parameter may be a scaled difference between the maximum and minimum values and the offset parameter may be a value which translates a range defined by the scaled maximum and minimum values to be symmetric about zero. Such parameters may be used for GEneral Matrix to Matrix Multiplication (GEMM) at inference time. A target quantization level for the set of activation tensors may also be defined or obtained, for example based on the capabilities of the processor (e.g. NPU) which will use the quantized LLM at inference time. The range parameters are primarily used to optimise the quantization of the activations during inference and thus are not dependent on the quantization of the weights. Thus the scale parameter is different to the scale vector described above. Where used, the range parameters may be defined using the target quantization level. The range parameters may be mathematically defined as:where fmin, fmaxare the minimum and maximum values for each activation tensor, a, / 3 E IR are the scale and offset parameters and= 2>™ where bw is the target quantization level expressed as a targeted bit-width. In this way, the minimum value for each activation tensor may be represented by a product of the scale parameter and the offset parameter. Similarly, the maximum value for each activation tensor may be represented by the product of the scale parameter with a sum of the offset parameter and a value based on the target quantization level. Other representations may be used, for example the scale parameters may simply be fmin, fmaxthe minimum and maximum values for each activation tensor, although these may be harder to optimise. The range parameters may thus be the minimum value and the maximum value for each activation tensor or may be parameters which represent the minimum value and the maximum value such as the scale parameter and range parameter which are mathematically linked to the minimum and maximum values using the target quantization level. For example, the scale parameter may be a scaled difference which is scaled using the target quantization level. The method may further comprise using the learned ranges to remove activation tensor outliers.

[0016] The method may further comprise defining weight clipping parameters for example as defined in Omni quant. Where weight clipping parameters are used, the method may further comprise jointly learning, using the calibration dataset, the weight equalization parameters, the weight clipping parameters and the range parameters. Weight clipping parameters are used to modulate extreme values of weights by optimizing the clipping threshold.

[0017] As detailed above, joint learning may comprise using a loss function. Joint learning may be an end-to-end optimisation rather than a layer-wise optimisation.

[0018] Applying per-tensor / channel weight quantization to weights of the LLM may comprise applying per-tensor / channel weight quantization at 4 bits or 8 bits. In other words, quantizing weights of the LLM may comprise reducing a size of the weights to 4 bits or 8 bits. Applying per-tensor activation quantization to activation tensors of the LLM may comprise applying per-tensor activation quantization at 8 bits or 16 bits. In other words, when using the quantized LLM, e.g. to generate an output during the jointly learning step, the method may further comprise reducing a size of each activation value in the activation tensors. The size may be reduced to 8 bits or 16 bits. Applying the weight and / or activation quantization may comprise using fixed-point integer representations. Fixed point representation has fixed number of bits for an integer part and for a fractional part of the represented value. Floating point representation do not reserve a specific number of bits for the integer part or the fractional part. Instead floating point representations reserves a certain number of bits for the number(called the mantissa or significand) and a certain number of bits to say where within that number the decimal place sits (called the exponent).

[0019] In a second approach of the present techniques, there is provided a computer- implemented method for generating an output using a large language model, LLM, which has been quantized as described above, the method comprising receiving an input; processing the received input using the quantized LLM to generate an output; and outputting the generated output. The input may be a note to be summarized, an email to be rephrased or a question to a chatbot. Similarly, the output may be a summary of the input note, a rephrased email or an answer to the question posed.

[0020] In a third approach of the present techniques, there is provided an electronic device for quantizing a large language model, LLM, the electronic device comprising: at least one processor coupled to memory for: applying per-tensor / channel weight quantization to weights of the LLM; and applying per-tensor activation quantization to activation tensors of the LLM. In other words, the at least one processor may be for defining a scaling vector which comprises a plurality of weight equalization parameters to equalize the set of weights and the set of activation tensors before quantization; defining range parameters which are representative of a minimum value for each activation tensor and a maximum value for each activation tensor; obtaining a calibration dataset; jointly learning, using the calibration dataset, the weight equalization parameters and the range parameters; by applying the scaling vector to the set of weights and the set of activation tensors to obtain a scaled set of weight and a scaled set of activation tensors; quantizing the scaled weights of the LLM to obtain a quantized LLM having a set of quantized weights and optimising the weight equalization parameters and the range parameters to minimise a difference between an output from the LLM and an output from the quantized LLM, while keeping the weights of the LLM fixed; and outputting to a user electronic device the quantized weights for the quantized LLM together with the optimised weight equalization parameters and the range parameters. The electronic device may also be termed an apparatus (and the terms may be used interchangeably). It will be appreciated that the at least one processor may individually or collectively carry out the steps above.

[0021] The features described above with respect to the first approach apply equally to the third approach and therefore, are not repeated.

[0022] In a fourth approach of the present techniques, there is provided an electronic device for using a quantized large language model, LLM, which has been quantized using the methods described herein. Such an electronic device may be termed a user electronic device or user device. The user device may comprise at least one processor, for example a neural processing unit. The at least one processor may be configured to process a quantized LLM which has weight quantization at 4 bits or 8 bits. The at least one processor may be configured to process a quantized LLM which has activation quantization at 8 bits or 16 bits. In other words, the quantization is adapted to the hardware of the user device. For example, the processor may be configured to process a quantized LLM which has weight quantization at 8 bits using 8-bit quantization for each activation value in the activation tensors or a quantized LLM which has weight quantization at 4 bits using 8-bit quantization for each activation value in the activation tensors. The activations may have a higher bitwidth than the weights because only the weights are stored on the user device. The activations are not stored and are only generated at inference, i.e. when the model is being used to process an input.

[0023] The user device may be a constrained-resource device, but which has the minimum hardware capabilities to use an LLM. The device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuumcleaner, smart oven, or smart lawn mower). It will be understood that this is a non-exhaustive and non-limiting list of example devices.

[0024] In a related approach of the present techniques, there is provided a computer- readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[0025] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0026] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[0027] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high- level compiled or interpreted language constructs.

[0028] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[0029] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD- ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[0030] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[0031] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[0032] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device, using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[0033] As mentioned above, the present techniques may be implemented using an Al model. A function associated with Al may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[0034] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, transformers, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[0035] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, self-supervised learning, semisupervised learning, or reinforcement learning.Brief Description of Drawings

[0036] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0037] Figure 1 is a flowchart of example steps to quantize an LLM on a resource- constrained device;

[0038] Figure 2 is a flowchart showing more detail of the steps in Figure 1;

[0039] Figure 3a illustrates a model comprising three layers;

[0040] Figure 3b illustrates the layer by layer application of the scaling vector of the weight equalization transformation to the model of Figure 3 a;

[0041] Figure 4 is a block diagram of a system comprising an electronic device that may be used to perform the method of Figures 1 and 2 and a user device that may use a quantized model resulting from the method of Figures 1 and 2; and

[0042] Figure 5 is a flowchart showing the steps when using the quantized model.Description of Embodiments

[0043] Broadly speaking, embodiments of the present techniques provide a method for quantizing a large language model, LLM, on and / or for a resource-constrained device, such as a smartphone.

[0044] Large language models (LLMs) have revolutionized language processing, delivering outstanding results across multiple applications. However, the deployment and operation of LLMs pose significant challenges in terms of memory, energy, and compute costs, particularly on edge devices such as mobile phones. Lower bitwidth quantization has emerged as a promising solution to mitigate these issues. Existing methods successfully quantize weights into 4 bits, offering large memory storage savings. Instead, quantizing activations beyond intl6 leads to accuracy degradation, leading to energy and latency inefficiencies.

[0045] The present techniques (MobileQLLM / MobileQuant) are a mobile-friendly quantization approach for LLMs that results in simultaneous computational efficiency, memory efficiency, and energy efficiency without compromising the accuracy of full-precision models on mobile devices. Furthermore, the present techniques explore a quantization paradigm that benefits from robust hardware support in mobile devices. Specifically, it applies per-tensor / channel weight quantization at 4 or 8 bits and per-tensor activation quantization at 8 or 16 bits, utilizing fixed-point integer representations. These settings are adopted for several reasons: (i) per-tensor / channel integer quantization is well -supported by mobile processors, (ii) integer computations enhance inference speed and reduce energy usage compared to floatingpoint computations, and (iii) the energy consumption of multiply-accumulate operations correlates directly with the activation and weight bit-widths.

[0046] Method

[0047] Figure l is a flowchart summarizing of example steps to quantize an LLM on a resource-constrained device. The method comprises obtaining a fully trained LLM at step SI 00. The method may further comprise jointly learning additional parameters to improve the quantization of the obtained LLM at step SI 02. The parameters may be a range of the activation tensors of the LLM and the weight quantization parameters. Once the joint learning is completed, the method comprises applying per-tensor / channel weight quantization to weights of the LLM (step SI 04) together with the optimised additional parameters. At inference, applying per-tensor activation quantization to activation tensors of the LLM will also occur (step SI 06).

[0048] Applying, at step SI 04, per-tensor / channel weight quantization to weights of the LLM may comprise applying per-tensor / channel weight quantization at 4 bits or 8 bits, i.e. reducing the weights to 4 bits or 8 bits. Applying, at step SI 06, per-tensor activation quantization to activation tensors of the LLM may comprise applying per-tensor activation quantization at 8 bits or 16 bits, i.e. reducing the activation values to 8 bits or 16 bits.

[0049] Applying the weight and / or activation quantization may comprise using fixed-point integer representations. Applying the weight quantization at step SI 04 may comprise applying the weight quantization to consecutive layers of the LLM with linear components, while not applying weight quantization to non-linear components. In the present case, the LLM has no trainable weights for the non-linear components. The method as may further comprise usingthe learned ranges to remove activation tensor outliers i.e. activations with extremely low or high magnitude.

[0050] Different quantization approaches require different levels of hardware supports across the number representations (i.e. fixed points vs floating points), the granularity of quantized groups (i.e. per-tensor vs per-token), and the implementations (i.e. quantizing with static vs dynamic ranges). The settings differ in their compute, memory, and energy requirements, which are crucial for on-device deployments. State-of-the-art quantization methods demonstrate strong performance on a server use case (i.e. high-end GPU). However, they either utilize on-the-fly dequantization and full-precision operations, which are computationally inefficient, or dynamic per-token quantization, i.e. the quantization thresholds depend dynamically on the test-time activation tensor, which has no support on edge devices.

[0051] The present techniques consider mobile-friendly settings that are widely supported on modem edge devices: i) fixed point weight and activation quantization with integer arithmetic operations; ii) per-tensor / channel quantization with static pre-computed ranges.

[0052] Figure 2 is a flowchart showing more detail of the method in Figure 1. As shown in the first step S200, a full precision LLM is obtained together with public text data. The full precision LLM has been trained priort to using the present techniques. The full-precision LLM may be any suitable model, for example TinyLlaMA described in “TinyLlaMA: An Opensource small language model” by Zhang et al published in arXiv 2401.02385 in 2024, StableLM described in “Stable LM 2 1.6b technical report” by Bellagente et al published in arXiv 2402.17834 or Gemma2B described in “Gemma: Open models based on Gemini research and technology” published by Google in 2024 in arXiv 2403.08295.

[0053] At step S202, a weight equalization transformation (which may also be termed a weight initial transformation) is defined and which builds on top of previous state-of-the-art approaches, SmoothQuant and Omniquant. SmoothQuant is described for example in “SmoothQuant: Accurate and efficient post-training quantization for large language models” by Xiao et al published in International Conference on Machined learning in 2023. Omni quant is described for example in “Omniquant: Omnidirectionally calibrated quantization for large language models” by Shao et al published in International Conference on Learning Representations in 2024. The present techniques aiming to adapt the prior art techniques to the mobile-friendly setting by staying within the limits of hardware support on device. Here, a brief review of the weight equivalent transformation proposed in these works is provided.

[0054] Weight Equivalent Transformation

[0055] Prior efforts on LLM quantization observed that compared to the model weights, activations are harder to quantize due to the outlier channel dimensions with diverse min-max ranges. Figure 3a illustrates a model comprising three consecutive layers: one normalization layer 300 (for example as described in “Layer Normalization” by Ba et al published on arXiv 1607.06450 in 2016, and two linear layers 302, 304. As an example, given a fully connected layer:Y = XW, where W is the weight matrix and W 6 IRWxM, X is the activation tensor and X 6, Y is the output Y E IRM. It will be appreciated that specific channel dimensions {i: 0 < i < N} in X may have a wide min-max range across different data samples, causing large per-tensor quantization errors. W - the weight matrix is a 2-D matrix and X - the activation tensor is a I D vector. The specific mathematical definitions for each layer are shown in Figure 3a. In the example of Figure 3 a, all layers are assumed to have the same hidden dimension N.

[0056] Previous methods proposed a weight equivalent transform defined by a scaling vector SY = XW = (XS-1) • (SW) = XW (1)where X. Y.W are as defined above. The scaling vector may be termed a smoothing vector or smoothing factor. S may be a diagonal matrix. An alternative expression for equation (1) which uses element-wise multiplication is shown below. Such an expression would avoid the use of a diagonal matrix for S.Y = XW = (x©s-1)(s©w) = XW where X. Y.W are as defined above, © denotes the element-wise multiplication.

[0057] The goal is to find the optimal scaling vector S such that both X and W are easier to quantize compared to the original X and W. The new model is mathematicaly equivalent to the original model when unquantized. The layer by layer application of the scaling vector is shown in Figure 3b. Thus, more specifically, the goal of weight transformation is to learn a scaling vector S such that the resulting weight matrices SoW^ and StW2of each linear layer as well as the activations X1(X2of each linear layer are easier to quantize. The parameters of the scaling vector are thus the only learnable parameters.

[0058] SmoothQuant reparameterized S aswhere a is the hyper-parameter which is searched for. The obtained S is similar to a closed- form solution. It is noted that the total effective quantization bits would be largest when all the channels have the same maximum magnitude. Thus, = max(|X ) would be a straightforward choice for the smoothing factor because this choice ensures that after the division, all the activation channels will have the same maximum value, which is easy to quantize. However, this formula pushes all the quantization difficulties to the weights. On the 1 other hand, sL= - — — would push all the quantization difficulty to the activations. Thus,1Tl(ZX(J Wjl) the reparametrization proposed in SmoothQuant introduces the hyper-parameter a to control how much difficulty is to be migrated from activation to weights. The formula ensures that the weights and activations at the corresponding channel share a similar maximum value, thus sharing the same quantization difficulty. Merely as an example, as an example, an original XW:Becomes XW when a = 0.5,:

[0059] As shown at step S204, each weight equalization parameter is initialized. This may be done using a maximum value of an activation for the channel, a maximum value of a weight for the channel and a hyperparameter, for example using the expression in SmoothQuant noted above.

[0060] Omni quant extended SmoothQuant by learning S, together with the weight clipping paramters via block-wise error minimization. The weight clipping parameters may be used to reduce the difficulty of quantizing the weights in an LLM. The weight clipping parameters y, / 3 may be defined by / WWg= clamp — + z, 0, 2bw—\L n .where Wqare the quantized weights, W are the full-precision weights, bw is the target bitwidth, [ J denotes a round operation, h is the normalisation factors for weights and z is the zeropoint value, wherey max h = -andThe clamp operation constrains the value with the range of bw -bit integeer, specifically [0, 2hw- 1], Omniquant also teaches that both S and S-1can be fused to the adjacent linear layers, making the tranformation mathematically equivalent to the original models.

[0061] The weight equivalent transformation approaches used in SmoothQuant and Omni Quant demonstrate strong performance on GPU-like hardware. However, two challenges remain for edge devices. Firstly, the re-scaling based transformations cannot propogate beyond non-linear operators, i.e. softmax, rmsnorm, layernorm, etc. To resolve this, as in previous approaches, the present techniques only apply weight transformations along consecutive layers with linear components, i.e. between linear layers or affine transformations in the normalization layers, while keeping the nonlinear activations in 16-bit integers. Thus, as small a set of transformation parameters for all possible chains of linear layers are defined to ease weight quantization. These transformations are linear-invariant.

[0062] Secondly, with the weight transformation, the distribution of the activations shifts accordingly. This causes essential difficulty for learning-based approaches like OmniQuant, when the min-max range for the activations change after each training iteration. OmniQuant proposed to bypass the issue with dynamic per-token quantization, which has no hardware support on-device. Thus, another solution is required and this is shown in steps S212 and S214 which are shown as being done in parallel to the steps associated with the definition of the weight equivalent transformation steps S202, S204 but could be done sequentially.

[0063] Learning the Per-tensor Range of the Activations

[0064] Fixing the min-max ranges for the activations during training is suboptimal. By learning the weight clipping and equalization, the distribution of the activations shift accordingly. Ideally, the quantization process should learn to adapt to the new activation distribution. And the ideal solution is to re-estimate the activation ranges across the training set after each training iteration. Computing the min-max range for each activation across the whole training set for each training iteration is prohibited and typically not differentiable. Accordingly, as shown at step S212, the set of rangesmin(X), fmax( ) for all activations is initialized using a calibration set. Merely as an example, the calibration set may be a subset of the Pile dataset which is described in “The Pile: An 800GB dataset of diverse text for language modelling”” by Gao et al published in arXiv 2101.0027 in 2022. The aim is to learn a minimum and maximum value for both weight and activation which are similar across each channel.

[0065] Given an activation tensor X, instead of learning the min and max values fmin( ) , fmax(^) directly, the correlation between fmin, fmaxand the scale and offset parameters, a, ft e IR, is leveraged for quantization. With the targeted bit-width bw, quantizing X can be formulated as:Here, Xintrefers to the quantized tensor of X, ste refers to straight-through estimator. It is therefore possible to learn fmin= -aft and fmax= aqmax- aft indirectly by learning a and ft, which are computationally more stable. In othe words at step S214, the ranges fmjn(X)7 / max(X) are reparameterised and may be termed range parameters. The learned ranges could potentially also help remove the activation outliers that contribute little to the performance but causes large quantization errors. Using a and ft adds parameters to the training but there areonly two parameters per activation and thus the increase in parameters is neglible. During the inference, a and can be fused into the GEMM operation with no runtime overhead.

[0066] Holistic Optimization vs Layer-wise Optimization

[0067] To learn the equivalent transformation, previous works either resort to closed form solutions, search-based methods (such as SmoothQuant), or layer-wise error minimization (such as OmniQuant). These solutions require limited training budget, but, as shown in the experiments, lead to sub-optimal performance. Particularly, given the restricted form of supervisions, these methods cannot scale with more training samples or iterations. As shown in Figure 2, at step S220, the present techniques propose to jointly optimize all the training parameters, including the weight equalization parameters S and the range parameters a, / 3 , and optionally weight clipping paramaters y, <J, for all layers in an end-to-end manner. In other words, a single loss function is used at the final layer. Optimising all the training parameters may be done using any suitable loss function between a first output from the unquantized (and frozen) LLM and a second output from the quantized model which has been scaled using the scaling vector. A suitable loss function may be a standard Mean Square Estimate, e.g. minimizing the 12 distance between the two outputs. To be specific, each of the first and second outputs may refer to the activation outputted from the last element (e.g. transformer block) in the LLM and quantized LLM. The weights of the LLM are frozen during joint training of the other parameters.

[0068] There is a check to see if the model is converged at step S222 and if not, the method loops back to the joint optimisation step. If the model is converged, the quantized weights of the quantized LLM can be output at step S224 together with the optimised parameters (e.g. range parameters, weight equalization parameters and optionally weight clipping parameters). The output is then stored on a user device at step S226 and can be used at inference.

[0069] The quantization applied may be any suitable quantization, particularly one which is widely supported and optimized for modem edge devices, e.g. Mobile NPUs. Such quantization techniques typically use fixed-point weight and activation quantization with integer arithmetic operations and per tensor / channel quantization with static pre-computed ranges. Two well-supported examples are:• W8A8: 8-bit per-tensor or per-channel quantization for the weights, 8-bit per-tensor quantization for the activations, except those linked to the non-linear operators like softmax and non-linearities.• W4A8: 4-bit per-channel quantization for model weights, and 8-bit per-tensor quantization, again excluding non-linear operators.

[0070] Thus, as an example, a combination of per-tensor and per-channel weight quantization at 4-bit or 8-bit may be applied together with per-tensor activation quantization at 8-bit or 16-bit, utilizing fixed-point integer representations for all operations. These quantizations are mobile friendly bit-widths with int8-int8 operations typically widely supported and most often optimized for. Other options are int4-intl6 and int8-intl6 but these are typically slower than int8-int8. It is also noted that quantizing using per-tensor and per- channel statistics is widely supported while using per-token statistics is not. Accordingly, the present techniques avoid using per-token statistics. The above techniques also use static quantization statistics which does not depend on the input data, typically, computed on a holdout calibration set as noted above. Dynamic quantization on the other hand requires online calibration from the input data and is not supported.

[0071] It is shown that compared to previous approaches that struggle with more training samples (> 128) and epochs (> 20), the present holistic optimization consistently improves the performance with larger training settings for different LLM architectures. Alsonote that, as the learned model is mathematically equivalent to the original full-precision model when quantization is not applied, the training will not overfit to specific calibration samples, achieving near-lossless zero-shot performance as shown in the experiments.

[0072] Architecture

[0073] Figure 4 is an example system for implementing the methods described above. A first electronic device 400 is used to implement the training method described above. The electronic device 400 comprises: at least one processor 402 coupled to a memory 404. The at least one processor 402 may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. The at least one processor 402 may include one or more central processing units (CPUs) and / or one or more graphics processing units (GPUs). The memory 404 may comprise volatile memory, such as random access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example. The electronic device may be any suitable system or device, e.g. a server, for example two Al 00 GPUs.

[0074] The electronic device 400 comprises the full ML model 410 which may be trained on device using the training data in the dataset 420 or may have been pre-trained on another device. The dataset 420 is shown as external to the electronic device 400 but may be stored within memory on the device. There is also an input / output interface for receiving data, e.g. from the dataset 420 or for transmitting information to a user device 450. The LLM 410 comprises weights and activation tensors.

[0075] The electronic device 400 comprises the per-channel scaling vector 412 for each linear layer of the full ML model 410 as well as the scaling and offset scalars 414 for each activation tensor. The vectors and scalars are learnt as described above. The quantized model 416 may also be stored on the electronic device 400 or may be stored on the user device 450 to which it is deployed. On the user device, the quantized model 460 is stored with the per-channel scaling vector 462 (i.e. the weight equalization parameters) and the scaling and offset scalars 464 (i.e. the range parameters). The quantized LLM may achieve near-lossless accuracy, compared to the original model. The quantized LLM may achieve lower latency, energy consumption and storage / memory footprint when compared to the original model. The quantized LLM may be deployed on edge devices (also termed user devices) to support on device generative Al (gen-AI) applications.

[0076] The user device may also comprise the standard components such as processor(s) 452 (e.g. CPUs, GPUs and Neural Processing units (NPUs), memory 454 and an I / O interface 458. The user device which receives the lightweight model, may be a constrained-resource device. The device may be any one of: a smartphone, tablet, laptop, computer or computing device, virtual assistant device, a vehicle, an autonomous vehicle, a robot or robotic device, a robotic assistant, image capture system or device, an augmented reality system or device, a virtual reality system or device, a gaming system, an Internet of Things device, or a smart consumer device (such as a smart fridge, smart vacuum cleaner, smart oven, or smart lawn mower). For example, the user device may be a Samsung Galaxy S24 with the Snapdragon 8 Gen HTP as the compute model. It will be understood that this is a non-exhaustive and nonlimiting list of example devices.

[0077] Figure 5 shows the steps which are carried out at inference time on the user device to which the quantized model has been deployed. For example, at step S500, the user device receives an input to be processed by the quantized model. At step S502, the input is processed by the quantized model, i.e. using the quantized weights together with the optimised weight equalization parameters and range parameters. It is at this stage, that the activation quantization will take place. At step S504, the output is generated. The input may be a note to be summarized, an email to be rephrased or a question to a chatbot. Similarly, the output maybe a summary of the input note, a rephrased email or an answer to the question posed. Merely, as an example, the original email may be input as:Subject: Meeting Reschedule RequestHi John,I hope you’re doing well. I’m writing to let you know that I have a scheduling conflict and need to reschedule our meeting. Can we move it to Thursday at 2 PM instead? Let me know if that works for you.Thanks,Jane

[0078] The output may be:

[0079] For a chatbot, there may be multiple sequential inputs and outputs, for example:User: Hi, I need help with my order.Chatbot: Hello! I’d be happy to assist you. Could you please provide your order number? User: Sure, it's #12345.Chatbot: Thank you. Let me check the status of your order. One moment, please.Chatbot: Your order#12345 was shipped on July 15th and is expected to be delivered by July 20th. Is there anything else I can help you with?User: Yes, I received the wrong item. What should I do?Chatbot: I apologize for the inconvenience. To resolve this, you can request a return or exchange. Would you like me to guide you through the process?User: Yes, please.

[0080] Experiments

[0081] The state-of-the-art weight-activation methods, Smooth Quant and OmniQuant, demonstrate strong performance on a server use case (i.e. high-end GPU). They utilize dynamic per-token quantization of the activations, i.e. the quantization thresholds depend dynamically on the test-time activation tensor. This however has no support on edge devices. Hence, the present applicant starts by adapting these methods to the mobile-friendly setting. Two well- supported adaptations are explored. A first is a static per-tensor activation quantization and these variants are referred to as OmniQuant-static and SmoothQuant static respectively. For SmoothQuant only evaluations on W8A8 are shown because this is the default setting using the original work.

[0082] Both of these variants, suffer from large performance degradation when evaluated on WikiText which is described in “Pointer sentinel mixture models” by Merrity et al published in CoRR in 2016. This performance drop appears to be caused by quantizing the activations for the last linear layer in each MLP head. Accordingly, a second adaptation in which an extra weight equalization transformation between consecutive linear layers in each MLP head (i.e. S between the up_proj and down proj layeres in TinyLLaMA) is introduced. The second adaptations are labelled SmoothQuant-edge and Omni Quant-edge respectively and have significantly better performance than the first variation.

[0083] The variants Omni Quant-static, SmoothQuant static, SmoothQuant-edge and OmniQuant-edge are compared with two variants of the process described above which is termed MobileQuant. The experiments are done by training and simulating the quantization on GPUs and further evaluating the on-device performance on a Samsung Galaxy S24, with the Snapdragon 8 Gen 3 HTP as the compute unit. All models are trained on two A100 GPUs with a maximum sequence length of 2048.

[0084] Table 1 below compares the various variants. The best performance is shown in bold and this is with MobileQuant, the method described above. The second best peformance is underlined. The gain of the proposed approached is also indicated when compared to the next strongest on device baseline. Thus, the proposed method demonstrates consistent improvements across model, quantization configurations and tasks

[0085] The table above shows the results for WikiText and similar results can also be obtained using other benchmarks such as ARC-challenge, HellaSwag and MMLU which are all described in the Harness benchmark which is described in “A framework for few-shot language model evaluation” by Gao et al published in 2024 in Zenodo. Thus, the proposed method improves the quantization performance for the common sense reasoning tasks in general, without using any in-domain data. The improvements are consistent for most benchmarks

[0086] In a further comparison, we use LAMBADA described in “The LAMBADA dataset: Word prediction requiring a broad discourse context” by Papemo et al published in Annual Meeting of the Association for Computational Linguistics” in 2016. The proposed method is compared with the full precision model together with quantization of the TinyLLaMA model at W8A16 and full W8A8 in the table below:

[0087] Lambada predicts the last token of a sentence given the previous context and in the test, we use the first 1000 samples for this task. The table above shows that using 16-buit activations (e.g. W8A16) achieves lossless performance. However, quantization all activations into 8-bit leads to near zero performance, highlighting the difficulty of activation quantization. The proposed MobileQuant model achieves near lossless performance in both tasks, approaching the performance of the full precision model (FP16). As above, the tasks were run on a Snapdragon 8 Gen 3 HTP processor.

[0088] In a further comparison, on-device latency is evaluated by running the quantized model in two modes. In the first mode, there is prompt encoding with a context length of 256 and in the second mode, there is auto-regressive generation with a maximum sequence length of 1024 and 2048. The results are shown in the table below:

[0089] As shown in the table above, for prompt encoding, using lower-bitwidth activations is critical to reducing the inference latency, as some of the operations, e.g. selfattention (batched matrix multiplication), are compute-intensive. Our model demonstrates significant advantages over the full W8A16 solution, reducing the latency by 40%. However, there is still a large gap between MobileQuant and the full W8A8 model, indicating the improvement margin. For auto-regressive generation, the latency gaps are smaller. This may be because the auto-regressive generation is not as compute-bound as prompt encoding, especially for lightweight models, but instead is partially memory access-bound. The MobileQuant method demonstrates a 20% latency reduction compared to W8A16, achieving the same latency as the full W8A8 model. In general, the advantage of low bitwidth activations correlates strongly with the scale of the computation. The comparison above can also be repeated for a larger model such as Gemma-2B and the improvements are more significant, i.e. reducing the latency of prompt encoding and autoregressive generation by 39% and 33%.

[0090] The table above also shows that energy consumption is another important aspect of on-device execution, which is often overlooked by quantization research. To measure the energy requirements of different models, the models are run on a number of identical mobile phones as used before continuously for 30 minutes. The phones are connected to the testing host machines via WiFi using an internal network without access to the internet, to avoid any undesired network activity. The phones are also not being charged and their screens are turnedoff. All phones begin each test at the same battery level and the final energy of running a model is calculated as the ratio of the total battery discharged over the duration of a test, minus reference discharge of a phone not running any model, divided by the number of times the model was run. The measurements are repeated three times, rotating the phones each time, and the average is reported. Peak memory required to run a model is recorded by the Linux Kernel (the so-called Virtual Memory High Water Mark).

[0091] In summary, measurements of latency, energy and memory are computed under sustained execution (30 minutes). Values are reported per single forward pass. As shown in the table, energy consumption aligns well with the latency. Compared to W8A16, MobileQuant reduces 50% of the power usage for prompt encoding and 35% for autoregressive generation. The peak memory usage for all models are similar as it is dominated by the model weight.

[0092] Further on-device evaluation for W4A8 is also presented. In a first variant, symmetric per-channel quantization is used. In symmetric quantization, the range of the original floating-point values is mapped to a symmetric range around zero in the quantized space. In a second variant, asymmetric quantization is used. In asymmetric quantization, the minimum and maximum values from the float range fmin, fmaxare mapped to the minimum and maximum values in the quantized space. This may shift the location of zero. The table below presents the results confirming the performance degradation compared to asymmetric counterparts.Summary

[0093] The present applicant introduces MobileQLLM (also termed MobileQuant), the first LLM quantization method that supports int8 activations without accuracy degradation, offering simultaneous efficiency across the memory, energy (25% energy cost reduction), and latency (50% faster execution on a mobile phone) axis. MobileQLLM advances previous efforts on learning equivalent transformation with a specific focus on activation quantization. The performance shortcomings of previous methods are analyzed and assessed in on-device applications, and a simple yet effective improvement is proposed. Furthermore, a comprehensive approach is proposed to learn equivalent transformations alongside other quantization parameters. The resulting model not only outperforms existing methods but also, for the first time, demonstrates superior on-device performance in terms of model accuracy, inference latency, and energy consumption, compared to using 16-bit activations in full.

[0094] MobileQLLM builds on the state-of-the-art Omniquant framework, but with a distinct focus on per-tensor activation quantization. To further reduce the performance gap between the quantized and full-precision models, learning the equivalent transformation is proposed, as in Omni quant, together with the other quantization parameters, in a holistic setting. Particularly, instead of resorting to closed-form solutions, search-based optimization, and block- wise error minimization, MobileQLLM jointly optimizes all smoothing parameters for equivalent transformation, as well as the range parameters end-to-end. Unlike prior methods that struggle to scale with increased samples and training steps, MobileQLLM benefits from more calibration samples and extended training periods. The present methodology enables the quantization of all activations to 8-bit integers, except for less than 1% linked to non-linearoperations such as normalization and softmax, with minimal impact on performance. This approach maximizes the potential of equivalent transformation-based methods for achieving linear-invariant weight equalization. Consequently, the model remains mathematically equivalent to its original version when unquantized, thus enhancing its generalizability compared to other learning-based quantization methods, such as Quantization Aware Training (QAT). To validate the performance of MobileQLLM on mobile devices, the model’s accuracy, inference latency, and energy consumption is comprehensively assessed as described above. The results demonstrate significant improvements in reducing inference latency and energy use, while maintaining accuracy compared to models utilizing 16-bit activations in full.

[0095] Some of the contributions of the present techniques are summarized as follows:(1) A mobile-friendly quantization approach is introduced for large language models (LLMs) that optimizes compute, memory, and energy efficiency while preserving the accuracy of fullprecision LLMs on mobile devices. Crucially, the present approach is fully supported by current hardware implementations and libraries, thus being directly deployable in real scenarios.(2) The present method improves upon prior works by implementing a holistic training paradigm that jointly optimizes all equivalent transformation and quantization parameters. This allows demonstration of the fact that all activations, except for a limited set involved with nonlinear operators, can be effectively quantized to a lower bitwidth (i.e. 8-bit) without any loss in performance.(3) A comprehensive on-device evaluation is introduced that assesses model accuracy, inference latency, and energy consumption. The results indicate that the present method significantly reduces both inference latency and energy usage compared to models using 16- bit activations, while still maintaining accuracy.

[0096] Related Work

[0097] Post-training Quantization (PTQ): Previous research in post-training quantization can be categorized into three main groups: Weight-only Quantization, Weightactivation Quantization and Learning to Round.

[0098] Weight-only Quantization focuses on only compressing the model weights to reduce storage requirements and memory transfer overheads. Representative works such as OmiQuant generally achieve performance comparable to full-precision models and maintain similar inference speeds on GPUs. However, these methods dequantize weights to full precision (16-bit values) on the fly, resulting in full -preci si on floating-point computations. This results in high peak memory usage, latency, and energy consumption, particularly on edge devices such as mobile phones, as demonstrated below.

[0099] Weight-activation Quantization extends quantization to both model weights and activations asiming to further reduce computational overhead. However, as indicated in prior works such as SmoothQuant, unlike weights, activations, which have dynamic ranges across different data distributions, are more challenging to quantize. Consequently, quantizing all activations to a lower bit-width often results in a significant performance decline. Leading solutions either retain some compute-intensive matrix multiplications in full precision or utilize dynamic per-token activation quantization, which is substantially slower on GPUs and lacks robust hardware support on mobile platforms. Compared to those, the present techniques quantizes all operations (not only a subset), and has full hardware support, including for edge devices.[000100] Learning to Round. Notable works also focus on weight-only quantization but introduce techniques for learning optimal weight rounding. Such works argue that the conventional round-to-nearest method is suboptimal, as it does not account for the interdependencies among adjacent weights. The present techniques run concurrently with this and could be integrated with these techniques.[000101] As an alternative to PTQ, Quantization Aware Training (QAT) involves retraining or fine-tuning full-precision models using differentiable quantizers. Recent research has shown that QAT outperforms post-training quantization, particularly in downstream applications. However, QAT requires extensive training, which is often impractical for large language models (LLMs). Additionally, QAT may be vulnerable to domain shifts if the data used for pretraining is unavailable. In contrast, the present approach is fully zero-shot, only requiring a minimal set of calibration samples and a limited compute budget. Once trained, the present model remains mathematically equivalent to the original model when unquantized, enhancing its adaptability to various downstream tasks. The mathematical equivalence indicates that the new model is potentially as good as the old model when generalized to other domains, while the QAT model suffers from domain shift even if it is unquantized.[000102] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive concept as defined in the appended claims.

Claims

Claims

1. A computer-implemented method for quantizing, using a first electronic device, a large language model, LLM, for use on a second electronic device, wherein the LLM comprises a set of weights and a set of activation tensors, the method comprising: defining a scaling vector which comprises a plurality of weight equalization parameters to equalize the set of weights and the set of activation tensors before quantization; defining range parameters which are representative of a minimum value for each activation tensor and a maximum value for each activation tensor; obtaining a calibration dataset for training the weight equalization parameters and the range parameters; jointly learning, using the calibration dataset, the weight equalization parameters and the range parameters by: applying the scaling vector to the set of weights and the set of activation tensors to obtain a scaled set of weights and a scaled set of activation tensors; quantizing the scaled weights of the LLM to obtain a quantized LLM having a set of quantized weights; and optimising the weight equalization parameters and the range parameters by minimising a difference between an output from the LLM and an output from the quantized LLM, while keeping the weights of the LLM and the weights of the quantized LLM fixed; and outputting the quantized weights for the quantized LLM to the second electronic device together with the optimised weight equalization parameters and the optimised range parameters.[Claim 21The method as claimed in claim 1, wherein each activation tensor has a plurality of channels and defining a scaling vector comprises defining, for each channel, a weight equalization parameter.

3. The method as claimed in claim 2, comprising for each channel defining a weight equalization parameter by initialising the weight equalization parameter using a maximum value of an activation for the channel, a maximum value of a weight for the channel and a hyperparameter.

4. The method as claimed in claim 3, comprising initialising each weight equalization parameter Sj for each channel i usingare the activation tensor values for channel i, Wj are the weight values for channel i and a is the hyperparameter.

5. The method as claimed in claim 1, wherein the LLM comprises a plurality of layers and the method further comprises defining a scaling vector which comprises weight equalization parameters for each linear layer of the LLM.

6. The method as claimed in claim 5, wherein applying the weight quantization comprises applying the weight quantization to consecutive linear layers of the LLM.

7. The method as claimed in claim 1, further comprising defining range parameters which comprise a scale parameter and an offset parameter.

8. The method as claimed in claim 1, further comprising: obtaining a target quantization level for the set of activation tensors; and wherein defining the range parameters comprises using the target quantization level.

9. The method as claimed in claim 8, further comprising defining range parameters such that the minimum value for each activation tensor is represented by a product of the scale parameter and the offset parameter, and the maximum value for each activation tensor is represented by the product of the scale parameter with a sum of the offset parameter and a value based on the target quantization level.

10. The method as claimed in claim 1, further comprising: defining weight clipping parameters; and jointly learning, using the calibration dataset, the weight clipping parameters with the weight equalization parameters and the range parameters.[Claim 111The method as claimed in claim 1, wherein quantizing the scaled weights of the LLM comprises reducing a size of the weights to 4 bits or 8 bits.

12. The method as claimed in claim 1, wherein outputting weights for the quantized LLM, the method further comprises quantizing each activation value in the activation tensors to 8 bits or 16 bits.

13. A computer-implemented method for generating an output using a large language model, LLM, which has been quantized according to any one of the preceding claims, the method comprising: receiving an input; processing the received input using the quantized LLM to generate an output; and outputting the generated output.

14. A computer-readable storage medium comprising instructions which, when executed by a processor on a first electronic device, causes the processor to carry out the method of claim 13.

15. An electronic device for quantizing a large language model, LLM, the electronic device comprising: memory storing an LLM which comprises a set of weights and a set of activation tensors; and at least one processor coupled to memory for: defining a scaling vector which comprises a plurality of weight equalization parameters to equalize the set of weights and the set of activation tensors before quantization; defining range parameters which are representative of a minimum value for each activation tensor and a maximum value for each activation tensor; obtaining a calibration dataset for training the weight equalization parameters and the range parameters; jointly learning, using the calibration dataset, the weight equalization parameters and the range parameters by: applying the scaling vector to the set of weights and the set of activation tensors to obtain a scaled set of weight and a scaled set of activation tensors; quantizing the scaled weights of the LLM to obtain a quantized LLM; and optimising the weight equalization parameters and the range parameters by minimising a difference between an output from the LLM and an output from the quantized LLM, while keeping the weights of the LLM fixed; and outputting the quantized weights for the quantized LLM to a user electronic device together with the optimised weight equalization parameters and the optimised range parameters.

Images