Quantization methods for language models and electronic devices

By employing outlier-paired encoding and an adaptive bias floating-point data type quantization method, the problems of inference speed and hardware applicability in large language models are solved, achieving fast quantization and improved accuracy. This method is applicable to both graphics processors and tensor processors.

CN116451769BActive Publication Date: 2026-11-10SHANGHAI JIAOTONG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202310395701.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-04-13
Publication Date
2026-11-10
Estimated Expiration
2043-04-13

AI Technical Summary

Technical Problem

Existing quantization techniques cannot efficiently improve inference speed while ensuring the performance of large language models, and they also incur significant hardware modification costs, making them unsuitable for widely used graphics processors and tensor processors.

Method used

An outlier-pruned value pairing encoding method is adopted. Normal and outlier values ​​are identified by determining the quantization scaling factor, and adaptive bias floating-point data type is used for quantization to prune outliers, configure outlier identifiers, and maintain memory access alignment.

Benefits of technology

It enables rapid quantization of large language models, improves inference speed while maintaining accuracy, is applicable to common hardware such as GPUs and TPUs, reduces hardware overhead, and achieves performance improvements close to the original accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116451769B_ABST
    Figure CN116451769B_ABST
Patent Text Reader

Abstract

This invention provides a method and electronic device for quantizing a language model. The method includes: determining a quantization scaling factor; identifying normal and outlier values ​​based on neural network parameters and the scaling factor; pairing every two adjacent positions in the tensor of the neural network model; pruning the other value in the pair when an outlier is found, and assigning an outlier identifier to the pair; and quantizing the pairs containing outliers. This invention enables rapid quantization of large language models, accelerating inference speed, reducing computational overhead, while maintaining model accuracy and performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence algorithm technology, and in particular to the field of neural network model quantization technology. Background Technology

[0002] With the continuous development and advancement of deep learning algorithms, artificial intelligence algorithms are increasingly being applied to various fields of real life. In recent years, large language models (LLMs) based on Transformers have developed particularly rapidly and have achieved great success in many natural language processing fields such as machine translation, information retrieval, question answering, and text summarization. However, large language models often improve accuracy by increasing their size. According to incomplete statistics, the size of language models increases 240 times every two years, while hardware computing power only increases by an average of 3.1 times every two years. Therefore, there is a huge gap between the two, which brings great challenges to the deployment and practical application of large language models. For example, the recent Transformer-based LLM OPT-175B has 175 billion parameters and cannot even be deployed on a GPU H100 with 80GB of video memory on a single card.

[0003] Neural network model quantization is a hardware-friendly and effective method that can improve the inference speed of neural networks and reduce computational overhead. It compresses the size of the neural network model and accelerates inference on hardware by using data types with lower memory overhead, at the cost of a negligible or small decrease in neural network recognition accuracy. However, as the size of large language models continues to increase, the performance of language models becomes significantly affected by some very large outliers during computation. Therefore, many previously proposed quantization methods, when applied to large language models, lead to a non-negligible decrease in recognition accuracy, failing to guarantee the recognition precision of the neural network.

[0004] Existing techniques for effectively quantizing large language models primarily rely on mixed-precision quantization based on outlier sparsity. This technique leverages the sparse nature of outliers encountered during neural network computation, using sparse matrix storage methods such as coordinate lists to record all outliers. During quantization, a mixed-precision quantization approach uses higher-precision data types to represent larger outliers and lower-precision data types to represent normal values. This achieves good compression of large language models while maintaining the model's recognition accuracy and performance.

[0005] However, this technique has certain problems. Whether it's the need to maintain an additional sparse coordinate list or the requirement to use different precision data types for outliers and normal values, both can lead to misaligned and random memory accesses, which will severely impact the inference speed of large language models on hardware. Furthermore, the change in memory access patterns caused by this sparse list technique makes it unsuitable for the graphics processing units (GPUs) and tensor processors (TPUs) widely used in current neural network model inference, requiring significant hardware modifications for its use.

[0006] Current quantization techniques present numerous challenges for large language models, failing to efficiently improve inference speed while maintaining overall performance. Given the ever-increasing number of parameters in large language models and the rapidly growing computational demands, efficient quantization techniques with minimal impact on model accuracy are urgently needed. Summary of the Invention

[0007] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a language model quantization method and electronic device that can quickly quantize large language models, accelerate their inference speed, reduce computational overhead, and ensure the accuracy and performance of the model.

[0008] To achieve the above and other related objectives, the present invention provides a method for quantizing a language model, comprising: determining a quantization scaling factor; identifying normal values ​​and outliers based on neural network parameters and the scaling factor; pairing every two adjacent positions in the tensor of the neural network model, pruning the other value in the pair when an outlier occurs in the pair, and configuring an outlier identifier for the pair; and quantizing the pairs containing outliers.

[0009] In one embodiment of the present invention, determining the quantization scaling factor includes: using the 3σ value of the neural network parameters as the initial scaling factor for quantization; searching for multiple undetermined scaling factors based on the paired encoding quantization error with outliers; and determining the undetermined scaling factor with the smallest encoding quantization error as the optimal scaling factor, which is used as the scaling factor for identifying normal values ​​and outliers.

[0010] In one embodiment of the present invention, pairing every two adjacent positions in the tensor of the neural network model includes: if the values ​​of the two adjacent positions are both normal values, integer quantization is used; if one value of the two adjacent positions is an outlier and the other is a normal value, the normal value is pruned and recorded as a pruning value; if the values ​​of the two adjacent positions are both outliers, the outliers with smaller values ​​are pruned.

[0011] In one embodiment of the present invention, configuring an outlier identifier for the pair includes: configuring an end value of an integer value range as an outlier identifier, and removing the end value from the integer value range; in a pair where an outlier exists, replacing the clipped value with the outlier identifier.

[0012] In one embodiment of the present invention, the quantization of pairs containing outliers includes: converting a floating-point data type to a floating-point data type with a bias.

[0013] sign×(1< <mb+mantissa)<<(exponent+bias);

[0014] Where sign represents the sign, exponent represents the exponent, bias represents the bias, mantissa represents the mantissa, and mb represents the number of bits in the mantissa.

[0015] To achieve the above and other related objectives, the present invention also provides an electronic device, including a memory for storing a computer program; and a processor for running the computer program to implement the steps of the language model quantization method as described above.

[0016] In one embodiment of the present invention, the processor is a graphics processor; wherein, each element dot product unit in each tensor computation core of the graphics processor is embedded with a decoder, an adder, and a shifter.

[0017] In one embodiment of the present invention, the processor is a tensor processor; wherein, a decoder is disposed at the boundary of the systolic array of the tensor processor; the size of the systolic array is n×m, and the number of decoders is n+m; n and m are the number of rows and columns of the systolic array, respectively.

[0018] In one embodiment of the present invention, the decoder includes: an outlier determination unit, used to determine whether there are outliers in the pairing of input data; a normal value decoder, which outputs the original encoding of the normal values ​​in the pairing; and an outlier decoder, which decodes the outliers in the pairing into exponential-integer pairing values.

[0019] In one embodiment of the present invention, for a floating-point data type with bias, the decoder decodes the exponent through the relationship between the exponent and the bias; and decodes the radix through the relationship between the radix and the input data.

[0020] As described above, the language model quantization method and electronic device of the present invention have the following beneficial effects:

[0021] This invention enables rapid quantization of large language models, accelerating inference speed and reducing computational overhead while maintaining model accuracy and performance. It achieves near-original accuracy and significantly improved inference speed on commonly used language models. It can handle outliers in neural network models with very low hardware overhead, achieving high performance and pushing the limits of 4-bit quantization in neural networks to a new advanced level. Furthermore, this invention can be effectively integrated into existing hardware processors, such as GPU tensor computing cores and TPU systolic arrays, providing efficient support for deploying large neural network models on various devices. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 The diagram shows a flowchart of a language model quantization method according to an embodiment of this application.

[0024] Figure 2 This is a schematic diagram of the outlier-pruning pairing process in the quantization method of a language model in one embodiment of this application;

[0025] Figure 3 The diagram shown is a schematic diagram of outlier-paired encoding in the quantization method of a language model in one embodiment of this application;

[0026] Figure 4 The diagram shown is a schematic diagram of the principle structure of a graphics processor in one embodiment of this application;

[0027] Figure 5 The diagram shown is a schematic diagram of the principle structure of a tensor processor in one embodiment of this application;

[0028] Figure 6 The diagram shows the principle structure of an outlier-paired decoder in one embodiment of this application.

[0029] Figure 7 The diagram shown is a schematic diagram of the principle structure of an outlier decoder in one embodiment of this application;

[0030] Figure 8 The diagram shows the overall implementation process of the language model quantization method in one embodiment of this application. Detailed Implementation

[0031] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0032] Current quantization techniques present numerous challenges for large language models, failing to efficiently improve inference speed while maintaining overall performance. Given the ever-increasing number of parameters in large language models and the rapidly growing computational demands, efficient quantization techniques with minimal impact on model accuracy are urgently needed.

[0033] The purpose of this embodiment is to provide a language model quantization method and electronic device, which is a hardware-friendly outlier-aware quantization method and an efficient hardware design and implementation. It can quickly quantize large language models, accelerate their inference speed, reduce the computational overhead, and ensure the accuracy and performance of the model.

[0034] The following will elaborate on the principles and implementation methods of the language model quantization method and electronic device of the present invention, so that those skilled in the art can understand the language model quantization method and electronic device of the present invention without creative effort.

[0035] Figure 1 The diagram shown is a flowchart illustrating a language model quantization method according to an embodiment of this application; as follows: Figure 1 As shown, this embodiment provides a method for quantizing a language model, wherein the speech model is a large language model (LLM), and the method includes the following steps S100 to S400.

[0036] Step S100: Determine the quantization scaling factor;

[0037] Step S200: Identify normal and outlier values ​​based on neural network parameters and the scaling factor;

[0038] Step S300: Pair up every two adjacent positions in the tensor of the neural network model. When an outlier appears in the pair, prune the other value in the pair and configure an outlier identifier for the pair.

[0039] Step S400: Quantize the pairs with outliers.

[0040] The quantization method of the language model in this embodiment uses outlier-visit pair (OVP) encoding. Outlier-visit pair encoding identifies outliers and maintains aligned memory accesses, followed by accurate quantization of the outliers at low precision using an adaptively biased floating-point data type.

[0041] The following provides a detailed description of steps S100 to S400 of the language model quantization method in this embodiment.

[0042] Step S100: Determine the quantization scaling factor.

[0043] This embodiment employs an algorithm that minimizes the mean squared error (MSE) to determine the quantization scaling factor (i.e., the threshold for distinguishing outliers from normal values). On one hand, a smaller threshold leads to more outlier-pruned value pairings, which may reduce the quantization error (MSE). On the other hand, it also increases the proportion of outlier-outlier pairings. However, if the number of outlier-outlier pairings is excessive, the resulting outlier pruning will increase the MSE. This invention can reasonably control the proportion of outlier-outlier pairings, preventing it from becoming too high, thereby improving the model's prediction accuracy.

[0044] In this embodiment, the optimal scaling factor, i.e., the threshold between outliers and normal values, is obtained for the parameters of each layer of the neural network using a scaling factor selection algorithm. After obtaining the optimal scaling factor, the neural network parameters and the optimal scaling factor are used as input to an outlier-pruned value pairing encoding algorithm to obtain the quantization result of the pairing encoding, and the quantized result is output.

[0045] Specifically, in this embodiment, determining the quantization scaling factor includes: using the 3σ value of the neural network parameters as the initial scaling factor for quantization; searching for multiple undetermined scaling factors based on the paired encoding quantization error with outliers; and determining the undetermined scaling factor with the smallest encoding quantization error as the optimal scaling factor, which is used to identify normal values ​​and outliers.

[0046] This embodiment primarily focuses on post-training quantization, which eliminates the need for gradient descent-based fine-tuning. Therefore, it is best suited for quantizing large language models, as fine-tuning such models requires significant hardware resources and considerable time. This embodiment only needs a batch of data from the training set to select the scaling factor, making it highly efficient and convenient. Inspired by the 3σ theorem, this embodiment uses the 3σ value of the tensor as the initial scaling factor for quantization. Then, it searches for the optimal scaling factor with the smallest MSE within a specific range of this initial value. This method has performed well in practical experiments. For quantization-aware training requiring fine-tuning, a straight-through estimator can be used to obtain a suitable scaling factor.

[0047] Specifically, in the scaling factor selection algorithm, the 3σ value of the neural network parameters is first calculated as the initial scaling factor. Then, a scaling factor search algorithm is used to select new undetermined scaling factors within a specific range. This scaling factor is then used to recalculate the outlier-prune pairing encoding of the neural network parameters, and the mean squared error (MSE) is used to estimate the quantization error of encoding using this scaling factor, updating the current best result. This process is repeated until all undetermined scaling factors have been explored, and the scaling factor with the smallest output error is the optimal scaling factor found.

[0048] Step S200: Identify normal and outlier values ​​based on neural network parameters and the scaling factor.

[0049] Normal and outlier values ​​in the neural network parameters are filtered based on the scaling factor, wherein the scaling factor is the threshold for distinguishing between outlier and normal values. Normal and outlier values ​​are identified by comparing the neural network parameters with the threshold.

[0050] Step S300: Pair every two adjacent positions in the tensor of the neural network model. When an outlier appears in the pair, prune the other value in the pair and configure an outlier identifier for the pair.

[0051] In this embodiment, pairing every two adjacent positions in the tensor of the neural network model includes:

[0052] 1) If the values ​​at two adjacent positions are both normal values, integer quantization is used;

[0053] 2) If one value in two adjacent positions is an outlier and the other is a normal value, the normal value is pruned and recorded as the pruning value;

[0054] 3) If the values ​​of two adjacent positions are both outliers, prune the outliers with smaller values.

[0055] Experiments show that pruning outliers has a significant impact on the accuracy of neural network models, even causing them to fail. However, removing an equal proportion of normal values ​​has only a negligible impact on the accuracy. Because outliers are sparsity-dependent, the vast majority of values ​​adjacent to them are normal values; therefore, pruning values ​​adjacent to outliers also has only a minor impact on the accuracy of the neural network model.

[0056] like Figure 2 As shown: In this embodiment, each pair of adjacent positions in the tensor of the neural network model is paired up. The numbers in these two positions can only have three possibilities:

[0057] (1) The values ​​at both positions are normal values, such as (1.5, 2.6) pairing. For cases where both are normal values, no pruning is required, only simple quantization. For normal values, this embodiment typically uses 4-bit integer (INT4) and 8-bit integer (INT8) quantization.

[0058] (2) One value is an outlier and the other is a normal value, for example, (17.6, 4.2) pairing. In this embodiment, the normal value in the pairing is pruned and called the pruning value, which is used to provide more coding space for the outlier.

[0059] (3) Both values ​​are outliers, such as the pair (30.7, 20.1). For pairs where both values ​​are outliers, this embodiment will retain the larger outlier and prune the smaller outlier.

[0060] Therefore, in this embodiment, only normal value-normal value pairings and outlier-pruned value pairings will remain in the tensor of the neural network model. The encoding design will then be applied to these two pairing patterns.

[0061] To distinguish it from normal value-normal value pairings, a special identifier is needed for outlier-clipped value pairings. This unique identifier cannot appear in normal value-normal value pairings, which means that a number needs to be excluded from the encoding of normal values.

[0062] Specifically, in this embodiment, configuring an outlier identifier for the pair includes: configuring one end value of the integer value range as an outlier identifier, and removing the end value from the integer value range.

[0063] In this embodiment, the outlier markers are shown in Table 1 below.

[0064] Table 1. Design of Outlier Identifiers

[0065]

[0066]

[0067] As shown in Table 1, assume that a signed int4 (4-bit integer) is used for normal value quantization. The original int4 can represent integers in the range [-8, 7], where 10002 represents the value -8. First, 10002 is used as an outlier identifier and removed from int4, thus changing its encoding range to [-7, 7]. Furthermore, the paired encoding of this invention can be extended to higher precision quantization (such as 8 bits). Similarly, 8-bit normal value encoding also requires eliminating one digit. For example, int8 can represent integers in the range [-128, 127], and this invention uses 10000000 as the outlier identifier for int8, narrowing its range to [-127, 127].

[0068] In this embodiment, in pairs containing outliers, the clipped value is replaced with the outlier identifier. That is, for each outlier-clipped value pair, the clipped value is replaced with an identifier. For example... Figure 3 As shown, normal values ​​are quantized normally, clipped values ​​will be replaced with 4-bit binary code 10002, while outliers will use a special outlier quantization scheme, which quantizes outliers using a dedicated outlier data type.

[0069] Step S400: Quantize the pairs with outliers.

[0070] In this embodiment, the quantization of pairs containing outliers includes: converting the floating-point data type to a floating-point data type with a bias.

[0071] sign×(1< <mb+mantissa)<<(exponent+bias);

[0072] Where sign represents the sign, exponent represents the exponent, bias represents the bias, mantissa represents the mantissa, and mb represents the number of bits in the mantissa.

[0073] Because outliers typically have very large absolute values ​​and a wide range, this embodiment uses floating-point data for quantization. To ensure compatibility with the encoding methods for normal values, this embodiment proposes a method to convert floating-point representation to fixed-point representation. Simultaneously, to prevent the encoding range of outliers from overlapping with that of normal values ​​and thus maximizing the numerical representation capability of the data type with the same bit length, this embodiment proposes a new data type called adaptive bias floating-point data type. Its key idea is to add an appropriate bias to the exponent so that all encoded values ​​can skip the interval containing normal values, thereby providing a larger encoding range for outliers. In other words, this embodiment uses an adaptive bias floating-point data type suitable for quantizing outliers, enabling accurate quantization of outliers with a low bit count.

[0074] To ensure compatibility with normal value encoding and avoid decimals, this invention first converts the floating-point encoding to a fixed-point encoding with an exponent. Furthermore, since fixed-point representation is more hardware-friendly than floating-point representation, this results in faster computation speeds and reduced hardware overhead. The original biased floating-point data type, composed of a sign, exponent, and mantissa, can be represented by the following formula:

[0075] sign×2 exponent+bias ×1.mantissa

[0076] Here, the bias is a constant. If 01012 is a floating-point data type with 1 sign bit (0), 2 exponent bits (10), and 1 mantissa bit (1), and the bias is 0, then its sign is positive and its exponent is 10² = 2. 10 Its last digit is 1.12 = 1.5 10 Therefore, the final true value is 2. 2+0 ×1.5=6.

[0077] Furthermore, this embodiment converts a floating-point data type to a floating-point data type with bias using the following formula:

[0078] sign×(1< <mb+mantissa)<<(exponent+bias);

[0079] Where sign represents the sign, exponent represents the exponent, bias represents the bias, mantissa represents the mantissa, and mb represents the number of bits in the mantissa.

[0080] Since this only involves shift operations and does not include decimals, this fixed-point representation is very hardware-friendly and efficient. In this embodiment, the E2M1 adaptive bias floating-point data type is used, which is a floating-point data type with 1 sign bit, 2 exponent bits, and 1 mantissa bit, with an adaptive bias added on top. Table 2 shows all the actual values ​​and their corresponding codes when the bias is 0.

[0081] Table 2 shows the encoding and actual values ​​of adaptive biased floating-point data types with a bias of 0.

[0082] Binary representation symbol index cardinality True value 0000 1 0 0 0 0001 1 0 3 3<<0=3 001x 1 1 2,3 2<<1=4,3<<1=6 010x 1 2 2,3 2<<2=8,3<<2=12 011x 1 3 2,3 2<<3=16,3<<3=24 1000 (identifier) -1 0 0 -0 1001 -1 0 3 -3<<0=-3 101x -1 1 2,3 -2<<1=-4,-3<<1=-6 110x -1 2 2,3 -2<<2=-8,-3<<2=-12 111x -1 3 2,3 -2<<3=-16,-3<<3=-24

[0083] Clearly, the encoding range of a floating-point data type with a bias of 0 overlaps with the normal value range. Therefore, this embodiment uses an adaptive bias floating-point data type. In the example above, the bias is set to 2, so the absolute value of its non-zero value encoding range is expanded by a factor of 4 from {3,…24} to {12,…96}. This is complementary to the normal value range {1,…7} of a 4-bit int, thus allowing all encoded values ​​to be used for quantization outliers. In this embodiment, for 8-bit quantization, an E4M3 adaptive bias floating-point data type is used, which is a floating-point data type with 1 sign bit, 4 exponent bits, and 3 mantissa bits.

[0084] Furthermore, this embodiment excludes outlier identifiers from the encoding of this data type; otherwise, it would be impossible to distinguish between outlier-clipping value pairs. Therefore, 1000 can be directly disabled for outliers to avoid conflicts with outlier identifiers.

[0085] As can be seen from the above, the outlier-pruned value pairing encoding algorithm in this embodiment uses the input scaling factor to identify outliers and normal values, uses an adaptive bias floating-point data type to quantize outliers, and uses a 4-bit integer to quantize normal values. When an outlier appears in a pair, it prunes the other value and embeds an outlier identifier, thus completing the outlier-pruned value pairing encoding.

[0086] This embodiment also provides an electronic device, comprising: a memory for storing a computer program; and a processor for running the computer program to implement the steps of the language model quantization method as described above. Since the steps of the language model quantization method have already been described in detail above, they will not be repeated here.

[0087] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0088] The processor is a Central Processing Unit (CPU). The memory is connected to the processor via a system bus and communicates with it. The memory stores computer programs, and the processor runs these programs to execute the quantization method of the language model. The memory may include Random Access Memory (RAM) and may also include non-volatile memory, such as at least one disk drive.

[0089] Furthermore, this embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the language model quantization method. The language model quantization method has already been described in detail above and will not be repeated here.

[0090] In this embodiment, as Figure 4 As shown, the processor can be a graphics processing unit; wherein, each element dot product unit in each tensor computation core of the graphics processing unit is embedded with a decoder, an adder, and a shifter.

[0091] The following explains how to apply the method of this embodiment to... Figure 4 In the tensor computing core architecture of the GPU.

[0092] like Figure 4 As shown, this embodiment uses the Turing architecture of the GPU as its foundation, which has 68 streaming multiprocessors (SMs), each with 8 tensor computation cores (544 in total). Each tensor computation core has 8 FEDP (quadruple dot product) units. Therefore, there are 68 × 8 × 2 × 8 × 4 = 34,816 16-bit floating-point multipliers. The Turing architecture natively supports mixed-precision computing: for example, the RTX 2080 Ti using the Turing architecture provides 107.6, 215.2, and 430.3 TOPS of computing power for 16-bit floating-point operations, 8-bit int, and 4-bit int, respectively. Therefore, as Figure 4As shown, the tensor computation core can simultaneously support 8-bit 8EDP (eight-element dot product) and 4-bit 16EDP (six-element dot product). Here, Int4 is the mantissa of a 4-bit integer, and Exp4 represents the 4-bit exponent. Calculations are performed through multiplication, addition, and right shift.

[0093] This embodiment can be easily embedded into GPUs employing a SIMD architecture. It only requires embedding a 4-bit outlier-pair (OVP) decoder for each 16EDP. Figure 4 (See the rightmost diagram). Furthermore, to support the new data types proposed in this embodiment, each 16EDP requires an adder and a shifter. Similarly, this embodiment also designs an 8-bit decoder for the 8EDP unit.

[0094] In this embodiment, as Figure 5 As shown, the processor can also be a tensor processor; wherein, a decoder is configured at the boundary of the systolic array of the tensor processor; the size of the systolic array is n×m, and the number of decoders is n+m; n and m are the number of rows and columns of the systolic array, respectively.

[0095] The following explains how to apply the method of this embodiment to... Figure 5 In the systolic array of the tensor processor.

[0096] The integration of this embodiment in the pulsating array is as follows: Figure 5 As shown in the diagram, systolic arrays use the same outlier-paired (OVP) decoder design as GPUs. However, unlike GPUs, decoders only need to be placed at the boundaries of the systolic array, resulting in a significant reduction in decoders compared to GPUs. Assuming an array size of n×m, only n+m decoders are needed, instead of n×m. This is the advantage of systolic arrays over the SIMD architecture of GPUs. Similarly, as... Figure 4 As shown in the multiply-accumulate unit, the new data type in this embodiment can also be supported by a systolic array processing unit (PE) with only one additional adder and shifter added. Here, Int4 is the mantissa of a 4-bit integer, and Exp4 represents the exponent of a 4-bit integer. Calculations are performed through multiplication, accumulation, and right shift. Furthermore, higher precision quantization (e.g., int8) can be supported by adding only one additional adder for every four PEs.

[0097] To support outlier-paired value decoding, this embodiment designs a new decoder that can be easily embedded into existing accelerators. In this embodiment, the decoder includes: an outlier detection unit for determining whether outliers exist in the pairings of input data; a normal value decoder for outputting the original encoding of the normal values ​​in the pairings; and an outlier decoder for decoding the outliers in the pairings into exponent-integer pairing values.

[0098] Figure 6 A 4-bit decoder is demonstrated that reads one byte (8 bits) of two 4-bit values ​​per cycle. This is the smallest addressable memory unit in many hardware architectures and is precisely an outlier-pruned value pair. If the pair contains the outlier identifier 1000, the decoder decodes it as 0 and uses the outlier decoder to decode the outlier, while the normal value decoder directly outputs the original encoding. The outlier decoder decodes the input value into an Exp-Int pair (exponent, mantissa pair) to accommodate the multiply-accumulate units in GPU and TPU architectures.

[0099] Similarly, this embodiment also designs an 8-bit value decoder, which only requires... Figure 5 The decoding bit width can be extended to 8 bits.

[0100] In this embodiment, for floating-point data types with bias, the decoder decodes the exponent through the relationship between the exponent and the bias; and decodes the radix through the relationship between the radix and the input data.

[0101] Specifically, such as Figure 7 As shown, for a 4-bit signed adaptive bias floating-point data type, assuming it is x = (b2b1b0)2, its exponent and base can be decoded using the following formula:

[0102] Exponent = Bias + (b2b1)2

[0103]

[0104] For example, when the bias is 2, 01012 will be decoded as 48. 10 Because its exponent is 2. 10 +102 = 41, the base number is 11203 10 Therefore, its value can be obtained as 3 << 4 = 48. Figure 7 The design of a 4-bit signed adaptive bias floating-point data type decoder is demonstrated. The design of an 8-bit outlier-paired decoder is similar, requiring only... Figure 7 The decoding bit width can be extended to 8 bits.

[0105] After decoding outliers and normal values, they are all converted into uniform exponent-integer pairs. For example... Figure 5 As shown, to support the calculation of the decoded exponent-integer pairs, a shifter and an adder need to be added to the multiply-accumulate (multiplication and accumulation) unit. For example, two exponent-integer (Exp-Int) pairs...<a,b> and<c,d> Where a and b are exponents, and d and b are integers, then<a,b> This means:

[0106] <a,b> =b<

[0107] Therefore, the result of multiplication is (b×d) << (a+c) =<a+c,b×d> .

[0108] The following section describes how this embodiment supports mixed-precision computation. The GPU's tensor computation unit natively supports mixed-precision operations. For the TPU's systolic array, this embodiment can use four processing units (PEs) to natively support 8-bit computation. For an int8 value x, it is first decomposed into the high 4 bits and the low 4 bits, as shown below:

[0109] x=(h x <<4)+l x =<4,h x >+<0,l x >

[0110] The following method can then be used to multiply two int8 numbers x and y:

[0111]

[0112] Therefore, four processing units (PEs) can be used to compute the multiplication of two int8 type numbers.

[0113] Similarly, multiplication of 8-bit adaptive bias floating-point data types can be supported in the same way. For an 8-bit adaptive bias floating-point data type number z, it is first decoded into an exponent e. z and an integer i z For i z Similarly, divide it into i z =(h z <<4)+l z Then we can get z = < 4 + e z ,h z >+ <e z ,l z Therefore, the same method can be used to calculate multiplication of 8-bit adaptive bias floating-point data types, which only has one more exponent e than int8. z .

[0114] For a 4-bit tensor computation core, the Turing architecture uses the instruction mma.s32.s4.s4.s32. These four operands are matrices D (int32), A (int4), B (int4), and C (int32), where D = A × B + C. To support computation on a GPU, this embodiment designs a new instruction mmaovp:

[0115] ​

[0116] Here, `mmaovp` represents an MMA (matrix multiplication and addition) instruction using OVP encoding. The first `s32` indicates that the address operand of the calculated result is in the original 32-bit integer format. `ovpi4` indicates that the two multipliers in the multiplication-addition instruction are in the OVP encoding format of 4-bit integers (`int4`), with -8 removed, representing a range of [-7, +7]. The second `s32` indicates that the addend in the multiplication-addition instruction is in the original 32-bit integer format. The last `s4` represents the original 4-bit integer operand, which represents the bias value in the floating-point encoding of the outlier. Outliers default to the E2M1 format.

[0117] Due to the memory alignment design of the data types, this embodiment preserves the original programming interface of the GPU. Instructions based on this embodiment (such as mmaovp) can be used to replace the original int type mma instructions, thereby easily building a quantization framework that supports this embodiment. Therefore, the framework of this embodiment has comprehensive and practical applicability, which is its most significant advantage.

[0118] Figure 8 This diagram illustrates the overall implementation process of a language model quantization method and electronic device according to an embodiment of this application. Figure 8 As shown, firstly, the optimal scaling factor, i.e., the threshold between outliers and normal values, is obtained for the parameters of each layer of the neural network using a scaling factor selection algorithm. After obtaining the optimal scaling factor, the neural network parameters and the optimal scaling factor are used as input to an outlier-pruned value pairing encoding algorithm to obtain the quantized result of the pairing encoding, and the quantized result is output.

[0119] In the scaling factor selection algorithm, the 3σ value of the neural network parameters is first calculated as the initial scaling factor. Then, a scaling factor search algorithm is used to select new undetermined scaling factors within a specific range. Using this scaling factor, the outlier-prune pairing encoding of the neural network parameters is recalculated, and the mean squared error (MSE) is used to estimate the quantization error of encoding using this scaling factor, updating the current best result. This process is repeated until all undetermined scaling factors have been explored, and the scaling factor with the smallest output error is the optimal scaling factor found.

[0120] The outlier-pruned value pairing encoding algorithm uses an input scaling factor to identify outliers and normal values. It uses an adaptive bias floating-point data type to quantize outliers and a 4-bit integer to quantize normal values. When an outlier is found in a pair, the other value is pruned and an outlier identifier is embedded, thus completing the outlier-pruned value pairing encoding.

[0121] In summary, this embodiment enables rapid quantization of large language models, accelerating inference speed and reducing computational overhead while maintaining model accuracy and performance. It achieves near-original accuracy and significantly improved inference speed on commonly used language models. It can handle outliers in neural network models with very low hardware overhead and achieve high performance, pushing the limits of 4-bit quantization in neural networks to a new advanced level. Furthermore, this embodiment can be effectively integrated into existing hardware processors, such as GPU tensor computing cores and TPU systolic arrays, providing efficient support for deploying large neural network models on various devices. Therefore, this embodiment effectively overcomes the various shortcomings of existing technologies and has high industrial applicability.

[0122] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A method for quantizing a language model, characterized in that: Applied to a graphics processor or tensor processor, each element dot product unit in the tensor computation core of the graphics processor is correspondingly configured with a decoder, adder and shifter; A decoder is configured at the boundary of the systolic array of the tensor processor; the method includes: Determine the quantitative scaling factor; Normal and outlier values ​​are identified based on neural network parameters and the scaling factor. In the tensor of the neural network model, each two adjacent positions are paired. When an outlier appears in a pair, the other value in the pair is pruned, and an outlier identifier is configured for the pair. Configuring an outlier identifier for the pair includes: configuring an end value of the integer range as an outlier identifier and removing the end value from the integer range; in the pair where an outlier exists, the pruned value is replaced with the outlier identifier. Quantization of pairs containing outliers; the quantization of pairs containing outliers includes: Convert floating-point data types to floating-point data types with bias: ; in, Symbols Indicates an index. Indicates bias. Indicates the last digit. The number of digits in the mantissa; The decoder determines whether there are outliers in the pairing of the input data, outputs the original encoding of the normal values ​​in the pairing, and decodes the outliers in the pairing into exponential-integer pairing values. The process of pairing every two adjacent positions in the tensor of the neural network model includes: The values ​​at two adjacent positions are both normal values, so integer quantization is used; If one value in two adjacent positions is an outlier and the other is a normal value, the normal value is pruned and recorded as the pruning value. If the values ​​at two adjacent positions are both outliers, prune outliers with smaller values.

2. The quantization method for language models according to claim 1, characterized in that: The quantitative scaling factor includes: Neural network parameters The value serves as the initial scaling factor for quantization; Based on the pairing encoding quantization error with outliers, multiple undetermined scaling factors are obtained through searching. The undetermined scaling factor with the smallest encoding quantization error is determined as the optimal scaling factor, which is used to identify normal and outlier values.

3. An electronic device, characterized in that: It includes a memory for storing a computer program; and a processor for running the computer program to implement the steps of the quantization method for the language model as described in any one of claims 1 to 2.

4. The electronic device according to claim 3, characterized in that: The processor is a graphics processing unit (GPU); wherein, each element dot product unit in each tensor computation core of the GPU is embedded with a decoder, adder, and shifter.

5. The electronic device according to claim 3, characterized in that: The processor is a tensor processor; wherein, a decoder is configured at the boundary of the systolic array of the tensor processor; the size of the systolic array is n×m, and the number of decoders is n+m; n and m are the number of rows and columns of the systolic array, respectively.

6. The electronic device according to claim 3 or 4, characterized in that: The decoder includes: An outlier detection unit is used to determine whether there are outliers in the pairing of input data; The normal value decoder outputs the original encoded normal values ​​from the pair; An outlier decoder decodes outliers in a pair into exponential-integer pair values.

7. The electronic device according to claim 6, characterized in that: For floating-point data types with bias, the decoder decodes the exponent by the relationship between the exponent and the bias; and decodes the radix by the relationship between the radix and the input data.

Citation Information

Patent Citations

  • Neural network model compression method and system based on mass spectrum data set

    CN112329922A

  • Early gastric cancer image recognition method based on evolutionary neural network model compression

    CN112580639A