Large language model weight quantization method and system based on optimal quantization coding and application
By blocking and normalizing the weights of large language models and using clustering methods to generate quantitative target encoding, the problem of failure to fully consider the characteristics of data distribution in the existing technology is solved, and higher precision quantization is achieved, reducing the Storage and computing overhead are suitable for a variety of application scenarios.
Patent Information
- Application Number
- CN202411122426.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing quantization methods fail to fully consider the data distribution characteristics of different weight tensors for the weight quantization of large language models, resulting in large quantization-inverse quantization errors, affecting model performance.
By blocking and normalizing the weights of the large language model, and using the clustering method to generate quantization target encodings, quantization encodings are generated according to the numerical distribution characteristics of each data block to reduce quantization errors. The specific steps include dividing the weight matrix into data blocks, performing normalization processing, using clustering methods such as K-mean clustering to generate quantization target encodings, and realizing quantization and inverse quantization by looking up tables.
It effectively reduces quantization error, improves the accuracy of the quantized model, reduces storage requirements and computing overhead, is suitable for devices with limited storage resources, improves model inference speed, and is especially suitable for real-time processing application scenarios.
Smart Images

Figure BDA0004995464680000021 
Figure BDA0004995464680000087 
Figure BDA0004995464680000088
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of large language model quantization, and relates to a large language model weight quantization method, system and application based on optimal quantization coding. Background Art
[0002] Pretrained Large Language Model (LLM) has become the hottest research topic in the field of artificial intelligence in the past two years. It is trained on massive amounts of text data and can perform well on a wide range of downstream language tasks with little or no fine-tuning. Large language models are generally composed of many transformer models, and their powerful capabilities mainly rely on their huge parameter scale (usually billions of parameters). In order to quickly train, fine-tune, infer, and deploy large language models (the target device for deployment may not be able to support high-precision storage and communication of huge parameters), model quantization technology is used to process the parameters of large language models.
[0003] In a narrow sense, model quantization usually refers to representing model parameters stored in floating point format (usually 32-bit full-precision floating point numbers or 16-bit half-precision floating point numbers) with integer data (usually 8-bit integers). Because the bit width of data storage is reduced, the storage space occupied by the model and the overhead during loading and calculation will be reduced, which will improve the efficiency of training and inference. Of course, the most primitive quantization (without any optimization and special processing) will also bring some loss in model performance. In order to reduce this loss, other data types are used instead of integer data as the target code for quantization, such as 8-bit floating point numbers, 4-bit floating point numbers, etc. Among them, Dettmers T et al. [1] A normal floating-point number (NormalFloat) data type is proposed. When using it as the target code for quantization, the quantization target code is regarded as a lookup table, and the value of the data to be quantized that is closest (with the smallest difference) in the lookup table is found, and its index value in the lookup table is used as the quantized integer; during dequantization, the floating-point number corresponding to the index value is still found by looking up the table.
[0004] This table-based quantization method generally specifies the quantization target code, that is, the lookup table, according to the distribution characteristics of the data. Existing quantization methods (especially those based on table-based quantization methods) use the same lookup table for all data, without fully considering the data distribution characteristics of different weight tensors, which will cause large "quantization-dequantization" errors and affect the performance of the quantized model. Summary of the invention
[0005] In order to address the deficiencies in the prior art, the purpose of the present invention is to provide a large language model weight quantization method, system and application based on optimal quantization coding, which improves the accuracy of the quantized model by using different lookup tables for quantization of different weight tensors.
[0006] The present invention provides a large language model weight quantization method based on optimal quantization coding. The method implements the weight quantization of the large language model by dividing the weight of the large language model into blocks, normalizing, generating quantization coding, quantizing and dequantizing, and comprises the following steps:
[0007] Step 1: Divide the weight matrix to be quantized into several data blocks;
[0008] Step 2: normalize the elements in each data block in step 1;
[0009] Step 3: According to the numerical distribution characteristics of the normalized weight matrix, a clustering method is used to generate a quantized target code;
[0010] Step 4: Map the elements in each data block after the normalization processing in step 2 to the index value of the quantization target code in step 3 to achieve quantization of the elements;
[0011] Step 5: Find the floating point number corresponding to the index value in the quantized target code, and multiply it by the normalization constant of the data block to obtain a dequantized floating point number.
[0012] In step 1, the compression effect and quantization error of the model are balanced by adjusting the block size. In actual use, the model can be compressed as much as possible under the condition of maximum acceptable quantization error. The block standard is optimized according to specific application requirements and data distribution characteristics to achieve the best quantization effect.
[0013] In step 2, the maximum value of the elements in each data block is used as the normalization constant of the corresponding data block, and the elements in the corresponding data block are normalized using the normalization constant; it is expressed as follows:
[0014]
[0015] Among them, W b Represents the original data block, N b Indicates the value with the largest absolute value in the data block, N b =max(|W b |), as a normalization constant, Represents a normalized data block.
[0016] In step 4, for each element in the normalized data block, according to the nearest neighbor principle, select the value in the target quantization code that is closest to the element, and use the index of the value in the target quantization code as the quantized value;
[0017] Among them, the elements in the normalized data block are represented as The target quantization code is expressed as The value is q i , index is i.
[0018] In step 5, find the floating point number q corresponding to the index i in the target quantization code Q i , the floating point number and the normalization constant N b Multiply to get the dequantized floating point number
[0019] Step three is the innovative focus of the present invention, which is as follows:
[0020] In step 3, cluster analysis is performed on the normalized data blocks using a clustering method to generate a quantized target code. The clustering method includes K-means clustering, and spectral clustering, hierarchical clustering, etc. can also be used. There is no special limitation on the clustering method, as long as it can achieve the desired effect of the present invention;
[0021] When the clustering method is K-means clustering, the initial cluster center is selected by one or more initialization methods to improve the effect of K-means clustering, reduce the number of iterations and convergence time, and finally obtain a quantized target code (lookup table) suitable for the current data;
[0022] The initialization method includes at least one of the following:
[0023] 1) Random initialization method
[0024] Randomly select K points in the data set as the initial cluster centers;
[0025] After selecting the initial cluster center, the K-means clustering algorithm is used to iteratively update the cluster center until convergence or the maximum number of iterations is reached;
[0026] 2) Sorting-based initialization method
[0027] The sorting-based initialization method sorts the normalized data from small to large values, then groups them in the positive and negative ranges, and finally takes the mean of each group as the initial cluster center. The sorting-based initialization can utilize the distribution characteristics of the data, improve the initial clustering effect, and reduce the number of iterations.
[0028] The specific steps are as follows:
[0029] Step 2.1. Sort the normalized weight tensors of the blocks from small to large values;
[0030] Step 2.2. Divide the sorted weight matrix into negative weight intervals and positive weight intervals;
[0031] Step 2.3. In each negative weight interval or positive weight interval, weight groups are performed according to the number of cluster centers;
[0032] Step 2.4. Take the mean of each group as the initial center of each group in the positive and negative weight intervals, and add fixed values of -1, 0 and 1 to form the initial cluster center;
[0033] After selecting the initial cluster center, the K-means clustering algorithm is used to iteratively update the cluster center until convergence or the maximum number of iterations is reached;
[0034] In step 2.3, the negative weight interval is divided into 2 N-1 -2 groups, divide the positive weight interval into 2 N-1 - 1 group; and / or,
[0035] In step 2.4, the number of initialization centers of the positive and negative weight intervals is 2 N -3, the number of the initial cluster centers is 2 N indivual.
[0036] 3) Initialization method based on existing quantization target encoding
[0037] Based on the existing quantitative target code as the initial clustering center, and then optimize these initial centers according to the specific data distribution. This method can use the existing knowledge to improve the clustering effect.
[0038] Step 3.1. Select an existing quantized target code as the initial cluster center;
[0039] Step 3.2. According to the data distribution characteristics of different weight tensors, the existing quantization target encoding is optimized to make it more suitable for the quantization requirements of the current data and form the initial clustering center;
[0040] After selecting the initial cluster center, the K-means clustering algorithm is performed and the cluster center is iteratively updated until convergence or the maximum number of iterations is reached.
[0041] In a specific implementation, the maximum number of iterations of the K-means clustering can be set to 500-1000 times, and the tolerance of the convergence threshold can be set to 1e-6; the iteration can be stopped as long as at least one of the above two parameters is satisfied.
[0042] The quantization target is encoded as a lookup table for weight quantization of a large language model, and the lookup table is expressed as where q i is a floating point number in the range [-1,1].
[0043] The present invention also provides a weight quantization system for implementing the above-mentioned large language model weight quantization method, the system comprising: a data preprocessing module, a clustering analysis module, a quantization module, and a dequantization module;
[0044] The data preprocessing module divides the weight matrix of the large language model into blocks and performs normalization processing;
[0045] The cluster analysis module generates a quantization target code (lookup table) to ensure that the quantization center is consistent with the actual data distribution;
[0046] The quantization module maps the normalized weight value to the quantization target code and records the corresponding index value;
[0047] The dequantization module restores the original floating-point value according to the quantized index value to ensure the accuracy of the model during reasoning.
[0048] The present invention also provides the above-mentioned large language model weight quantization method, or the application of the above-mentioned weight quantization system in model quantization deployed in edge computing systems, model quantization in cloud computing platforms, large model fine-tuning deployment, etc.
[0049] The present invention also provides a hardware system for implementing the above-mentioned large language model weight quantization method, and the hardware system includes: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the above-mentioned large language model weight quantization method is implemented.
[0050] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned large language model weight quantization method is implemented.
[0051] The beneficial effects of the present invention include: the method of the present invention realizes the weight quantization of a large language model by means of "blocking + normalization + lookup table", aiming to reduce storage requirements, reduce computational overhead and reduce quantization errors. The method of the present invention significantly reduces storage requirements by converting high-precision floating-point weights into low-precision integers or low-bit floating-point numbers, and is suitable for devices with limited storage resources. The reduction in computational overhead improves the model reasoning speed and reduces latency, and is particularly suitable for application scenarios that require real-time processing; in addition, block normalization and clustering methods based on data distribution effectively reduce quantization errors, ensure the accuracy of the quantized model, and are suitable for applications with high-precision requirements. The ability to dynamically adjust block size and balance compression effect and quantization error enables the method of the present invention to be optimized according to specific application requirements, and to achieve the best quantization effect in different application scenarios. Supporting multiple clustering methods further improves the flexibility and accuracy of quantization, and enhances the versatility and adaptability of the method. In short, while maintaining model performance, the method of the present invention greatly improves storage and computing efficiency, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0053] Figure 1 : A simple structural diagram of the converter model in the LLaMA model. In order to conveniently show the linear layers of different modules, the normalization layer and residual connection are omitted in the figure.
[0054] Figure 2 : Flowchart of the large language model weight quantization of the present invention.
[0055] Figure 3 : Comparison of the average relative errors of the q-weight matrix in the self-attention module of the 32 transformer layers of the llama 7B model after 8-bit quantization using different quantization target encodings.
[0056] Figure 4 : Comparison of the average relative errors of the k weight matrices in the self-attention module of the 32 transformer layers of the llama 7B model after 8-bit quantization using different quantization target encodings.
[0057] Figure 5: Comparison of the average relative errors of the v weight matrix in the self-attention module of the 32 transformer layers of the llama 7B model after 8-bit quantization using different quantization target encodings.
[0058] Figure 6 : Comparison of the average relative errors of the o weight matrices in the self-attention module of the 32 transformer layers of the llama 7B model after 8-bit quantization using different quantization target encodings.
[0059] Figure 7 :Comparison of the average relative errors of the gate weight matrices in the feedforward network module of the 32 converter layers of the llama 7B model after 8-bit quantization using different quantization target encodings.
[0060] Figure 8 :Comparison of the average relative errors of the up weight matrix in the feedforward network module of the 32 converter layers of the llama 7B model after 8-bit quantization and encoding with different quantization targets.
[0061] Fig. 9 :Comparison of the average relative errors of the down weight matrix in the feedforward network module of the 32 converter layers of the llama 7B model with 8-bit quantization and encoding using different quantization targets. DETAILED DESCRIPTION
[0062] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.
[0063] The method of the present invention adopts the method of "blocking + normalization + lookup table" to realize the weight quantization of a large language model, that is, firstly, the weight matrix is divided into data blocks, and normalized according to the maximum absolute value in the data block, and the data in the block is normalized to between -1 and 1; then, according to the numerical distribution characteristics of the normalized weight matrix, a quantized target code (that is, a quantized lookup table) is obtained by clustering, and the original weight matrix can be quantized accordingly. The method of the present invention fully considers the numerical distribution characteristics of each weight matrix, and thus can effectively suppress quantization loss.
[0064] In order to explain the technical details of the method of the present invention in detail and clearly, the following will introduce the method according to the following parts: blocking, normalization, generating target quantization code (lookup table), quantization and inverse quantization. The main innovation of the method of the present invention is concentrated on the part of generating quantization code (lookup table).
[0065] 1. Chunking
[0066] Generally, in the quantization of a large language model, in order to improve the quantization accuracy and reduce the quantization loss, the data to be quantized can be divided into blocks, and then the same quantization parameters are used for quantization in each data block. For example, for the weight tensor W to be quantized (assuming the shape of W is 4096×4096), it is expanded into a one-dimensional vector by row and divided by block size B=64, resulting in 262144 data blocks W b ,b=1,…,262144. The block size can be adjusted: if the block size B is large, fewer quantization parameters are obtained, and the compression effect is better, but if there are outliers in the block, then the quantization error in the block will be larger due to the influence of the outliers. The limit case is no block division (that is, the block size is 4096×4096); if the block size B is small, there will be more quantization parameters, the compression effect is not obvious, but the quantization loss will be relatively small, and the limit case is that one element is divided into one block (that is, the block size is 1); therefore, in actual use, the block size is determined by balancing the compression effect and the quantization error.
[0067] 2. Normalization
[0068] After block division, for each data block W b , find its maximum absolute value N b =max(|W b |) as a normalization constant, and then all elements in the data block are normalized using this normalization constant, that is, After processing, the value range of the elements in the data block becomes [-1,1].
[0069] 3. Generate quantization code (lookup table)
[0070] After normalizing all the values in the weight matrix W to [-1,1], the corresponding quantization target code (lookup table) is generated according to the numerical distribution characteristics of the entire weight matrix. During quantization, the quantization bit width N is selected, that is, the weight to be quantized (usually 32 bits or 16 bits) is quantized into an N-bit number. Typical values of N are 2, 3, 4 or 8. The smaller N is, the higher the compression rate, but the greater the quantization error. The method of the present invention does not limit the bit width of quantization, and any bit width can be quantized using the method of the present invention.
[0071] Because the present invention adopts table lookup quantization, the goal is to find a lookup table whose entry length is 2 N (That is, the index values of the lookup table are 0, 1, 2, ..., 2 N-1), and the return value is a floating point number between [-1,1] (specifically 16 bits or 32 bits, depending on the type of the original data). To this end, a clustering method is used to obtain this lookup table, that is, the quantized target code. Specifically, K-means clustering (K-Means) can be used, and of course other clustering methods (such as spectral clustering, hierarchical clustering, etc.) can also be used, and the method of the present invention is not limited.
[0072] In the specific implementation process, the convergence conditions of the K-means clustering algorithm can be used as follows (any of the following conditions can be met):
[0073] a) The maximum number of iterations can be set to 500-1000 times;
[0074] b) The convergence threshold tolerance can be set to 1e-6.
[0075] Generating quantization codes (lookup tables) using K-means clustering includes the following:
[0076] The input is the weight tensor after block normalization That is, the number of samples in the data is 4096×4096, and the number of features is 1. The goal is to get 2 N Cluster centers are the quantized target codes to be generated, which are the return values of the lookup table.
[0077] The results of K-means clustering are more dependent on the initial location of the cluster center, so it is proposed that different initialization methods including at least one of the following can be used for clustering, as follows:
[0078] i. Random initialization
[0079] Random initialization does not require any initial cluster centers, and the algorithm is responsible for selecting them. This is the most common initialization method.
[0080] ii. Sorting-based initialization
[0081] The starting point of this initialization method is to make full use of every bit in the quantization target code, so that quantization can achieve relatively low quantization loss within different precision ranges. The specific implementation method is as follows:
[0082] 2.1) Sorting: Normalize the weight tensor of the blocks Sort in order from smallest to largest;
[0083] 2.2) Get the positive and negative intervals: Use 0 as the demarcation point for the sorted weights to get negative weights and positive weights
[0084] 2.3) Grouping in positive and negative intervals: First, determine the number of cluster centers in the positive and negative intervals. Generally, in the negative interval, the number of cluster centers is 2. N-1 -2; in the positive interval, the number of cluster centers is 2 N-1 -1. Then, according to the number of cluster centers, the weight is divided into 2 in the positive and negative intervals. N-1 -1 group (positive interval) and 2 N-1 -2 groups (negative interval);
[0085] 2.4) Obtain the initialization center of the positive and negative intervals: In the positive and negative intervals, take the mean of each group as the initialization center of the group, so that a total of (2 N-1 -2)+(2 N-1 -1)=2 N - 3 initialization centers;
[0086] 2.5) Get all the initial cluster centers: N -3 initialization centers, together with -1, 0, 1, make up a total of 2 N The initial centers are input into the K-means clustering algorithm as the initial clustering centers.
[0087] iii. Initialization based on some existing encoding
[0088] You can also use some other existing quantization target encoding as the initial cluster center of K-means clustering, such as Dettmers T et al. [1] In this way, the existing quantization target encoding can be further optimized according to the data distribution characteristics of different weight tensors, making it more suitable for the quantization of this weight tensor.
[0089] After this step, the target encoding for the quantization of a certain weight tensor is obtained, which is a lookup table where q i is a floating point number in the range [-1,1]. If the original weight tensor is a 16-bit floating point number (FP16), then q i is also 16 bits; the original weight tensor is a 32-bit floating point number (FP32), so q i That's 32 bits.
[0090] 4. Quantification
[0091] For the block-normalized weight tensor Each element in According to the nearest neighbor principle, select the quantized target code The value q that is closest to thei The index i of is taken as the quantized value (an N-bit integer).
[0092] 5. Dequantization
[0093] When dequantizing, first use the quantization target encoding Get the corresponding q according to index i i Then, according to the normalization constant N of each saved data block b , get the dequantized floating point number
[0094] Compared with the existing quantization method based on table lookup, the method of the present invention fully considers the data distribution characteristics of each weight tensor, can effectively reduce the quantization loss, and achieve a smaller "quantization-dequantization" error. Figure 3-9 shown.
[0095] In the present invention Figure 3-9 The numerical values of the quantization-dequantization error are as follows:
[0096] Absolute error: (average value for all layers)
[0097]
[0098] Relative error: (average value of all layers)
[0099]
[0100] It can be seen from the attached drawings and the data in the above table that, compared with 8-bit normal floating-point numbers, the method of the present invention adopts three clustering methods, namely random initialization, sorting-based initialization and initialization based on 8-bit normal floating-point numbers, all of which can achieve smaller relative errors of quantization and dequantization.
[0101] Example
[0102] Now take the llama 7B model as an example, quantize the FP16 weights to 8 bits:
[0103] llama 7B consists of multiple transformer models, which can be divided into two modules: Multi-head Self Attention (MSA) module and FeedForward Network (FFN) module. Each module is composed of several linear layers, each of which performs matrix multiplication on the input and weights to obtain the output. The linear layers in the multi-head attention module of llama 7B include (see Appendix Figure 1): query (query, hereinafter referred to as q), key value (key, hereinafter referred to as k), value (value, hereinafter referred to as v) and output (output, hereinafter referred to as o); the linear layer in the feedforward network module includes (see Appendix Figure 1 ): gate, dimension up and dimension down. Each linear layer has a weight matrix that needs to be quantized. The process is as follows Figure 2 As shown, the details are as follows:
[0104] 1. Divide and normalize each weight matrix to be quantized;
[0105] 2. According to the numerical distribution characteristics of the normalized weight matrix, the K-means clustering method is used to obtain three quantized target codes after three initialization methods, namely: the quantized target codes obtained by random initialization Quantized target encoding based on sorting And the quantized target code initialized with 8-bit normal floating point numbers
[0106] 3. Quantize each block and the normalized weight matrix according to the three quantization target codes obtained in the previous step;
[0107] 4. When necessary (for example, when performing floating-point inference calculations), dequantize the quantized weights obtained in the previous step.
[0108] The quantized llama 7B model was analyzed, and the average relative error of each weight matrix was calculated as the "quantization-dequantization" error, and compared with 8-bit normal floating-point numbers. The results can be found in the attached Figure 3-9 The expression of the mean relative error is as follows:
[0109]
[0110] Among them, W is the weight before quantization, is the weight after quantization-dequantization.
[0111] The results in the table below show the performance of the proposed method on multiple public datasets for the llama7B and llama13B models under different quantization initialization methods. These results are measured by two indicators: accuracy (the larger the value, the better) on the mmlu dataset and perplexity (ppl, the smaller the value, the better) on the wikitext / c4 / c4_new dataset.
[0112] mmlu data set: (the larger the value, the better)
[0113]
[0114]
[0115] Test ppl (perplexity) on the wikitext / c4 / c4_new dataset: (the smaller the value, the better)
[0116]
[0117] These results show that the 8-bit normal floating point initialization method performs slightly better than or close to other initialization methods on various datasets. Specifically:
[0118] On the mmlu dataset, the initialization method based on 8-bit normal floating-point numbers has an accuracy of 0.3515 and 0.4647 on llama7B and llama13B, respectively, which is slightly better than other methods.
[0119] In the perplexity tests on the wikitext, c4 and c4_new datasets, the initialization method based on 8-bit normal floating-point numbers also showed good performance, with perplexity values close to or lower than other initialization methods, especially on the wikitext dataset, where it performed best.
[0120] These results verify that the quantization method proposed in this invention can effectively reduce storage and computing overhead while maintaining model performance, providing support for the efficient deployment of large language models.
[0121] References
[0122] [1]Dettmers T, Pagnoni A, Holtzman A, et al. Qlora: Efficient finetuning of quantized llms[J]. arXiv preprint arXiv:2305.14314,2023.
[0123] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0127] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.
[0128] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.
[0129] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.
Claims
1. A large language model weight quantization method based on optimal quantization coding, It is characterized in that The method includes block division, normalization, generating quantization codes, quantization and inverse quantization, and realizes weight quantization of a large language model, including the following steps: Step 1: Divide the weight matrix to be quantized into several data blocks; Step 2: normalize the elements in each data block; Step 3: According to the numerical distribution characteristics of the normalized weight matrix, a clustering method is used to generate a quantized target code; Step 4: Map the elements in each data block after the normalization processing in step 2 to the index value of the quantization target code in step 3 to achieve quantization of the elements; Step 5: Find the floating point number corresponding to the index value in the quantized target code, and multiply it by the normalization constant of the data block to obtain a dequantized floating point number.
2. The method according to claim 1, It is characterized in that In step 1, the compression effect and quantization error are balanced by adjusting the block size; and / or, In step 2, the maximum value of the elements in each data block is used as the normalization constant of the corresponding data block, and the elements in the corresponding data block are normalized using the normalization constant; it is expressed as follows: Among them, W b Represents the original data block, N b Indicates the value with the largest absolute value in the data block, N b =max(|W b |), as a normalization constant, represents a normalized data block; and / or, In step 4, each element in the normalized data block is quantized to the index of the nearest value in the quantized target code, where the elements in the normalized data block are represented as The target quantization code is expressed as The target quantization code has a value of q i ,q i ∈[-1, 1], indexed by i; and / or, In step 5, the quantized target encoding index i is used to obtain the corresponding floating point number q i , the floating point number and the normalization constant N b Multiply to get the dequantized floating point number 3. The method according to claim 1, It is characterized in that In step 3, cluster analysis is performed on the normalized data blocks by using a K-means clustering method to generate a quantized target code; The initialization method of the k-means clustering includes a random initialization method, a sorting-based initialization method, and an initialization method based on existing quantization target coding.
4. The method according to claim 3, It is characterized in that The random initialization method refers to randomly selecting K points in the data set as initial cluster centers; and / or, The sorting-based initialization method comprises the following steps: Step i, sort the normalized weight tensors of the blocks from small to large values; Step ii, dividing the sorted weight matrix into a negative weight interval and a positive weight interval; Step iii, in each of the negative weight intervals or the positive weight intervals, weight grouping is performed according to the number of cluster centers; Step iv, taking the mean of each group as the initialization center of each group in the positive and negative weight intervals, adding fixed values -1, 0, 1 to form the initial clustering center; and / or, The initialization method based on the existing quantization target code refers to optimizing the initial cluster center corresponding to the existing quantization target code according to the actual data distribution.
5. The method according to claim 4, It is characterized in that In step iii, the negative weight interval is divided into 2 N-1 -2 groups, divide the positive weight interval into 2 N-1 - 1 group; and / or, In step iv, the number of initialization centers of the positive and negative weight intervals is 2 N -3, the number of the initial cluster centers is 2 N indivual.
6. A weight quantization system for implementing the large language model weight quantization method according to any one of claims 1 to 5, It is characterized in that The system includes: a data preprocessing module, a cluster analysis module, a quantization module, and a dequantization module; The data preprocessing module divides the weight matrix of the large language model into blocks and performs normalization processing; The cluster analysis module generates a quantization target code to ensure that the quantization center is compatible with the actual data distribution; The quantization module maps the normalized weight value to the quantization target code and records the corresponding index value; The dequantization module restores the original floating-point value according to the quantized index value to ensure the accuracy of the model during reasoning.
7. An application of a large language model weight quantization method as described in any one of claims 1-5, or a weight quantization system as described in claim 6 in model quantization deployed in edge computing systems, model quantization on cloud computing platforms, and large model fine-tuning deployment.
8. A hardware system for implementing the method according to any one of claims 1 to 5, It is characterized in that The hardware system comprises: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 5 is implemented.
9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
Citation Information
Cited By
Sliding fence product quantification method and device
CN120011695A
Large language model compression method and device, equipment and storage medium
CN120123803A
Weight quantization method and device based on hybrid segment coding and server
CN120996117A
Weight quantization method, device and server based on hybrid segment coding
CN120996117B
