Large language model quantification method and device, equipment, storage medium and product

Abstract

The invention provides a large language model quantification method and device, equipment, a storage medium and a product. The quantification method comprises the following steps: pruning a weight matrix of a large language model; performing fine adjustment on the big language model of which the weight matrix is pruned by utilizing a verification set; the elements of the weight matrix of the large language model after fine tuning are divided into T groups, T = 2n-1, and n represents quantization bit width; and generating a codebook with the length of 2n based on the T groups, and quantizing the weight matrix by using the codebook. According to the quantization method disclosed by the invention, the sparse property of the large language model is fully utilized, and the reasonable quantization codebook is determined based on real data distribution, so that the quantization error is reduced, and the quantized model performance is ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning technology, and in particular to a large language model quantization method, device, equipment, storage medium and product. Background Art

[0002] In the field of Natural Language Processing (NLP), large language models based on the Transformer architecture have achieved remarkable results and have become the core model in this field. However, large language models usually have a large number of parameters, which leads to a sharp increase in storage costs and inference delays. In order to alleviate these problems, many research works have begun to develop and apply low-bit quantization technology for model weights.

[0003] The core technology of low-bit quantization is to determine the appropriate quantization step size and quantization codebook. Existing low-bit quantization technologies can be divided into two categories: uniform quantization and non-uniform quantization. Generally speaking, uniform quantization uses low-bit integer data as the codebook, such as AWQ (Activation-aware Weight Quantization), GPTQ (Generative Pre-trained Transformer-Q), SmoothQuant, etc. Uniform quantization is very sensitive to the quantization step size, and the optimization direction lies in how to find the optimal quantization step size. Non-uniform quantization directly uses low-bit floating-point numbers as the codebook, such as FP8 (using 8-bit floating-point numbers to represent model parameters and activation values), FP4 (using 4-bit floating-point numbers to represent model parameters and activation values), NF4 (4-bit quantization method based on normalized floating-point numbers), etc., which is more robust to the quantization step size.

[0004] However, the existing quantization methods do not consider the sparse characteristics of large language models, and the codebooks near the zero point have serious redundancy. When the quantization bit width is reduced to 4 bits or less, the waste of codebooks seriously affects the performance of the quantized model. In addition, the quantization codebooks used in the current quantization methods are pre-defined and do not take into account the actual data distribution. The quantization error increases sharply with the reduction of the quantization bit width, and the performance of the quantized model decreases significantly.

[0005] The content of this background technology description is only for facilitating understanding of the relevant technology in this field and is not regarded as an admission of the prior art. Summary of the invention

[0006] In view of this, the present disclosure provides a large language model quantization method, apparatus, device, storage medium and product, which fully utilize the sparse nature of the large language model and determine a reasonable quantization codebook based on the actual data distribution, thereby reducing the quantization error and ensuring the performance of the quantized model.

[0007] In a first aspect, the present disclosure provides a large language model quantization method, the quantization method comprising:

[0008] Prune the weight matrix of a large language model;

[0009] Using the validation set, fine-tuning the large language model after the weight matrix has been pruned;

[0010] The elements of the weight matrix of the fine-tuned large language model are divided into T groups, where T = 2 n -1, n represents the quantization bit width;

[0011] Based on the T groups, a length of 2 is generated n A codebook is used to quantize the weight matrix.

[0012] Furthermore, the weight matrix of the large language model is pruned, including:

[0013] Normalizing each element in the weight matrix using the element with the largest absolute value in the weight matrix;

[0014] The elements in the weight matrix whose absolute values ​​are smaller than the pruning threshold after normalization are set to zero.

[0015] Furthermore, the elements of the weight matrix of the fine-tuned large language model are divided into T groups, including:

[0016] Deleting the elements with a value of zero in the weight matrix, and sorting the remaining elements from small to large;

[0017] Divide the sorted elements into T groups and number them from 1 to T.

[0018] Further, after dividing the elements of the weight matrix of the fine-tuned large language model into T groups, the quantization method further includes:

[0019] Determine whether there is a group in the T groups that has both positive and negative elements;

[0020] If the judgment result is that it does not exist, the T groups are kept unchanged, otherwise:

[0021] Determine the number k of the group whose elements are both positive and negative;

[0022] Dividing the sorted elements into a negative array and a positive array;

[0023] The negative array is equally divided into k groups, and the positive array is equally divided into Tk groups, to obtain T new groups to replace the T groups.

[0024] Further, based on the T groups, a length of 2 is generated. n The codebook includes:

[0025] Calculate the mean of each group of elements and obtain T means;

[0026] The length is 2, which is composed of T means and 0. n The codebook.

[0027] In addition, quantizing the weight matrix using the codebook includes:

[0028] For 2 in the codebook n Number the codes;

[0029] For each element in the weight matrix, the serial number of the code closest to the element in the codebook is stored to achieve quantization of the weight matrix.

[0030] In a second aspect, the present disclosure provides a large language model quantization device, the quantization device comprising:

[0031] A pruning unit, configured to prune a weight matrix of a large language model;

[0032] A fine-tuning unit is configured to use a validation set to fine-tune the large language model after the weight matrix is ​​pruned;

[0033] A grouping unit is configured to divide the elements of the weight matrix of the fine-tuned large language model into T groups, where T=2 n -1, n represents the quantization bit width;

[0034] A quantization unit is configured to generate a quantization unit of length 2 based on the T groups. n A codebook is used to quantize the weight matrix.

[0035] In a third aspect, the present disclosure further provides an electronic device, comprising a memory, a processor, and a computer program / instructions stored on the memory, wherein the processor executes the computer program / instructions to implement the large language model quantization method of the present disclosure.

[0036] In a fourth aspect, the present disclosure further provides a computer-readable storage medium having a computer program / instruction stored thereon, which implements the large language model quantization method of the present disclosure when the computer program / instruction is executed by a processor.

[0037] In a fifth aspect, the present disclosure further provides a computer program product, including a computer program / instruction, which implements the large language model quantization method of the present disclosure when executed by a processor.

[0038] In the large language model quantization method used in the present invention, the sparse characteristics of the large language model are used to perform unstructured pruning on the model, thereby achieving the purpose of saving codebooks near the zero point and solving the problem of codebook waste near the zero point; on this basis, the true distribution of weights is used to generate a more accurate codebook, which reduces the quantization error of the large language model and ensures the performance of the quantized large language model.

[0039] Other optional features and technical effects of the embodiments of the present disclosure are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. The elements shown are not limited by the proportions shown in the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements, wherein:

[0041] Figure 1 An example flow chart of a large language model quantization method according to an embodiment of the present disclosure is shown;

[0042] Figure 2 An exemplary structural diagram of a large language model quantization device according to an embodiment of the present disclosure is shown;

[0043] Figure 3 An exemplary structural diagram of an electronic device capable of implementing the method according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present disclosure more clear, the present disclosure is further described in detail below in conjunction with specific implementation methods and drawings. Here, the illustrative implementation methods and descriptions of the present disclosure are used to explain the present disclosure, but are not intended to limit the present disclosure.

[0045] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0046] The core idea of ​​the present disclosure is to utilize the sparse characteristics of a large language model to perform unstructured pruning on the model to save codebooks near the zero point, and to utilize the true distribution of weights to generate a more accurate codebook to reduce the quantization error of the large language model.

[0047] Figure 1 An exemplary process of a large language model quantization method according to an embodiment of the present disclosure is shown.

[0048] like Figure 1 As shown, the quantification method includes:

[0049] Step S1: Pruning the weight matrix of the large language model;

[0050] Step S2: fine-tuning the large language model after the weight matrix is ​​pruned using the validation set;

[0051] Step S3: Divide the elements of the weight matrix of the fine-tuned large language model into T groups, where T = 2 n -1, n represents the quantization bit width;

[0052] Step S4: Generate a 2-bit integer based on the T groups. n A codebook is used to quantize the weight matrix.

[0053] The large language model in this disclosure refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. The large language model performs well in a variety of application scenarios. It can not only perform simple language tasks such as spell checking and grammar correction, but also handle complex tasks such as text summarization, machine translation, sentiment analysis, dialogue generation and content recommendation. By pre-training on large-scale data sets, the large language model has acquired powerful general modeling and generalization capabilities. Recently, large language models have also begun to be gradually applied to specific fields such as finance, medical care and education.

[0054] The weight matrix is ​​a very important component of the large language model. The weight matrix refers to the weights connecting the layers in the large language model, that is, each element in the weight matrix represents the connection weights between the layers. During the training process of the large language model, these weights are adjusted to optimize the performance of the large language model. The size of the weight matrix depends on the number of nodes in each layer. For example, if the input layer has 3 nodes and the hidden layer has 4 nodes, the weight matrix is ​​a 3×4 matrix containing 12 elements. The large language model used as the quantization object in the present disclosure is a trained large language model, that is, a large language model whose weight matrix has been adjusted.

[0055] In a specific embodiment of the present disclosure, the weight matrix of the large language model is pruned, including: using the element with the largest absolute value in the weight matrix to normalize each element in the weight matrix; setting the elements in the weight matrix whose absolute values ​​after normalization are less than a pruning threshold to zero.

[0056] Pruning is to prune some redundant connections in a large language model. Since the connections between neurons in a neural network model are mathematically represented as weight matrices, the specific implementation of pruning is to change some elements in the weight matrix to zero elements. A weight matrix with a large number of zero elements after pruning is called a sparse matrix, and conversely, a weight matrix with most of the elements being non-zero is called a dense matrix. The present disclosure uses pruning technology to solve the problem of codebook waste near the zero point.

[0057] For example, assuming that the pruning threshold is δ, all elements of the weight matrix of the large language model are x0, x1, x2, ... m , where the element with the largest absolute value is x max In the pruning process, first use x max For x0, x1, x2, ... x m Normalization is performed, and the calculation method can be, for example, x' i =x i / |x max |(i=0,1,2……m), thus converting all elements of the weight matrix to the interval [-1,1]. Next, all the elements x'0, x'1, x'2……x' m The elements whose absolute values ​​are less than the pruning threshold δ are set to zero, thus completing the pruning process of the weight matrix of the large language model.

[0058] In order to ensure the performance of the large language model after the weight matrix is ​​pruned, the large language model after the weight matrix is ​​pruned can be fine-tuned using the validation set. For example, when the large language model is applied to sentiment analysis, the SST-2 dataset can be used to construct a validation set. Fine-tuning the large language model after the weight matrix is ​​pruned can be understood as fine-tuning the element values ​​of the weight matrix of the pruned large language model.

[0059] In a specific embodiment of the present disclosure, the elements of the weight matrix of the fine-tuned large language model are divided into T groups, including: deleting the elements with a value of zero in the weight matrix, and sorting the remaining elements from small to large; dividing the sorted elements into T groups, and numbering them from 1 to T in sequence. Where T = 2 n-1, n represents the quantization bit width. The smaller the quantization bit width n, the higher the quantization degree of the large language model, the smaller the size of the quantized model, and the more storage space can be saved. The quantization bit width n can usually be 2, 4, 8, etc. according to needs.

[0060] For example, assuming that the quantization bit width n=2, then T=3. In addition, assuming that there are 12 elements in the weight matrix of the fine-tuned large language model, each element is represented by w ij , that is, w ij represents the element in the i-th row and j-th column of the weight matrix, i = 1, 2, 3, j = 1, 2, 3, 4, then all elements of the weight matrix of the large language model are {w 11 ,w 12 ,w 13 ,w 14 ,w 21 ,w 22 ,w 23 ,w 24 ,w 31 ,w 32 ,w 33 ,w 34}. Assume that the element w in row 1 and column 2 is 12 , the element w in the 2nd row and 1st column 21 , the element w in the 3rd row and 4th column 34 is zero, then the remaining elements after deleting these three zero elements are: 11 ,w 13 ,w 14 ,w 22 ,w 23 ,w 24 ,w 31 ,w 32 ,w 33}. Sort these elements in ascending order. Assume that all the elements after sorting are: {w 22 ,w 23 ,w 24 ,w 11 ,w 32 ,w 33 ,w 13 ,w 14 ,w 31}. Divide all the elements sorted above into T = 3 groups: {w 22 ,w 23 ,w 24}、{w 11 ,w 32 ,w 33}、{w 13 ,w 14 ,w 31}, and number these three groups 1, 2, and 3 respectively.

[0061] In a specific embodiment of the present disclosure, after the elements of the weight matrix of the fine-tuned large language model are divided into T groups, the quantization method further includes: judging whether there is a group with both positive and negative elements in the T groups; if the judgment result is that there is no group, the T groups are kept unchanged, otherwise: determining the number k of the group with both positive and negative elements; dividing the sorted elements into a negative array and a positive array; dividing the negative array into k groups, and dividing the positive array into Tk groups, to obtain new T groups to replace the T groups. In this way, it can be ensured that all elements of each group have the same sign.

[0062] Continuing with the previous example, we divide the weight matrix of the fine-tuned large language model into three groups {w 22 ,w 23 ,w 24}、{w 11 ,w 32 ,w 33}、{w 13 ,w 14 ,w 31}, assuming that the elements in each of the three groups are all positive or all negative, then it is judged that there is no group with both positive and negative elements in the three groups, so keep the three groups {w 22 ,w 23 ,w 24}、{w 11 ,w 32 ,w 33}、{w 13 ,w 14 ,w 31} remains unchanged. Otherwise, when it is determined that there is a group with both positive and negative elements in these three groups, regroup them as follows. Assume that group number 2 {w 11 ,w 32 ,w 33} 11 is a negative number, w 32 and w 33 is a positive number, then k = 2. 22 ,w 23 ,w 24 ,w 11 ,w 32 ,w 33 ,w 13 ,w 14 ,w 31 All negative numbers in} are divided into negative array W-: {w 22 ,w 23 ,w 24 ,w 11}, and divide all positive numbers into positive arrays W + : {w 32 ,w 33 ,w 13 ,w 14 ,w 31 For the negative array W-, divide it into k = 2 groups: {w 22 ,w 23}、{w 24 ,w 11}, for a positive array W + , and divide it into Tk=1 groups: {w 32 ,w 33 ,w 13 ,w 14 ,w 31}. Thus, we get three new groups {w 22 ,w 23}、{w 24 ,w 11}、{w 32 ,w 33 ,w 13 ,w 14 ,w 31} to replace the original three groups {w 22 ,w 23 ,w 24}、{w 11 ,w 32 ,w 33}、{w 13 ,w 14 ,w 31}.

[0063] Note that, during the grouping process, if it is not possible to evenly divide the data into T groups, Tk groups, or k groups, the number of elements in the last group can be made smaller than the number of elements in other groups, or the number of elements in the first group can be made larger than the number of elements in other groups. 22 ,w 23 ,w 24 ,w 11 ,w 32 ,w 33 ,w 13 ,w 14 ,w 31} is divided into 5 groups, then the grouping result can be {w 22 ,w 23}、{w 24 ,w 11}、{w 32 ,w 33}、{w 13 ,w 14}、{w 31}, the last group {w 31} has fewer elements than the first four groups; if you need to replace {w 22 ,w 23 ,w 24 ,w 11 ,w 32 ,w 33 ,w 13 ,w 14 ,w 31} is divided into 7 groups, then the grouping result can be {w 22 ,w 23 ,w 24}、{w 11}、{w 32}、{w 33}、{w 13}、{w 14}、{w 31}, the number of elements in the first group is greater than the number of elements in the following six groups.

[0064] In a specific embodiment of the present disclosure, a length of 2 is generated based on the T groups. n The codebook includes: calculating the mean of each group of elements to obtain T means; forming a codebook of length 2 by using T means and 0 n Since the elements of the T groups are sorted from small to large, the T means are also sorted from small to large. Simply insert 0 into the boundary between negative and positive numbers in the T means to obtain a codebook sorted from small to large, from negative to 0 and then to positive. In this way, a quantized codebook is obtained based on the real data distribution of the weight matrix of the large language model, which reduces the quantization error and ensures the performance of the quantized model.

[0065] Still using the previous example, assume that the elements of the weight matrix of the fine-tuned large language model are regrouped to obtain three new groups {w 22 ,w 23}、{w 24 ,w 11}、{w 32 ,w 33 ,w 13 ,w 14 ,w 31}. Calculate the mean of each group of elements, that is, the mean of the elements in each group: w1 = (w 22 +w 23 ) / 2, w2=(w 24 +w 11 ) / 2, w3=(w 32 +w 33 +w 13 +w 14 +w 31) / 5. According to the previous assumption, the first two groups {w 22 ,w 23} and {w 24 ,w 11} are all negative, so their respective means w1 and w2 are also negative, and the third group {w 32 ,w 33 ,w 13 ,w 14 ,w 31} are all positive numbers, so its mean w3 is also positive. The three means w1, w2 and w3 and 0 are combined to form a matrix of length 2 n =4 codebook [w1,w2,0,w3], the elements in the codebook are also sorted in order from small to large, from negative to 0 and then to positive.

[0066] In a specific implementation of the present disclosure, the weight matrix is ​​quantized using the codebook, including: quantizing the 2 n codes are numbered; for each element in the weight matrix, the number of the code in the codebook that is closest to the element is stored to achieve quantization of the weight matrix. In this quantization method, when storing a large language model, it is not necessary to store a large-scale weight matrix, but only to store the number and codebook corresponding to each element in the weight matrix, which greatly saves storage resources. When applying the quantized large language model, the code corresponding to each element in the weight matrix can be found in the codebook as an approximate value of the element according to the number corresponding to the element in the weight matrix.

[0067] Still using the previous example. Assume that all elements of the weight matrix of the fine-tuned large language model are {w 11 ,w 12 ,w 13 ,w 14 ,w 21 ,w 22 ,w 23 ,w 24 ,w 31 ,w 32 ,w 33 ,w 34}, the resulting codebook is [w1,w2,0,w3]. n = 4 codes [w1,w2,0,w3] are numbered [0,1,2,3] respectively. Assume that all elements of the codebook corresponding to the weight matrix {w 11 ,w 12 ,w 13 ,w 14 ,w 21 ,w 22 ,w 23 ,w24 ,w 31 ,w 32 ,w 33 ,w 34} are {w2,0,w3,w3,0,w1,w1,w2,w3,w3,w3,0}, then all elements of the weight matrix {w 11 ,w 12 ,w 13 ,w 14 ,w 21 ,w 22 ,w 23 ,w 24 ,w 31 ,w 32 ,w 33 ,w 34} is quantized into the code {1,2,3,3,2,0,0,1,3,3,3,2} of the code {w2,0,w3,w3,0,w1,w1,w2,w3,w3,w3,0} and stored. When applied to the large language model, the approximation of the original weight matrix {w2,0,w3,w3,0,w1,w1,w2,w3,w3,w3,0} can be obtained only based on the stored quantization weights {1,2,3,3,2,0,0,1,3,3,3,2} and the codebook [w1,w2,0,w3] and used as the weight matrix of the large language model.

[0068] In the large language model quantization method used in the present invention, the sparse characteristics of the large language model are used to perform unstructured pruning on the model, thereby achieving the purpose of saving codebooks near the zero point and solving the problem of codebook waste near the zero point; on this basis, the true distribution of weights is used to generate a more accurate codebook, which reduces the quantization error of the large language model and ensures the performance of the quantized large language model.

[0069] Figure 2 FIG. 2 shows an exemplary structural block diagram of a large language model quantization device according to an embodiment of the present disclosure. Figure 2 As shown, the quantization device includes a pruning unit, a fine-tuning unit, a grouping unit and a quantization unit.

[0070] A pruning unit is configured to prune the weight matrix of the large language model; a fine-tuning unit is configured to fine-tune the large language model after the weight matrix is ​​pruned using a validation set; a grouping unit is configured to divide the elements of the weight matrix of the fine-tuned large language model into T groups, where T=2 n -1, n represents the quantization bit width; the quantization unit is configured to generate a quantization bit width of length 2 based on the T groups. n A codebook is used to quantize the weight matrix.

[0071] The specific functions of each unit of the quantization device refer to the large language model quantization method according to the embodiment of the present disclosure as described above. In some embodiments, the large language model quantization device can be combined with the features of the large language model quantization method of any embodiment, and vice versa, which will not be repeated here.

[0072] In an embodiment of the present disclosure, an electronic device is provided, including: a memory, a processor, and a computer program / instruction stored in the memory, wherein the processor executes the computer program / instruction to implement the large language model quantization method described in the embodiment of the present disclosure.

[0073] Figure 3 A schematic diagram of an electronic device 1000 that can implement a method or implement an embodiment of the present disclosure is shown, and in some embodiments, more or fewer electronic devices than shown may be included. In some embodiments, it can be implemented using a single or multiple electronic devices. In some embodiments, it can be implemented using cloud or distributed electronic devices.

[0074] like Figure 3 As shown, the electronic device 1000 includes a processor 1001, which can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) 1002 or the programs and / or data loaded from the storage part 1008 to the random access memory (RAM) 1003. Processor 1001 can be a multi-core processor, or it can include multiple processors. In some embodiments, processor 1001 can include a general main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 1003, various programs and data required for the operation of electronic device 1000 are also stored. Processor 1001, ROM 1002 and RAM 1003 are connected to each other via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0075] The processor and the memory are used together to execute the computer program / instructions stored in the memory. When the computer program / instructions are executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0076] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, a touch screen, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program / instruction read therefrom is installed into the storage section 1008 as needed. Figure 3 Only some components are schematically shown in the figure, which does not mean that the electronic device 1000 only includes Figure 3 Components shown.

[0077] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.

[0078] Although not shown, in an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the large language model quantization method described in the embodiment of the present disclosure is implemented.

[0079] The storage media in the embodiments of the present disclosure include permanent and non-permanent, removable and non-removable items that can be implemented by any method or technology to store information. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0080] Although not shown, the embodiment of the present disclosure further provides a computer program product, including: a computer program / instruction, which implements the large language model quantization method described in the embodiment of the present disclosure when the computer program / instruction is executed by a processor.

[0081] The methods, programs, systems, devices, etc. of the embodiments of the present disclosure may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of the present specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.

[0082] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.

[0083] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present disclosure do not have to be performed in a specific order and can still achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0084] In this article, multiple embodiments of the present disclosure are described, but for the sake of brevity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the embodiments may be omitted. In this article, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are meant to be applicable to at least one embodiment or example according to the present disclosure, rather than all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.

[0085] The exemplary systems and methods of the present disclosure have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present disclosure as defined in the appended claims.

Claims

1. A large language model quantization method, characterized in that: The quantification method comprises: Prune the weight matrix of a large language model; Using the validation set, fine-tuning the large language model after the weight matrix has been pruned; The elements of the weight matrix of the fine-tuned large language model are divided into T groups, where T = 2 n -1, n represents the quantization bit width; Based on the T groups, a length of 2 is generated n A codebook is used to quantize the weight matrix.

2. The large language model quantization method according to claim 1, characterized in that: Prune the weight matrix of the large language model, including: Normalizing each element in the weight matrix using the element with the largest absolute value in the weight matrix; The elements in the weight matrix whose absolute values ​​are smaller than the pruning threshold after normalization are set to zero.

3. The large language model quantization method according to claim 2, characterized in that: The elements of the weight matrix of the fine-tuned large language model are divided into T groups, including: Deleting the elements with a value of zero in the weight matrix, and sorting the remaining elements from small to large; Divide the sorted elements into T groups and number them from 1 to T.

4. The large language model quantization method according to claim 3, characterized in that: After dividing the elements of the weight matrix of the fine-tuned large language model into T groups, the quantization method further includes: Determine whether there is a group in the T groups that has both positive and negative elements; If the judgment result is that it does not exist, the T groups are kept unchanged, otherwise: Determine the number k of the group whose elements are both positive and negative; Dividing the sorted elements into a negative array and a positive array; The negative array is equally divided into k groups, and the positive array is equally divided into Tk groups, to obtain T new groups to replace the T groups.

5. The large language model quantization method according to claim 4, characterized in that: Based on the T groups, a length of 2 is generated n The codebook includes: Calculate the mean of each group of elements and obtain T means; The length is 2, which is composed of T means and 0. n The codebook.

6. The large language model quantization method according to any one of claims 1 to 5, characterized in that: Quantizing the weight matrix using the codebook includes: For 2 in the codebook n Number the codes; For each element in the weight matrix, the serial number of the code closest to the element in the codebook is stored to achieve quantization of the weight matrix.

7. A large language model quantization device, characterized in that: The quantization device comprises: A pruning unit, configured to prune a weight matrix of a large language model; A fine-tuning unit is configured to use a validation set to fine-tune the large language model after the weight matrix is ​​pruned; A grouping unit is configured to divide the elements of the weight matrix of the fine-tuned large language model into T groups, where T=2 n -1, n represents the quantization bit width; A quantization unit is configured to generate a quantization unit of length 2 based on the T groups. n A codebook is used to quantize the weight matrix.

8. An electronic device comprising a memory, a processor, and a computer program / instruction stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the large language model quantization method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the large language model quantization method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the large language model quantization method according to any one of claims 1 to 6 is implemented.

Images