Large model quantitative compression method and system based on polar coordinate system

Abstract

The invention belongs to the technical field of artificial intelligence, and provides a large model quantification compression method and system based on a polar coordinate system in order to solve the problem that an existing large language model quantification method cannot guarantee low-cost hardware deployment and reasoning efficiency of a large model at the same time. The large model quantitative compression method based on the polar coordinate system comprises the steps of calling original weight parameter distribution of a pre-stored large language model from a first storage unit, processing the original weight parameter distribution into standard Gaussian distribution, obtaining a corresponding weight vector conforming to the standard Gaussian distribution, and then converting the weight vector into corresponding polar coordinate representation; and respectively constructing a direction codebook and a modular length codebook represented by each polar coordinate, obtaining a vector codebook conforming to the distribution characteristic and the quantitative sensitivity characteristic, and transmitting the vector codebook to a second storage unit for storage. The low-cost hardware deployment and reasoning efficiency of a large model can be guaranteed at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of artificial intelligence, and particularly relates to a large model quantization and compression method and system based on a polar coordinate system. Background Art

[0002] The statements in this part merely provide background technical information related to the present invention and do not necessarily constitute prior art.

[0003] Large Language Model (LLM) is a natural language processing model based on deep learning. Its huge number of parameters poses great challenges to efficient inference and practical deployment. Post Training Quantization (PTQ) methods (e.g., using Vector Quantization (VQ)) can transform the parameters of the large language model from high-precision floating-point values to low-precision integer values to effectively compress the model size. Traditional VQ methods directly perform clustering and optimization on the original data distribution through Mean Square Error (MSE) in the Cartesian coordinate system and use the cluster centers as new vector representations. It does not analyze the characteristics of the original data distribution, and the effect of the quantization model largely depends on the iterative optimization effect of the clustering algorithm; moreover, in the weights of the large language model, the direction of the vector is more sensitive to the quantization process than the magnitude of the vector.

[0004] In summary, the existing large language model quantization methods cannot simultaneously ensure low-cost hardware deployment and inference efficiency of large models. Summary of the Invention

[0005] To solve the technical problems existing in the above background art, the present invention provides a large model quantization and compression method and system based on a polar coordinate system, which can simultaneously ensure low-cost hardware deployment and inference efficiency of large models.

[0006] To achieve the above object, the present invention adopts the following technical solutions: The first aspect of the present invention provides a large model quantization and compression method based on a polar coordinate system.

[0007] A large model quantization and compression method based on a polar coordinate system, which includes: Retrieve the original weight parameter distribution of the pre-stored large language model from the first storage unit and process it into a standard Gaussian distribution to obtain corresponding weight vectors that conform to the standard Gaussian distribution; Convert each of the weight vectors that conform to the standard Gaussian distribution into corresponding polar coordinate representations; Construct the direction codebook and magnitude codebook for each of the polar coordinate representations respectively, obtain a vector codebook that conforms to the distribution characteristics and quantization sensitivity characteristics, and transmit it to the second storage unit for storage; Among them, the storage format of the vector codebook in the second storage unit is direction index, magnitude index, and element value; the direction index and magnitude index are concatenated to form a final index, and the final index is integrated in units of the original weight parameters of the large language model to form a quantized model weight matrix; the quantized model weight matrix is used to provide to the computing unit to run the large language model on the computing unit.

[0008] As an implementation, the process of processing the original weight parameter distribution of the large language model into a standard Gaussian distribution includes: Calculate the covariance matrix of the original weight parameters of the large language model; Perform a Karhunen-Loeve transform on the covariance matrix to obtain an enhanced matrix; Multiply the enhanced matrix by the Hadamard matrix for fusion to obtain an enhanced rotation matrix; Among them, the parameter distributions in the enhanced rotation matrix are the standard Gaussian distribution.

[0009] As an implementation, the weight vector that conforms to the standard Gaussian distribution contains two variables, namely: direction and magnitude.

[0010] As an implementation, the process of constructing the direction codebook for each of the polar coordinate representations includes: Select the vector directions of the E8 lattice at a given magnitude as the base class; Use the greedy algorithm to select a specified number of subsets from the base class according to the quantization bits of the direction to obtain the direction codebook.

[0011] As an implementation, the process of constructing the magnitude codebook for each of the polar coordinate representations includes: According to the root distribution of the chi-square distribution of the weight vector that conforms to the standard Gaussian distribution, obtain the distribution of the magnitude, and then calculate the probability density function of the magnitude; Quantize the magnitude using the Llyod-MAX algorithm according to the probability density function of the magnitude to form the magnitude codebook.

[0012] As an implementation, the process of quantizing the magnitude using the Llyod-MAX algorithm is: Select the initial quantization grid points and quantization boundaries; Calculate the centroid and boundaries in turn according to the initial quantization grid points and quantization boundaries; Determine whether the Lloyd-MAX algorithm converges. If the change in quantization distortion between two adjacent iterations is less than a preset threshold or the preset number of iterations is reached, stop the iteration; otherwise, continue to calculate the centroid and boundary iteratively until the Lloyd-MAX algorithm converges, and finally obtain the quantized codebook lattice points.

[0013] The second aspect of the present invention provides a large model quantization and compression system based on the polar coordinate system.

[0014] A large model quantization and compression system based on the polar coordinate system includes: A weight parameter distribution standardization module, which is used to retrieve the original weight parameter distribution of the large language model pre-stored in the first storage unit and process it into a standard Gaussian distribution to obtain a corresponding weight vector that conforms to the standard Gaussian distribution; A weight parameter polar coordinate conversion module, which is used to convert each of the weight vectors that conform to the standard Gaussian distribution into a corresponding polar coordinate representation; A vector codebook construction and storage module, which is used to construct a direction codebook and a magnitude codebook for each of the polar coordinate representations respectively, obtain a vector codebook that conforms to the distribution characteristics and quantization sensitivity characteristics, and transmit it to the second storage unit for storage; Among them, the storage format of the vector codebook in the second storage unit is direction index, magnitude index, and element value; the direction index and the magnitude index are spliced to form a final index, and the final index is integrated in units of the original weight parameters of the large language model to form a quantized model weight matrix; the quantized model weight matrix is used to provide to the computing unit to run the large language model on the computing unit.

[0015] The third aspect of the present invention provides a computer-readable storage medium.

[0016] A computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the steps in the above-mentioned large model quantization and compression method based on the polar coordinate system.

[0017] The fourth aspect of the present invention provides a computer program product.

[0018] A computer program product includes a computer program / instructions, and when the computer program / instructions are executed by a processor, it implements the steps in the above-mentioned large model quantization and compression method based on the polar coordinate system.

[0019] The fifth aspect of the present invention provides an electronic device.

[0020] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the large model quantization compression method based on the polar coordinate system as described above are implemented.

[0021] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention is applicable to low-bit quantization compression of a large language model. The distribution of the original weight parameters of the large language model is reshaped into a Gaussian distribution, and the vector direction and modulus are separated by a polar coordinate system. An independently optimized direction codebook and modulus codebook are constructed, thereby obtaining a vector codebook that conforms to the distribution characteristics and quantization sensitivity characteristics. The vector codebook is only stored in the format of direction index, modulus index and element value storage, thereby avoiding the quantization error instability problem caused by clustering effect differences in traditional methods. While improving the performance of the compression model, it also ensures low-cost hardware deployment of the large language model.

[0022] (2) The polar coordinate quantization framework proposed in the present invention supports independent bit allocation for direction and modulus length. It alleviates its high quantization sensitivity by increasing the proportion of directional bits and reduces redundant storage of modulus length, thus reducing the memory usage of the codebook exponentially. This breaks through the problem of mismatch between quantization sensitivity and quantization method caused by the coupling of direction and modulus length in traditional methods.

[0023] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 It is a comparison diagram of the quantization of the direction and modulus of the weight parameter vector in the LLaMA-2-70B model using the existing vector quantization method; Figure 2 is a flow chart of a large model quantization compression method based on a polar coordinate system according to an embodiment of the present invention; Figure 3 is a process diagram of processing the original weight parameter distribution of a large language model into a standard Gaussian distribution according to an embodiment of the present invention; Figure 4 (a) is a diagram showing the effect of the traditional Hadamard rotation; FIG4( b ) is a diagram showing the enhanced Hadamard rotation effect proposed by the present invention; Figure 5 is a process diagram of constructing a directional codebook represented by polar coordinates according to an embodiment of the present invention; Figure 6 It is a process diagram for constructing a magnitude codebook for polar coordinate representation in an embodiment of the present invention; Figure 7 It is a schematic diagram of vector quantization direction error and magnitude error in an embodiment of the present invention; Figure 8 It is a schematic diagram of polar coordinate quantization lattice points in an embodiment of the present invention; Figure 9 It is a schematic diagram of the structure of a large model quantization and compression system based on a polar coordinate system in an embodiment of the present invention. Detailed implementation manners

[0026] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.

[0028] It should be noted that the terms used herein are only for describing specific implementation manners and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0029] Term explanation: The large language model (LLM) is a natural language processing model based on deep learning. It forms an ultra-large neural network through training with a vast amount of text data, with billions to trillions of parameters, and is capable of understanding semantics, generating coherent text, and performing various language tasks. For example, storing the parameters of the LLaMA-2-70B model in the FP16 format requires 140GB of memory, which exceeds the capacity of high-performance GPUs and makes multi-GPU deployment difficult. Such a large scale has a significant impact on memory capacity and hard disk storage, and requires a large amount of bandwidth during inference. Among them, the LLaMA-2-70B model is a known large language model with 70 billion parameters.

[0030] Figure 1 The effects of quantifying the direction and magnitude of the weight parameter vectors in the LLaMA-2-70B model using existing vector quantization methods are compared. From Figure 1It can be seen that under the experimental settings of various quantization bit numbers (the number of cluster centers), the accuracy loss of direction quantization is greater than that of magnitude quantization. However, the existing measurement index MSE (mean squared error) is more sensitive to the change of magnitude, which is contrary to the actual situation. And the codebook of the existing method couples the direction and magnitude of the vector, and cannot adjust the weights of the direction and magnitude in the VQ method, ultimately resulting in the inability to ensure both low-cost hardware deployment and inference efficiency of large models.

[0031] Figure 2 The process of the large model quantization and compression method based on the polar coordinate system according to the embodiment of the present invention is given. Combining Figure 2 with this, the large model quantization and compression method based on the polar coordinate system according to the embodiment of the present invention specifically includes the following steps S201 to S203.

[0032] S201: Retrieve the original weight parameter distribution of the pre-stored large language model from the first storage unit, and process it into a standard Gaussian distribution to obtain the corresponding weight vector that conforms to the standard Gaussian distribution.

[0033] First, the deep learning framework defaults to using the Gaussian distribution for network parameter initialization. Second, the normalization layers in the deep neural network, such as batch normalization (Batch Norm) and layer normalization (Layer Norm), further stabilize the mean and variance, making the weights and activation values close to the Gaussian distribution. Finally, L2 regularization is a commonly used technique in the training of most large language models, and its prior distribution is essentially a Gaussian distribution, which strengthens the effectiveness of modeling the weight parameters with the Gaussian distribution.

[0034] The standard Hadamard rotation cannot align the variances between channels, which also limits its ability to transform the original distribution into a Gaussian distribution. This embodiment proposes an enhanced Hadamard rotation method to process the original weight parameter distribution of the large language model into a standard Gaussian distribution. As Figure 3 shown, the process of processing the original weight parameter distribution of the large language model into a standard Gaussian distribution includes: S2011: Calculate the covariance matrix of the original weight parameters of the large language model; Given a model weight parameter matrix (meeting the following assumption: the columns of the matrix are zero-mean) , whose dimension is (n, m), the Hadamard transformation matrix H has the dimension (m, m), and the covariance matrix of the transformation matrix can be expressed as formula (1): (1); where K represents the eigenvector matrix obtained by eigenvalue decomposition, Denote the diagonal eigenvalue matrix, and the superscript T represents the matrix transpose operation. Then, The -th dimension of the diagonal can be expressed by Equation (2): (2); Where: is -th diagonal value of the dimension. For a given value, Equation (2) represents the variance of the -th channel.

[0035] S2012: Perform the Karhunen-Loeve transform on the covariance matrix to obtain an enhanced matrix; Since the vector varies with l, the channel variances cannot be numerically proven to be close. In addition, considering that H is a fixed matrix while K and are input-related, the Hadamard transform cannot adjust for channel differences in all scenarios. Therefore, this property of the prior art will inevitably lead to differences in variances between channels, resulting in suboptimal Gaussian distribution conversion capabilities.

[0036] An enhanced Hadamard rotation method based on the Karhunen-Loeve transform (KLT) is used to align the variances between channels of the model weight parameters and enhance the Gaussianization effect of the rotation.

[0037] Specifically, perform eigenvalue decomposition on the covariance matrix to apply KLT, as shown in Equation (3): (3); S2013: Multiply the enhanced matrix by the Hadamard matrix for fusion to obtain an enhanced rotation matrix; Among them, the parameter distributions in the enhanced rotation matrix are the standard Gaussian distribution.

[0038] Calculate the KLT enhanced rotation matrix HK, as shown in Equation (4): (4); Equation (1) is transformed into Equation (5): (5); Where, represents the identity matrix.

[0039] Correspondingly, Equation (2) is transformed into Equation (6): (6); In this way, the variance of each channel becomes the same, which can make the distribution after rotation transformation closer to the Gaussian distribution. Figure 4(a) is the effect diagram of traditional Hadamard rotation, and Figure 4(b) is the effect diagram of the enhanced Hadamard rotation proposed by the present invention. By comparison, it can be found that the method proposed in the embodiments of the present invention can further reduce outliers and make the quantization errors of more data smaller, reducing the overall quantization error.

[0040] Rotation can, to a certain extent, suppress the outliers of the original weight parameters and increase the representation ability of the codebook. At the same time, it is easy to solve the vector direction and modulus length that conform to the Gaussian distribution, which is helpful for the construction of the direction and modulus length codebooks. In this embodiment, an enhancement matrix is introduced before Hadamard rotation, which can align the variances between different channels of the original weight parameters, reduce outliers to a certain extent, and is more conducive to Hadamard rotation to process the original weight parameters into a standard Gaussian distribution. When the distribution characteristics of the quantization variables are known, the quantization lattice points that conform to the original distribution can be theoretically deduced.

[0041] S202: Convert each of the weight vectors that conform to the standard Gaussian distribution into corresponding polar coordinate representations.

[0042] The direction of the weight vector of the large language model is more sensitive to quantization than the modulus length (because the direction has a higher dimension, higher spatial freedom, and higher compression loss in the case of low bits). As Figure 1 shown, the embodiments of the present invention design a set of control verification experiments to perform clustering on the directions and modulus lengths of the model weight parameter vectors at different bit widths respectively, and then evaluate the average value of the zero-shot accuracy of the quantization model on multiple data sets. The results show that under the same bit width setting, the quantization accuracy of the modulus length is higher than that of the direction quantization, and as the quantization bit decreases, the accuracy of the direction quantization drops more significantly, while the accuracy of the modulus length quantization still remains close to floating point.

[0043] However, the existing clustering methods based on (mean square error) are more sensitive to changes in the modulus length. The first reason is that is essentially a measure of the relative distance between spaces. It only covers the consideration of direction because it is artificially stipulated in VQ that the starting points of all vectors are at the origin. From the formula form, and the calculation of the modulus length ( ) have a similar form, as shown in formulas (7) and (8): (7); (8); where and respectively represent the original vector to be quantized and the clustering center, represents the dimension of the vector. On the other hand, can also be expressed by formula (9): (9); where and respectively represent and the magnitudes of represents and the included angle between represents the cosine function. If is expressed in the form of ( represents the change in magnitude), then formula (9) can be further evolved into: (10); It can be seen from formula (10) that the influence of the magnitude error on MSE is of the square level. In the case of clustering, is usually small, and under the Taylor expansion is approximated to , so the influence of the direction on MSE is of the linear level. In addition, the embodiment of the present invention also conducts the following verification experiments: statistically calculate the direction error of all vector quantizations as shown in formula (11) and the magnitude error as shown in formula (12): (11); (12); where: represents the direction error of vector quantization; represents the magnitude error; represents the sine function, represents the absolute value operation.

[0044] The errors in direction and magnitude are reflected by a visualization method in Figure 7 . It is found that in almost 100% of the cases, the direction error is three times the magnitude error, which shows that the traditional clustering method is more inclined to make the magnitudes approximate.

[0045] To solve the above problems, this embodiment proposes a vector quantization method using polar coordinate representation, which converts the vector representation in the original Cartesian coordinate system into a combination of direction and magnitude, and quantizes the respective sets of direction and magnitude separately. This method can allocate more clustering centers for the direction, so as to adapt to the extremely high quantization sensitivity of the direction. For any dimensional vector , its polar coordinate representation is magnitude and direction : (13); (14).

[0046] When constructing the vector codebook, quantization is performed on the direction and magnitude respectively; when performing quantization coding, the optimal direction and the optimal magnitude are selected from the direction codebook and the magnitude codebook respectively, and combined into an optimal vector.

[0047] In this embodiment, the direction and magnitude of the vector codebook are separated and quantized independently, reducing the mutual interference between each other. After Gaussianization, the direction and magnitude of the weight parameters of the large language model are independent random variables, so quantization in polar coordinates conforms to its distribution characteristics. In addition, the importance ratio can be adjusted by adjusting the number of bits allocated to the direction and magnitude in the index to adapt to the different sensitivities of the direction and magnitude to quantization.

[0048] S203: Construct the direction codebook and the magnitude codebook of each of the polar coordinate representations respectively, obtain a vector codebook that conforms to the distribution characteristics and quantization sensitivity characteristics, and transmit it to the second storage unit for storage; Among them, the storage format of the vector codebook in the second storage unit is the direction index, the magnitude index, and the element value; the direction index and the magnitude index are concatenated to form a final index, and the final index is integrated in units of the original weight parameters of the large language model to form a quantized model weight matrix; the quantized model weight matrix is used to be provided to the calculation unit to run the large language model on the calculation unit.

[0049] Among them, the directions of the Gaussian distribution are uniform on each magnitude, so it is necessary to construct a direction codebook with uniform spatial directions. The definition of direction uniformity is: maximizing the minimum adjacent angle within this set of directions. The present invention proposes to use a greedy algorithm based on the E8 lattice directions to construct a spatially uniform direction codebook. The E8 lattice is the only positive definite, even, unimodular lattice in 8-dimensional space, and its definition requires that all coordinates are integers or half-integers and the sum is even, as shown in formula (15): (15); Where: represents the eight-dimensional integer set, is used to represent any vector; represents taking the union; represents taking the intersection.

[0050] It has self-duality, that is, its structure completely coincides with its dual lattice, and this property stems from the unimodular property (the determinant is ±1). In the sphere packing problem, the E8 lattice achieves the maximum density packing in 8-dimensional space. At a given magnitude, the directions from the origin to the E8 lattice points are spatially uniform. However, its quantity may not necessarily meet the requirements of the number of bits.

[0051] As Figure 5 shown, the process of constructing the direction codebook for each of the polar coordinate representations includes: S20311: Select the vector directions of the E8 lattice at a given modulus length as the base class; S20312: Use the greedy algorithm to select a specified number of subsets from the base class according to the quantization bits of the directions, that is, obtain the direction codebook.

[0052] Among them, the process of using the greedy algorithm to select a specified number of subsets from the base class is: Step a1: Initialize a selected set A and add any one direction; Step a2: Take the remaining directions as the candidate set B; Step a3: Select a direction from the set B and add it to the set A, and the sum of the angles between this direction and all the directions in the set A is the largest; Step a4: Repeat step a3 until the number of directions in the set A meets the requirements.

[0053] This embodiment improves the uniformity and selection flexibility of the directions. The E8 lattice is the optimal spherical packing scheme in eight-dimensional space. Existing schemes use the vectors inside the E8 lattice within a limited modulus length as the codebook. However, this approach ignores the phase problem between the vector directions at different modulus lengths and cannot represent as many directions as possible to the greatest extent. In contrast, taking the vector directions of the E8 lattice at a given modulus length improves the difference between the centers of the direction codebook and enhances the modeling ability in terms of directions. In addition, the number of directions of E8 at each type of modulus length is fixed. Using the greedy algorithm helps to improve the flexibility of direction bit selection and is conducive to adaptively coordinating the important relationship between directions and modulus lengths.

[0054] Considering that the modulus length of the model weight parameter vector after Gaussianization conforms to the root distribution of the chi-square distribution, the present invention proposes a Lloyd-MAX algorithm based on distribution characteristics. As Figure 6 shown, the process of constructing the modulus length codebook for each of the polar coordinate representations includes: S20321: According to the fact that the weight vector conforming to the standard Gaussian distribution conforms to the root distribution of the chi-square distribution, obtain the distribution of the modulus length, and then calculate the probability density function of the modulus length.

[0055] The definition of the chi-square distribution with degrees of freedom is the sum of the squares of standard Gaussian random variables, so its root distribution is the distribution of the modulus length. Suppose there is a modulus length variable such that The chi-square distribution. First, the probability density function (PDF) of the chi-square distribution is: (16); where y > 0, is the gamma function, and e is the natural number. Since and is non-negative, the PDF of can be derived by variable transformation. The transformation function is: (17); Its inverse function is: (18); According to the variable transformation formula, we have: (19); where: is the derivative of .

[0056] where is 's PDF. Substituting formula (18) and its derivative, we get: (20); Substitute with , we obtain: (21); Solving for gives the expression: (22); Substituting into formula (16) gives: (23); Formula (23) describes the distribution of the norm of the Gaussianized model weight parameter vector.

[0057] S20322: Quantize the norm according to the probability density function of the norm using the Llyod-MAX algorithm to form a norm codebook.

[0058] Among them, the process of quantizing the norm using the Llyod-MAX algorithm is as follows: Step b1: Select the initial quantization grid points and quantization boundaries; select the initial quantization grid points and quantization boundaries .

[0059] Step b2: Calculate the centroid and boundaries successively according to the initial quantization lattice points and quantization boundaries; The calculation formula for the centroid is: ; The calculation formula for the boundary is: ; Step b3: Determine whether the Lloyd-MAX algorithm converges. If the change in quantization distortion between two adjacent iterations is less than a preset threshold or the preset number of iterations is reached, stop the iteration; otherwise, return to Step b2 to continue the iteration until the Lloyd-MAX algorithm converges, and finally obtain the quantized codebook lattice points.

[0060] As Figure 8 shown, the codebook lattice points in a two-dimensional space are taken as an example. Figure 8 In it, the blue line represents the direction of spatial uniformity, and the orange line represents the modulus length of the root distribution that conforms to the chi-square distribution.

[0061] It should also be noted here that the first storage unit and the second storage unit can be implemented using other memories such as disks or computer storage spaces (such as memory).

[0062] In this embodiment, only the direction index, modulus length index, and element values of the model weight vector are stored in the second storage unit. This reduces the space occupied by model storage, and can effectively reduce the model storage requirement to 12.5% of the original model. For example, a 7B model only requires 1.7 GB of disk space to complete model storage.

[0063] During the deployment process of the quantized model, the following steps can be included: Load the quantized model from the second storage unit into the GPU (Graphics Processing Unit) video memory; Construct the direction and modulus length codebooks and load them into the L1 cache (primary cache) of the computing unit; According to the model inference process, sequentially load the indexes from the computing unit video memory into the L1 cache (primary cache); Divide the indexes into direction indexes and modulus length indexes according to the bit positions and left-right positions of the direction and modulus length; Decode the direction and modulus length from the direction and modulus length codebooks respectively according to the direction and modulus length indexes; Send the direction and modulus length into the computing unit for operation with the activation value.

[0064] In this embodiment, during the process of quantized model inference, the direction and modulus length codebooks are only loaded into the L1 cache of the GPU once. During each inference process, only the direction and modulus length indexes of the vector are loaded from the video memory into the L1 cache for table lookup, and then the table lookup results are sent into the computing unit for operation with the activation.

[0065] This embodiment does not require moving the codebook between the L1 cache and the GPU video memory multiple times, and transmits information through indexes, reducing the memory access time of the system and improving the inference efficiency of large models (for example, about 8 times of inference acceleration can be achieved on the LLaMA-2-70B model).

[0066] Figure 9 The schematic structural diagram of the large model quantization and compression system based on the polar coordinate system according to the embodiment of the present invention is given. In combination with Figure 9 , a large model quantization and compression system based on the polar coordinate system according to an embodiment of the present invention at least includes: A weight parameter distribution standardization module 901, which is used to retrieve the original weight parameter distribution of the large language model pre-stored in the first storage unit, and process it into a standard Gaussian distribution to obtain a corresponding weight vector conforming to the standard Gaussian distribution; A weight parameter polar coordinate conversion module 902, which is used to convert each of the weight vectors conforming to the standard Gaussian distribution into corresponding polar coordinate representations; A vector codebook construction and storage module 903, which is used to construct a direction codebook and a magnitude codebook for each of the polar coordinate representations respectively, obtain a vector codebook conforming to the distribution characteristics and quantization sensitivity characteristics, and transmit it to the second storage unit for storage; Among them, the storage format of the vector codebook in the second storage unit is direction index, magnitude index, and element value; the direction index and the magnitude index are spliced to form a final index, and the final index is integrated in units of the original weight parameters of the large language model to form a quantized model weight matrix; the quantized model weight matrix is used to be provided to the computing unit to run the large language model on the computing unit.

[0067] It should be noted here that each module in the large model quantization and compression system based on the polar coordinate system according to the embodiment of the present invention corresponds one by one to each step in the above-mentioned large model quantization and compression method based on the polar coordinate system, and the specific implementation process is the same, which will not be elaborated here.

[0068] The large model quantization and compression method and system based on the polar coordinate system of this embodiment first use an enhanced Hadamard rotation method to process the original weight parameter distribution into an approximately standard Gaussian distribution, so that the morphological representation of the distribution is solvable; secondly, represent the rotated weight parameter vector in polar coordinates, and propose to quantize the direction and magnitude respectively to adapt to the quantization sensitivity difference between the direction and the magnitude; finally, on the basis of considering the data distribution characteristics after rotation, design construction schemes for the direction codebook and the magnitude codebook respectively, so that the codebook distribution approximately conforms to the original distribution and reduces the quantization error; through the storage and loading methods of the quantization model of the present invention, the deployment and inference efficiency of the large model are improved.

[0069] In one or more embodiments, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps in the large model quantization compression method based on the polar coordinate system as described above are implemented. Figure 2 shown.

[0070] In one or more embodiments, a computer program product is provided, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the steps in the large model quantization compression method based on the polar coordinate system as described above are implemented. Figure 2 shown.

[0071] In one or more embodiments, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps in the large model quantization compression method based on the polar coordinate system as described above are implemented. Figure 2 shown.

[0072] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processors of general-purpose computers, special-purpose computers, embedded processors, or other programmable data processing devices to generate a machine, so that the instructions executed by the processors of the computer or other programmable data processing devices generate means for implementing the specified functions in one Figure 1 or more flows and / or blocks Figure 1 or more blocks.

[0073] The above are only preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A large model quantization compression method based on polar coordinate system, characterized in that: include: Retrieving the original weight parameter distribution of the pre-stored large language model from the first storage unit, and processing it into a standard Gaussian distribution, to obtain a corresponding weight vector that conforms to the standard Gaussian distribution; Convert each of the weight vectors that conform to the standard Gaussian distribution into corresponding polar coordinate representations; Constructing the direction codebook and the modulus length codebook represented by the polar coordinates respectively, obtaining a vector codebook that meets the distribution characteristics and quantization sensitivity characteristics and transmitting the codebook to the second storage unit for storage; The storage format of the vector codebook in the second storage unit is direction index, modulus index and element value; The directional index and the modulus index are concatenated to form a final index, and the final index is integrated in units of the original weight parameters of the large language model to form a quantized model weight matrix; the quantized model weight matrix is ​​used to provide to a computing unit to run the large language model on the computing unit.

2. The large model quantization compression method based on polar coordinate system according to claim 1, characterized in that: The process of processing the original weight parameter distribution of the large language model into a standard Gaussian distribution includes: Calculate the covariance matrix of the original weight parameters of the large language model; Performing Karhunen-Loeve transformation on the covariance matrix to obtain an enhanced matrix; The enhancement matrix is ​​multiplied with the Hadamard matrix to obtain an enhancement rotation matrix; The distribution of each parameter in the enhanced rotation matrix is ​​a standard Gaussian distribution.

3. The large model quantization compression method based on polar coordinate system according to claim 1, characterized in that: The weight vector conforming to the standard Gaussian distribution includes two variables: direction and modulus.

4. The large model quantization compression method based on polar coordinate system according to claim 1, characterized in that: The process of constructing the direction codebook represented by each polar coordinate includes: The vector direction of the E8 lattice at a given module length is selected as the base class; According to the number of quantization bits of the direction, a specified number of subsets are selected from the base class using a greedy algorithm to obtain a directional codebook.

5. The large model quantization compression method based on polar coordinate system according to claim 1, characterized in that: The process of constructing the modulus length codebook represented by each polar coordinate includes: According to the root distribution of the weight vector that conforms to the standard Gaussian distribution and the chi-square distribution, the distribution of the modulus length is obtained, and then the probability density function of the modulus length is calculated; According to the probability density function of the modulus length, the Llyod-MAX algorithm is used to quantize the modulus length to form a modulus length codebook.

6. The large model quantization compression method based on polar coordinate system according to claim 5, characterized in that: The process of quantifying the modulus length using the Llyod-MAX algorithm is as follows: Selecting initial quantization grid points and quantization boundaries; According to the initial quantization grid points and quantization boundaries, the centroid and the boundaries are calculated in turn; Determine whether the Llyod-MAX algorithm has converged. If the change in quantization distortion between two adjacent iterations is less than a preset threshold or reaches a preset number of iterations, stop the iteration; otherwise, continue to iterate the centroid and boundary until the Llyod-MAX algorithm converges and finally obtains the quantized codebook grid.

7. A large model quantization compression system based on polar coordinate system, characterized in that: include: A weight parameter distribution standardization module, which is used to retrieve the original weight parameter distribution of the pre-stored large language model from the first storage unit, and process it into a standard Gaussian distribution to obtain a corresponding weight vector that conforms to the standard Gaussian distribution; A weight parameter polar coordinate conversion module, which is used to convert each weight vector that conforms to the standard Gaussian distribution into a corresponding polar coordinate representation; A vector codebook construction storage module, which is used to respectively construct the direction codebook and the modulus length codebook represented by each polar coordinate, obtain a vector codebook that meets the distribution characteristics and quantization sensitivity characteristics, and transmit it to the second storage unit for storage; The storage format of the vector codebook in the second storage unit is direction index, modulus index and element value; The directional index and the modulus index are concatenated to form a final index, and the final index is integrated in units of the original weight parameters of the large language model to form a quantized model weight matrix; the quantized model weight matrix is ​​used to provide to a computing unit to run the large language model on the computing unit.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the large model quantization compression method based on the polar coordinate system as described in any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps in the large model quantization compression method based on the polar coordinate system as described in any one of claims 1 to 6 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the large model quantization compression method based on the polar coordinate system as described in any one of claims 1 to 6 are implemented.

Images