Large language model ternary quantization and text generation method based on iterative parameter optimization

Abstract

The application provides a large language model ternary quantization method based on iterative parameter optimization and a text generation method, the quantization method comprising: inputting sample text into a pre-trained large language model, running a forward inference process, and collecting activation values of each quantization layer by using a callback function; for each weight matrix to be quantized, constructing an approximate weight composed of a scaling factor, a ternary matrix, and a bias; designing a judgment threshold, initializing the ternary matrix according to the judgment threshold; initializing the scaling factor and the bias; constructing an activation-aware error function based on calibration data, and iteratively updating the scaling factor, the bias, and the ternary matrix to minimize the activation-aware error; and generating quantized weights by using the optimized scaling factor, the bias, and the ternary matrix, to obtain a quantized large language model. The application makes the quantized weights as close as possible to the original weight distribution, thereby realizing high-precision ternarization without gradient training.

Description

Technical Field

[0001] This application relates to the field of post-training technology, specifically to a method for ternarizing a large language model and a text generation method based on iterative parameter optimization. Background Technology

[0002] Quantization techniques for large language models are an important method to reduce model energy consumption, improve inference efficiency, and enhance model versatility by representing the model's weights and activations with low-bit values. It is also one of the current research hotspots in the field of artificial intelligence. Among these, ternary quantization is an extremely low-bit quantization method that restricts weights to {-1, 0, 1} and approximates the original weight distribution through scaling factors and bias representations, achieving both model compression and acceleration. Currently, research on ternary quantization for large language models is relatively limited, especially in post-training quantization (PTQ) scenarios, where many challenges remain. Existing techniques such as TWN, GPTQ, and AWQ still struggle to maintain ideal performance under extremely low-bit quantization configurations, mainly because: On the one hand, ternary transformation under PTQ requires determining relevant parameters without the aid of training or gradient backpropagation, which places high demands on the determination and adjustment of parameters; On the other hand, ternary quantization, as an extremely low bit quantization, is itself affected by the data distribution. In block quantization, the differences between different columns may lead to inconsistent data distribution within the block, thereby introducing a large error.

[0003] A search revealed that Chinese patent application CN202211475716.8 discloses a Transformer quantization method, including post-training quantization, per-channel bit weight allocation, and parameter optimization based on minimizing reconstruction error through hierarchical activation. While this method can maintain high accuracy under conventional low bit conditions, it does not effectively address the significant accuracy degradation caused by weight distribution differences in extremely low bit (e.g., ternary) quantization. Summary of the Invention

[0004] To address one of the shortcomings of existing technologies, the purpose of this application is to provide a method for ternary quantization and text generation of large language models based on iterative parameter optimization.

[0005] The first aspect of this application provides a ternary quantization method for large language models based on iterative parameter optimization, comprising: Input the sample text into the pre-trained large language model, run the forward inference process, and use the callback function to collect the activation values ​​of each quantization layer as calibration data. For each weight matrix to be quantized, an approximate weight is constructed consisting of a scaling factor, a three-valued matrix, and a bias, wherein the elements of the three-valued matrix are taken from {-1, 0, 1}. Design a judgment threshold, and initialize the three-valued matrix according to the judgment threshold; Initialize the scaling factor and bias; Based on the calibration data, an activation-sensing error function is constructed, and the scaling factor, the bias, and the three-valued matrix are iteratively updated with the goal of minimizing the activation-sensing error. The quantized weights are generated using the iteratively updated scaling factor, the bias, and the three-valued matrix to obtain the quantized large language model.

[0006] Optionally, the approximate weights are expressed as in, This is an approximate weight matrix. Scaling factor For bias, It is a three-valued matrix.

[0007] Optionally, the design formula for the judgment threshold is:

[0008] in, Let be the threshold value for the i-th row. The centered weight matrix The element in the i-th row and j-th column, where m is the column number, is the decentralized weight matrix. , This indicates the initial bias.

[0009] Optionally, initializing the three-valued matrix according to the judgment threshold includes: The elements of the centered weight matrix are compared with the judgment threshold. If the element is greater than the judgment threshold, the corresponding position of the three-value matrix is ​​initialized to 1; If the element is less than the negative threshold, the corresponding position in the three-value matrix is ​​initialized to -1; If the element is between a negative threshold and the threshold, the corresponding position in the three-value matrix is ​​initialized to 0; During the iterative update process, the values ​​of the three-valued matrix change.

[0010] Optionally, it also includes rearranging the order of the weight matrix to be quantized, a process that occurs before iterative parameter optimization, specifically: The size of the quantization block is determined to be k columns, where k is an integer greater than or equal to 2; Clustering algorithms are used to perform cluster analysis on all column vectors of the weight matrix to identify columns with similar numerical distributions; The columns are rearranged based on the clustering results so that columns with similar distributions are adjacent, and the blocks are divided into multiple quantization blocks according to k columns per block. Generate a permutation matrix P representing the transformation relationship of the record column positions.

[0011] Optionally, the step of generating quantization weights using the iteratively updated scaling factor, the bias, and the three-valued matrix to obtain the quantized large language model includes: Save the iteratively updated scaling factor, bias, and permutation matrix; The three-valued matrix is ​​encoded and compressed to generate quantized weight data; The quantized weight data is integrated with the scaling factor, the bias, and the permutation matrix to generate the quantized large language model.

[0012] A second aspect of this application provides a text generation method based on a large language model, comprising: Obtain the natural language query text input by the user; Load the quantized large language model obtained by the ternary quantization method of the large language model based on iterative parameter optimization; The natural language query text is input into the quantized large language model for inference; Output the text generated by the quantized large language model.

[0013] Optionally, the reasoning process is: The input natural language query text is converted into an initial embedding vector, which is used as the input activation value for the first quantization layer. The input activation value of the current layer is input into the quantized large language model and propagated layer by layer forward. In each quantization layer, the input activation values ​​are ordered according to the permutation matrix P to match the order of the rearranged and stored quantization weights. The output activation value of this layer is calculated by using the matched input activation value and the quantization weight, and then used as the input activation value of the next layer. After completing the forward propagation of all layers, the final prediction result is output.

[0014] A third aspect of this application provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can be used to perform any of the methods described above.

[0015] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform any of the methods described above.

[0016] The large language model ternary quantization method based on iterative parameter optimization provided in this application introduces row-level scaling factors and biases, and takes quantization error as the optimization target. It combines iterative updates and discrete search methods to gradually optimize the scaling factors, biases and ternary weight matrices, so that the quantized weights are as close as possible to the original weight distribution, thereby achieving high-precision ternary quantization without gradient training.

[0017] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0018] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a large language model ternary quantization method based on iterative parameter optimization according to an exemplary embodiment; Figure 2 This is a schematic diagram illustrating the structure of a large language model ternaryization method based on iterative parameter optimization according to an exemplary embodiment; Figure 3 This is a flowchart illustrating a text generation method based on a large language model according to an exemplary embodiment. Detailed Implementation

[0019] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0020] In ternary low-bit quantization, it is difficult to determine the quantization parameters in post-training quantization (PTQ) scenarios. To address this issue, this application provides a large language model ternaryization method based on iterative parameter optimization to solve the aforementioned problems.

[0021] Reference Figure 1 and Figure 2 As shown in one embodiment of this application, a method for ternaryization of a large language model based on iterative parameter optimization is provided, including the following steps: S100: Input the sample text into the pre-trained large language model, run the forward inference process, and use the callback function to collect the activation values ​​of each quantization layer as calibration data. S200: For each weight matrix to be quantized, construct an approximate weight consisting of a scaling factor, a three-valued matrix, and a bias, where the elements of the three-valued matrix are taken from {-1, 0, 1}. S300, design the judgment threshold, and initialize the three-value matrix according to the judgment threshold; S400, initialize scaling factor and bias; S500 constructs an activation sensing error function based on calibration data, aiming to minimize the activation sensing error, and iteratively updates the scaling factor, bias, and three-valued matrix. S600 uses the iteratively updated scaling factor, bias, and three-valued matrix to generate quantized weights, thus obtaining the quantized large language model.

[0022] The above embodiments employ asymmetric ternary initialization and iterative updates, which can provide a reasonable starting point for ternary quantization without training and gradually reduce quantization errors through multiple rounds of updates, making the ternary weights closer to the original weight distribution, thereby improving the accuracy and stability under extremely low bit quantization.

[0023] This application employs an asymmetric ternary quantization method based on {-1, 0, 1}. Given the discreteness of these ternary values, a scaling factor and bias are introduced to construct approximate weights, thereby reducing the quantization error between the approximate weights and the original weight matrix to be quantized. In some specific embodiments of this application, the approximate weights in S200 are expressed as follows: ； in, This is an approximate weight matrix. Scaling factor For bias, It is a three-valued matrix.

[0024] The above embodiments reduce quantization error and maintain accuracy by fitting the weight distribution with scaling factors and biases, thereby achieving low-bit, high-efficiency inference.

[0025] To correctly represent the original weights to be quantized as three values, this application designs a judgment threshold and initializes the three-value matrix based on the judgment threshold. In some specific embodiments of this application, S300, designing the judgment threshold and initializing the three-value matrix based on the judgment threshold specifically involves: S301, assuming a uniform weight distribution, we can obtain an empirical approximation conclusion that the quantization effect is better when the judgment threshold is set to the following values:

[0026] Let be the threshold value for the i-th row. These are the elements of the centered weight matrix, where m is the number of columns; The decentralized weight matrix is Where μ is the initial bias.

[0027] S302, compare the elements of the centered weight matrix with the judgment threshold; If an element is greater than the threshold, the corresponding position in the three-value matrix is ​​initialized to 1. If an element is less than the negative threshold, the corresponding position in the three-value matrix is ​​initialized to -1. If an element is between the negative threshold and the threshold, the corresponding position in the three-value matrix is ​​initialized to 0.

[0028] In the above embodiment, by setting a judgment threshold based on the statistical characteristics of the weight distribution, and performing a three-value division on the decentralized weight matrix based on this, the initial three-value matrix can better reflect the magnitude relationship of the original weights, thereby reducing the initial quantization error and improving the convergence stability of the subsequent parameter optimization process.

[0029] In some specific embodiments of this application, S400, initializing the scaling factor and bias includes the following steps: S401: Initialize the scaling factor. To minimize the error between the quantized weights and the original weights after initialization, the scaling factor is initialized as follows:

[0030] in Let be the scaling factor for the i-th row. These are the elements of the centered weight matrix. Let m be the elements of the ternary matrix, where m is the number of columns. S402: Initialize the bias, setting the bias for each row. Set as the row mean of the weight matrix.

[0031] In the above embodiments, by reasonably initializing the scaling factor and bias of each row, the quantization weights are made as close as possible to the original weight distribution in the initial stage. This initialization method provides a stable and accurate starting point for subsequent iterative optimization, thereby effectively improving the accuracy and stability of ternary quantization even without training.

[0032] To obtain the optimal scaling factor and bias, in some specific embodiments of this application, in step S500, based on calibration data, an activation sensing error function is constructed, and iterative optimization is performed with the goal of minimizing the activation sensing error to obtain the optimized scaling factor, bias, and three-valued matrix. This can be achieved through the following steps: S501, considering the impact of quantization weights on model output, introduce activation-aware error. ,in These are the outputs of the full-precision model and the corresponding layer of the quantized model, respectively.

[0033] S502, with To optimize the objective, iterative updates are performed through multiple rounds of differentiation. In each round, the three-valued matrix is ​​determined by searching for each element in {-1, 0, 1} with the objective of minimizing the quantization error. The search is performed alternately with continuous updates to the scaling factor and bias.

[0034] The above embodiments combine iterative updates and discrete search methods to gradually optimize α, μ, and the ternary weight matrix, making the quantized weights as close as possible to the original weight distribution, thereby achieving high-precision ternary quantization without gradient training.

[0035] To address the issue of significant quantization errors caused by varying weight distributions within blocks during ternary block quantization, this application proposes a matrix rearrangement strategy based on column-to-column structural similarity in the structural rearrangement section. Some specific embodiments of this application further include sequentially rearranging the weight matrix to be quantized, specifically as follows: Before iterative updates, the size of the quantization block is determined to be k columns, where k is an integer greater than or equal to 2; Clustering algorithms are used to perform cluster analysis on all column vectors of the weight matrix to identify columns with similar numerical distributions; The columns are rearranged based on the clustering results so that columns with similar distributions are adjacent, and the blocks are divided into multiple quantization blocks according to k columns per block. Generate a permutation matrix P representing the transformation relationship of the record column positions.

[0036] The above embodiments, by performing K-means clustering on the columns, aggregate similar columns into the same quantization block, thereby reducing intra-block variance and mitigating the impact of outliers on quantization accuracy, resulting in more uniform and easily represented quantized weights. This rearrangement technique, combined with iterative optimization, works synergistically to achieve high-precision, low-bit ternary quantization without gradient training. However, using iterative optimization alone is less effective than combining both methods because iterative optimization reduces the error between the quantized weights and the original weights, while matrix rearrangement reduces intra-block variance and isolates outliers, allowing iterative optimization to act on a more uniform data distribution, further reducing quantization error and improving accuracy and stability under extremely low-bit quantization.

[0037] After the above steps and weight rearrangement, the optimized scaling factor, bias, ternary matrix, and permutation matrix are obtained. In some specific embodiments of this application, the above parameters are stored, and quantization encoding is performed on each quantization block to generate the final quantization weights for subsequent quantization inference.

[0038] Based on the same technical concept, other embodiments of this application propose a text generation method based on a large language model, such as... Figure 3 As shown, it includes: S1, Obtain the natural language query text input by the user; Specifically, natural language query text refers to any human language prompts or instructions that users input into the quantization model when they are actually using it. These can be in the form of a single sentence, paragraph, or question and answer.

[0039] S2, Load the quantized large language model obtained by the iterative parameter optimization-based ternary quantization method of the large language model according to any one of the terms; Specifically, pre-trained large language models include LLaMA-7B, LLaMA-13B, LLaMA-65B, Qwen3-14B, etc.

[0040] Specifically, the ternary quantization method for large language models based on iterative parameter optimization includes the following steps: S21, Input the sample text into the pre-trained large language model, run the forward inference process, and use the callback function to collect the activation values ​​of each quantization layer as calibration data; S22, For each weight matrix to be quantized, construct an approximate weight consisting of a scaling factor, a three-valued matrix, and a bias, where the elements of the three-valued matrix are taken from {-1, 0, 1}; S23, Design the judgment threshold, and initialize the three-value matrix according to the judgment threshold; S24, initialize scaling factor and bias; S25. Based on the calibration data, an activation sensing error function is constructed. With the goal of minimizing the activation sensing error, the scaling factor, bias, and three-valued matrix are iteratively updated. S26. Quantization weights are generated using the iteratively updated scaling factor, bias, and three-valued matrix to obtain the quantized large language model.

[0041] The steps in the above examples of this application can be implemented by referring to the steps of the large language model ternary method based on iterative parameter optimization in the above embodiments, and will not be repeated here.

[0042] S3 inputs the natural language query text into the quantized large language model for inference; S4 outputs the text generated by the quantized large language model.

[0043] Specifically, the text represents the answers or content generated by the model. It is the final deliverable of the quantitative model's practical application, representing the actual value that the technical solution ultimately provides to users (such as intelligent question answering, content creation, etc.). For example, when a user enters the query text "How to quickly relieve work stress?", the response text generated by the Quantization Large Language Model is: "Methods to relieve work stress include: 1. Taking short breaks and deep breathing; 2. Prioritizing tasks; 3. Moderate exercise and sufficient sleep. It is recommended to try the Pomodoro Technique to improve efficiency." The above embodiments offer significant hardware adaptation advantages. By loading a large language model optimized with ternary quantization, the model weights are compressed from high-precision floating-point numbers to low-bit ternary data, significantly reducing model storage requirements and GPU memory bandwidth pressure during inference. Simultaneously, the sparsity of ternary weights simplifies matrix operations, drastically reducing hardware computational load and power consumption. This enables the efficient deployment and operation of large-scale language models on resource-constrained hardware devices, such as consumer-grade graphics cards with limited GPU memory (e.g., NVIDIA GeForce RTX 3060, RTX 4090), power-sensitive mobile SoCs (e.g., Qualcomm Snapdragon 8 Gen 3, Apple M-series chips), and edge computing devices (e.g., NVIDIA Jetson series). This method improves the real-time performance and hardware compatibility of text generation tasks, enabling the practical application of large models on the terminal side.

[0044] In some specific embodiments of this application, during the inference process, the input text is propagated layer by layer through the quantization model. At each quantization layer, column permutation P is first applied to adjust the weights and activations, and then the quantized weights are directly used for calculation to obtain the quantized activation output for that layer. After the layer-by-layer forward propagation is completed, the prediction result of the final quantization model is output, achieving efficient inference under low-bit quantization.

[0045] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0046] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0047] Example 1: Inference Optimization of Large Language Models Based on Low-Bit Ternary Quantization Step 1: Collect calibration data The input text is subjected to one or more complete forward inferences. The activation values ​​of each quantization layer are collected as calibration data through a callback function, which is used for subsequent ternary quantization and quantizer parameter optimization.

[0048] Step 2: Weighted Block Rearrangement and Clustering The weight matrix is ​​divided into quantization blocks by column (128 columns per block by default). Clustering methods are used to group columns with similar numerical distributions into the same block to generate a permutation matrix, reducing channel differences within the block and improving ternary accuracy.

[0049] Step 3: Asymmetric ternary initialization Asymmetric ternary initialization is performed on the weights of each layer to be quantized in the model. A scaling factor and bias are introduced for each row, and a ternary matrix is ​​initialized to ensure that the quantization covers the weight distribution.

[0050] Step 4: Iterative optimization Considering the impact of quantization on the model output, the quantization error function is used as the optimization objective. The scaling factor and bias are iteratively optimized, and the three-valued matrix is ​​determined by grid search, thereby gradually reducing the quantization error.

[0051] Step 5: Generate quantization weights The ternary parameters and permutation matrix obtained in the above steps are used to perform quantization on each quantization block, generate the final quantization weights and save them for subsequent inference.

[0052] Step 6: Quantization Model Inference The input text is propagated layer by layer through the quantization model. In the quantization layer, a permutation matrix is ​​first applied to adjust the weights and activations, and then the output is calculated using the quantization weights. After completing the forward propagation layer by layer, the final prediction result is obtained, achieving low-bit-efficiency inference.

[0053] Experimental results To verify the feasibility of the proposed method, it was applied to several large language models and compared with existing methods (AWQ, GPTQ, Slim-LLM, PB-LLM): Perplexity Level (PPL) metric: Taking LLaMA-13B as an example: the PPL of this method is 9.21, which is significantly better than GPTQ's 18.97 and AWQ's 2.30e5.

[0054] Zero-shot average accuracy: Taking LLaMA-65B as an example, the average accuracy of this method is 55.67%, which is better than GPTQ's 54.13% and AWQ's 32.12%.

[0055] Model storage compression (using LLaMA-7B as an example): The model size compression effect is significant: the original FP16 model is 13.48GB, and after quantization by this method, it is only 1.79GB, achieving a compression of about 7.17×, which is better than GPTQ (2.19GB, 6.16×) and Slim-LLM (2.30GB, 5.86×).

[0056] This application enables low-bit quantization on large language models, significantly improving inference efficiency while maintaining high prediction accuracy. It can be widely applied to text generation, question answering, and multimodal tasks. Experimental results show that this method outperforms existing methods in terms of model compression ratio, quantization time, and zero-shot task accuracy, fully verifying its practicality and efficiency advantages.

[0057] Based on the same technical concept, in other embodiments of this application, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to perform the above-described method.

[0058] Based on the same technical concept, in other embodiments of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, can be used to perform the above-described method.

[0059] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc., and the aforementioned computer programs and computer instructions can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0060] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.

[0061] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.

[0062] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0063] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A ternary quantization method for large language models based on iterative parameter optimization, characterized in that, include: Input the sample text into the pre-trained large language model, run the forward inference process, and use the callback function to collect the activation values ​​of each quantization layer as calibration data. For each weight matrix to be quantized, an approximate weight is constructed consisting of a scaling factor, a three-valued matrix, and a bias, wherein the elements of the three-valued matrix are taken from {-1, 0, 1}. Design a judgment threshold, and initialize the three-valued matrix according to the judgment threshold; Initialize the scaling factor and bias; Based on the calibration data, an activation-sensing error function is constructed, and the scaling factor, the bias, and the three-valued matrix are iteratively updated with the goal of minimizing the activation-sensing error. The quantized weights are generated using the iteratively updated scaling factor, the bias, and the three-valued matrix to obtain the quantized large language model.

2. The ternary quantization method for large language models based on iterative parameter optimization according to claim 1, characterized in that, The approximate weight is expressed as: in, This is an approximate weight matrix. Scaling factor For bias, It is a three-valued matrix.

3. The ternary quantization method for large language models based on iterative parameter optimization according to claim 1, characterized in that, The design formula for the judgment threshold is: ； in, Let be the threshold value for the i-th row. The centered weight matrix The element in the i-th row and j-th column, where m is the column number, is the decentralized weight matrix. , This indicates the initial bias.

4. The ternary quantization method for large language models based on iterative parameter optimization according to claim 3, characterized in that, The initialization of the three-valued matrix based on the judgment threshold includes: The elements of the centered weight matrix are compared with the judgment threshold. If the element is greater than the judgment threshold, the corresponding position of the three-value matrix is ​​initialized to 1; If the element is less than the negative threshold, the corresponding position in the three-value matrix is ​​initialized to -1; If the element is between a negative threshold and the threshold, the corresponding position in the three-value matrix is ​​initialized to 0; During the iterative update process, the values ​​of the three-valued matrix change.

5. The ternary quantization method for large language models based on iterative parameter optimization according to claim 1, characterized in that, This also includes rearranging the order of the weight matrix to be quantized, a process that occurs before iterative parameter optimization, specifically: The size of the quantization block is determined to be k columns, where k is an integer greater than or equal to 2; Clustering algorithms are used to perform cluster analysis on all column vectors of the weight matrix to identify columns with similar numerical distributions; The columns are rearranged based on the clustering results so that columns with similar distributions are adjacent, and the blocks are divided into multiple quantization blocks according to k columns per block. Generate a permutation matrix P representing the transformation relationship of the record column positions.

6. The ternary quantization method for large language models based on iterative parameter optimization according to claim 5, characterized in that, The step of generating quantized weights using the iteratively updated scaling factor, the bias, and the three-valued matrix to obtain the quantized large language model includes: Save the iteratively updated scaling factor, bias, and permutation matrix; The three-valued matrix is ​​encoded and compressed to generate quantized weight data; The quantized weight data is integrated with the scaling factor, the bias, and the permutation matrix to generate the quantized large language model.

7. A text generation method based on a large language model, characterized in that, include: Obtain the natural language query text input by the user; The natural language query text is input into a quantized large language model for inference; The quantized large language model is the quantized large language model obtained by the method according to any one of claims 1-6; Output the text generated by the quantized large language model.

8. The text generation method based on a large language model according to claim 7, characterized in that, The reasoning process is as follows: The input natural language query text is converted into an initial embedding vector, which is used as the input activation value for the first quantization layer. The input activation value of the current layer is input into the quantized large language model and propagated layer by layer forward. In each quantization layer, the input activation values ​​are ordered according to the permutation matrix P to match the order of the rearranged and stored quantization weights. The output activation value of this layer is calculated by using the matched input activation value and the quantization weight, and then used as the input activation value of the next layer. After completing the forward propagation of all layers, the final prediction result is output.

9. A terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it can be used to perform the method of any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, this program can be used to perform the method of any one of claims 1-8.

Images