Large text generation model quantization method and device, electronic equipment and storage medium

Abstract

The application provides a large text generation model quantization method and device, electronic equipment and storage medium, the method comprises the steps of: quantizing the benchmark text generation model based on each candidate quantization bit number, and replacing the i-th layer model parameter of the benchmark text generation model with the i-th layer model parameter of each single precision quantization model obtained, to obtain a plurality of candidate quantization models corresponding to the i-th layer; determining the sensitivity of each candidate quantization bit number to the i-th layer based on the effect index of the benchmark text generation model and the effect index of each candidate quantization model corresponding to the i-th layer; based on the sensitivity of each layer to each candidate quantization bit number, a target function is constructed and solved to determine the target quantization bit number of each layer from each candidate quantization bit number, and the model parameters of each layer in the benchmark text generation model are updated based on the target quantization bit number of each layer to obtain a quantized target text generation model. The application reduces the amount of calculation and ensures the quantization accuracy of the model.

Description

Technical Field

[0001] This invention relates to the field of model quantization technology, and in particular to a method, apparatus, electronic device, and storage medium for quantizing large-scale text generation models. Background Technology

[0002] Text generation model quantization refers to using a smaller number of bits to represent model parameters, thereby reducing the storage space required for these parameters. For example, if the original text generation model uses 16 bits to represent model parameters, converting it to use 4 bits can significantly reduce the space occupied by the model parameters.

[0003] Currently, GPTQ (Accurate Post-Training Quantization for Generative Pre-trained Transformers) is often used to quantize text generation models. This involves dividing the model parameters into several groups, quantizing a portion of the parameters in each group to obtain the quantization error, and then updating the unquantized parameters based on the quantization error. After quantizing each group of parameters, the quantization error is calculated, and the unquantized model parameters are updated based on the quantization error. Finally, all model parameters are quantized. This method is computationally intensive and time-consuming. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for quantizing large-scale text generation models, in order to solve the defects of excessive computational load and excessive time consumption in the quantization of text generation models in the prior art.

[0005] This invention provides a method for quantizing large-scale text generation models, comprising:

[0006] Multiple candidate quantization bit numbers are determined, and the benchmark text generation model is quantized based on each candidate quantization bit number to obtain multiple single-precision quantization models;

[0007] The i-th layer model parameters of the baseline text generation model are replaced with the i-th layer model parameters of each single-precision quantization model to obtain multiple candidate quantization models corresponding to the i-th layer.

[0008] Based on the performance metrics of the benchmark text generation model and the performance metrics of each candidate quantization model corresponding to the i-th layer, the sensitivity of the i-th layer to the number of candidate quantization bits is determined.

[0009] Based on the sensitivity of each layer to each candidate number of quantization bits, an objective function is constructed and the objective function is solved to determine the target number of quantization bits for each layer from each candidate number of quantization bits.

[0010] The model parameters of each layer in the baseline text generation model are updated using the model parameters of the corresponding single-precision quantization model based on the target quantization bit count of each layer, resulting in the quantized target text generation model.

[0011] According to a quantization method for a large-scale text generation model provided by the present invention, the construction of an objective function based on the sensitivity of each layer to the number of candidate quantization bits includes:

[0012] Based on the sensitivity of each layer to each candidate quantization bit number, and the sensitivity of each layer to the maximum candidate quantization bit number, the profit function in the objective function is constructed.

[0013] The cost function in the objective function is constructed based on the number of candidate quantization bits and the number of bits in the basic storage unit.

[0014] According to a quantization method for a large-scale text generation model provided by the present invention, the step of solving the objective function and determining the target quantization bit number for each layer from each candidate quantization bit number includes:

[0015] Maximize the benefit corresponding to the benefit function, and ensure that the cost corresponding to the benefit is less than or equal to the storage resource limit. Determine the target number of quantized bits for each layer from the candidate number of quantized bits. The storage resource limit is determined based on the space of the hardware storage model parameters and the space occupied by the unquantized part of the model. The cost corresponding to the benefit is determined based on the cost function.

[0016] According to a quantization method for a large-scale text generation model provided by the present invention, the step of maximizing the return corresponding to the return function, wherein the cost corresponding to the return satisfies storage resource constraints, and determining the target quantization bit number for each layer from each candidate quantization bit number includes:

[0017] To maximize the return corresponding to the return function, multiple quantization strategies of the benchmark text generation model are determined, wherein the quantization strategy refers to the number of candidate quantization bits corresponding to each layer in the benchmark text generation model;

[0018] Based on the cost function, the cost corresponding to each quantization strategy is determined, and the number of candidate quantization bits corresponding to each layer in the quantization strategy with the minimum cost is taken as the target number of quantization bits for each layer.

[0019] According to the quantization method for a large-scale text generation model provided by the present invention, the profit function is:

[0020]

[0021] Among them, profit {i,n} Delta_ppl represents the gain after quantization of the i-th layer using n candidate quantization bits. {i,n}This represents the sensitivity of the i-th layer to the number of candidate quantization bits n, where n m This represents the maximum number of candidate quantization bits;

[0022] The cost function is:

[0023]

[0024] Among them, cost {i,n} This represents the cost of quantization at layer i using the number of candidate quantization bits n. b This indicates the number of bits in the basic storage unit.

[0025] According to a quantization method for a large-scale text generation model provided by the present invention, the step of constructing an objective function based on the sensitivity of each layer to each candidate quantization bit number, solving the objective function, and determining the target quantization bit number for each layer from each candidate quantization bit number includes:

[0026] Based on the sensitivity of each layer to each candidate quantization bit number, and the sensitivity of each layer to the maximum candidate quantization bit number, the profit function in the objective function is constructed.

[0027] With the goal of maximizing the return corresponding to the aforementioned return function, the target number of quantized bits for each layer is determined from the candidate number of quantized bits.

[0028] According to the present invention, a method for quantizing a large-scale text generation model is provided, wherein the benchmark text generation model is determined based on the following steps:

[0029] Obtain the original text generation model after training;

[0030] The original text generation model is quantized based on the initial number of quantization bits to obtain the baseline text generation model; the initial number of quantization bits is greater than the number of candidate quantization bits.

[0031] The present invention also provides a large-scale text generation model quantization device, comprising:

[0032] The determination unit is used to determine multiple candidate quantization bit numbers and quantize the benchmark text generation model based on each candidate quantization bit number to obtain multiple single-precision quantization models.

[0033] The replacement unit is used to replace the i-th layer model parameters of the benchmark text generation model with the i-th layer model parameters of each single-precision quantization model to obtain multiple candidate quantization models corresponding to the i-th layer.

[0034] The calculation unit is used to determine the sensitivity of the i-th layer to the number of candidate quantization bits based on the performance index of the benchmark text generation model and the performance index of each candidate quantization model corresponding to the i-th layer.

[0035] The construction unit is used to construct an objective function based on the sensitivity of each layer to each candidate number of quantization bits, and solve the objective function to determine the target number of quantization bits for each layer from each candidate number of quantization bits.

[0036] The quantization unit is used to update the model parameters of each layer in the reference text generation model with the model parameters in the corresponding single-precision quantization model based on the target quantization bit number of each layer, so as to obtain the quantized target text generation model.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the large text generation model quantization method as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the large text generation model quantization method as described above.

[0039] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the large text generation model quantization method as described above.

[0040] The present invention provides a method, apparatus, electronic device, and storage medium for quantizing large-scale text generation models. It quantizes each layer of a benchmark text generation model based on each candidate quantization bit number, determines the sensitivity of each layer to each candidate quantization bit number, and thus determines the target quantization bit number of each layer based on the objective function constructed based on the sensitivity of each layer to each candidate quantization bit number. This not only significantly reduces the amount of computation but also ensures the quantization accuracy of the model. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the quantization method for large-scale text generation models provided by the present invention;

[0043] Figure 2 This is one of the schematic diagrams of the sensitivity curves of each layer to different quantization bit numbers provided by the present invention;

[0044] Figure 3This is the second schematic diagram of the sensitivity curves of each layer to different quantization bit numbers provided by the present invention;

[0045] Figure 4 This is one of the schematic diagrams illustrating the quantization effect on the OPT model provided by the present invention;

[0046] Figure 5 This is the second schematic diagram of the quantization effect on the OPT model provided by the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of the large-scale text generation model quantization device provided by the present invention;

[0048] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0050] Because trained text generation models typically contain a large number of model parameters, these parameters occupy a significant portion of the storage space. However, during inference, these models often need to temporarily store some variables. If the model parameters occupy most of the storage space, the space available for storing variables will be significantly reduced, thus affecting the user experience. Furthermore, during inference, these model parameters need to be loaded into different storage areas. Using a smaller number of bits to represent these parameters allows the GPU to move them faster, improving inference speed. Therefore, this invention quantizes the trained text generation model to reduce the storage space required for the model parameters while maintaining quantization accuracy.

[0051] In response, this invention provides a method for quantizing large-scale text generation models. Figure 1 This is a flowchart illustrating the quantization method for large-scale text generation models provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0052] Step 110: Determine multiple candidate quantization bit numbers, and quantize the benchmark text generation model based on each candidate quantization bit number to obtain multiple single-precision quantization models.

[0053] Here, the baseline text generation model can be either the trained text generation model or a model quantized with a larger number of quantization bits. The candidate quantization bit count refers to the number of candidate bits used to quantize the baseline text generation model. The candidate quantization bit count can also be used to characterize quantization accuracy; a larger candidate quantization bit count corresponds to higher quantization accuracy.

[0054] In some specific implementations, multiple single-precision quantization models can be obtained by using the specific precision provided by the open-source large model (usually including 8-bit and 4-bit quantization versions), or quantization can be performed on the basis of a high-precision model (such as 16-bit).

[0055] Step 120: Replace the i-th layer model parameters of the baseline text generation model with the i-th layer model parameters of each single-precision quantization model to obtain multiple candidate quantization models corresponding to the i-th layer.

[0056] In some specific implementations, the candidate quantization bit numbers include 4 bits, 3 bits, and 2 bits. Using 4 bits to perform single-precision quantization on the benchmark text generation model yields single-precision quantization model 1; using 3 bits to perform single-precision quantization on the benchmark text generation model yields single-precision quantization model 2; and using 2 bits to perform single-precision quantization on the benchmark text generation model yields single-precision quantization model 3.

[0057] Next, the i-th layer model parameters of the baseline text generation model are replaced with the i-th layer model parameters of single-precision quantization model 1 to obtain candidate quantization model 1 corresponding to the i-th layer; the i-th layer model parameters of the baseline text generation model are replaced with the i-th layer model parameters of single-precision quantization model 2 to obtain candidate quantization model 2 corresponding to the i-th layer; the i-th layer model parameters of the baseline text generation model are replaced with the i-th layer model parameters of single-precision quantization model 3 to obtain candidate quantization model 3 corresponding to the i-th layer. Following the above method, candidate quantization models corresponding to each layer in the baseline text generation model are obtained.

[0058] Step 130: Based on the performance metrics of the benchmark text generation model and the performance metrics of each candidate quantization model corresponding to the i-th layer, determine the sensitivity of the i-th layer to the number of candidate quantization bits.

[0059] Specifically, sensitivity is used to characterize the influence of each candidate quantization bit number on each layer of the model. It can be represented by the difference between the performance metrics of each candidate quantization model corresponding to layer i and the performance metrics of the benchmark text generation model. The larger the difference between the two, the greater the influence of the corresponding candidate quantization bit number on layer i, that is, the higher the sensitivity of layer i to that candidate quantization bit number. The model performance metric can be represented by perplexity, which measures the perplexity of the model when predicting the next word or token in a given text. The smaller the perplexity, the better the model's understanding of the text, and the higher the model's performance metric; conversely, the larger the perplexity, the worse the model's understanding of the text, and the lower the model's performance metric.

[0060] Optionally, the sensitivity of the i-th layer to the number of candidate quantization bits is delta_ppl {i,n} The following formula can be used to calculate: delta_ppl {i,n} =PPL {i,n} -PPL base PPL {i,n} PPL represents the perplexity of the candidate quantized model obtained by replacing the parameters of the i-th layer model of the baseline text generation model with the parameters of the i-th layer model of the single-precision quantization corresponding to the number of quantization bits n. base This represents the perplexity of a 16-bit or benchmark text generation model.

[0061] Figure 2 The graph shows the sensitivity (delta_ppl) of each layer in the 1.3-billion-parameter OPT (Optimized Parameterized Architecture) model to 4-bit, 3-bit, and 2-bit parameters. Figure 3 The graph shows the sensitivity (delta_ppl) of each layer to 4-bit, 3-bit, and 2-bit in a 30-billion-parameter OPT model. Figure 2 and Figure 3 It can be seen that some layers are less sensitive to 4-bit, 3-bit, and 2-bit, meaning that quantization of these layers has little impact on the model's performance metrics. Therefore, lower precision can be used to quantize these layers. For layers with higher sensitivity, higher precision is used for quantization.

[0062] Step 140: Based on the sensitivity of each layer to each candidate number of quantization bits, construct an objective function and solve the objective function to determine the target number of quantization bits for each layer from each candidate number of quantization bits.

[0063] Step 150: Update the model parameters of each layer in the baseline text generation model with the model parameters in the corresponding single-precision quantization model based on the target quantization bit count of each layer, and obtain the quantized target text generation model.

[0064] Specifically, sensitivity is used to characterize the influence of each candidate quantization bit number on each layer in the model. Higher sensitivity indicates a greater influence of the corresponding candidate quantization bit number on the corresponding layer, meaning that a larger candidate quantization bit number is needed for that layer to ensure its quantization accuracy. Lower sensitivity indicates a smaller influence of the corresponding candidate quantization bit number on the corresponding layer, meaning that the quantization accuracy difference for that layer is smaller across different candidate quantization bit numbers. Therefore, a smaller candidate quantization bit number can be selected for quantization of that layer, reducing the storage space occupied by the quantized model parameters.

[0065] In some specific implementations, a reward function in the objective function can be constructed based on the sensitivity of each layer to each candidate number of quantized bits and the sensitivity of each layer to the maximum candidate number of quantized bits. In order to ensure the quantization accuracy of each layer, the reward function is solved with the goal of maximizing the reward corresponding to the reward function, and the target number of quantized bits for each layer is determined from each candidate number of quantized bits.

[0066] After determining the target quantization bit count for each layer, the model parameters of each layer in the baseline text generation model are updated using the model parameters in the corresponding single-precision quantization model based on the target quantization bit count for each layer, resulting in the quantized target text generation model. For example, if the baseline text generation model has three layers, with a target quantization bit count of 4 bits for layer 1, 3 bits for layer 2, and 2 bits for layer 3, then the parameters of layer 1 in the baseline text generation model are updated using the 4-bit model parameters corresponding to layer 1 in the single-precision quantization model, the parameters of layer 2 in the baseline text generation model are updated using the 3-bit model parameters corresponding to layer 2 in the single-precision quantization model, and the parameters of layer 3 in the baseline text generation model are updated using the 2-bit model parameters corresponding to layer 3 in the single-precision quantization model, resulting in the target text generation model after mixed-precision quantization.

[0067] Furthermore, the embodiments of the present invention not only guarantee the quantization accuracy of the model but also significantly reduce the computational load. For example, the 175B model has 96 layers. For each layer, 4 bits, 3 bits, and 2 bits can be selected as candidate quantization bits. If the existing GPTQ technology is used to quantize the model, 96 layers would be required. 3The quantization of various quantization schemes and the selection of the optimal result requires a significant amount of time. To address this, this embodiment of the invention first obtains single-precision quantization models corresponding to 4-bit, 3-bit, and 2-bit quantization based on GPTQ, and then individually replaces the number of quantization bits for each layer, determining the difference in model performance metrics before and after the replacement, i.e., the sensitivity of each layer to the corresponding number of quantization bits. The computational cost throughout the implementation process is: ① 3 calls to the GPTQ method (single-precision quantization); ② determining the sensitivity of each layer to 4-bit, 3-bit, and 2-bit quantization, totaling 3 × 96 inferences; ③ calculating and determining the target number of quantization bits for each layer based on the sensitivity (the cost is relatively small and negligible). That is, for a model with l layers and n candidate quantization bits, the computational cost of determining the target number of quantization bits for each layer in this embodiment is l × n inferences, which is significantly less than the l × n inferences required in existing technologies. n In terms of inference, the embodiments of the present invention not only guarantee the quantization accuracy of the model, but also significantly reduce the amount of computation.

[0068] It is understandable that after obtaining the quantized target text generation model, a text summary can be generated based on the quantized target text generation model. For example, the text to be generated can be input into the quantized target text generation model, and the quantized target text generation model can output the corresponding summary text.

[0069] The text generation model method provided in this invention quantizes each layer of the benchmark text generation model based on each candidate quantization bit number, determines the sensitivity of each layer to each candidate quantization bit number, and thus determines the target quantization bit number of each layer based on the objective function constructed based on the sensitivity of each layer to each candidate quantization bit number. This not only significantly reduces the amount of computation, but also ensures the quantization accuracy of the model.

[0070] Based on the above embodiments, and based on the sensitivity of each layer to the number of candidate quantization bits, an objective function is constructed, including:

[0071] Based on the sensitivity of each layer to each candidate number of quantized bits, and the sensitivity of each layer to the maximum candidate number of quantized bits, the profit function in the objective function is constructed.

[0072] Based on the number of bits for each candidate quantization and the number of bits for the basic storage unit, the cost function in the objective function is constructed.

[0073] Specifically, the reward function characterizes the quantization accuracy of the model; the higher the reward function, the higher the quantization accuracy. However, higher quantization accuracy also means a larger storage space occupied by the quantized model parameters. To address this, this embodiment of the invention constructs a cost function in the objective function based on the number of candidate quantization bits and the number of bits in the basic storage unit. This cost function characterizes the storage space occupied by the model parameters; that is, the higher the cost function, the larger the storage space occupied by the model parameters. The number of bits in the basic storage unit refers to the number of parameters in the standard model used to evaluate model performance, which is typically 8 bits.

[0074] In some specific implementations, the payoff function is:

[0075]

[0076] Among them, profit {i,n} Delta_ppl represents the gain after quantization of the i-th layer using n candidate quantization bits. {i,n} This represents the sensitivity of the i-th layer to the number of candidate quantization bits n, where n m This represents the maximum number of candidate quantization bits. For example, if the number of candidate quantization bits includes 4 bits, 3 bits, and 2 bits, then n∈{2,3,4}, n m =4.

[0077] In some specific implementations, the cost function is:

[0078]

[0079] Among them, cost {i,n} This represents the cost of quantization at layer i using the number of candidate quantization bits n. b Indicates the number of bits in the basic storage unit; optionally, n b =8.

[0080] Based on any of the above embodiments, the objective function is solved, and the target quantization bit number for each layer is determined from each candidate quantization bit number, including:

[0081] Maximize the return corresponding to the return function, and ensure that the cost corresponding to the return is less than or equal to the storage resource limit. Determine the target quantization bit number for each layer from the candidate quantization bit numbers. The storage resource limit is determined based on the space occupied by the hardware storage model parameters and the space occupied by the unquantized part of the model. The cost corresponding to the return is determined based on the cost function.

[0082] Specifically, maximizing the return corresponding to the return function is to maximize the quantization accuracy of the model. However, the higher the quantization accuracy, the larger the storage space occupied by the quantized model parameters. To address this, this embodiment of the invention, while maximizing the return corresponding to the return function, limits the cost corresponding to the return to meet storage resource constraints. That is, while ensuring the quantization accuracy of the model, it minimizes the storage space occupied by the quantized model parameters.

[0083] The storage resource constraint is determined based on the space occupied by the hardware storage model parameters and the space occupied by the unquantized part of the model, and can be expressed by the following formula: constraint = MEM total -MEM base , of which MEM total The space representing the parameters of the hardware storage model, MEM base This represents the space occupied by the unquantized portion of the model.

[0084] Based on any of the above embodiments, to maximize the return corresponding to the return function, and to ensure that the cost corresponding to the return satisfies the storage resource constraint, the target quantization bit number for each layer is determined from each candidate quantization bit number, including:

[0085] To maximize the return corresponding to the return function, multiple quantization strategies are determined for the benchmark text generation model. The quantization strategy refers to the number of candidate quantization bits corresponding to each layer in the benchmark text generation model.

[0086] Based on the cost function, the cost corresponding to each quantization strategy is determined, and the number of candidate quantization bits corresponding to each layer in the quantization strategy with the minimum cost is taken as the target number of quantization bits for each layer.

[0087] Optionally, maximizing the return corresponding to the return function yields multiple quantization strategies for the benchmark text generation model. After obtaining multiple quantization strategies, a cost function can be used to calculate the cost corresponding to each quantization strategy. The obtained costs are then sorted in ascending order. The benchmark text generation model is quantized using the quantization strategy corresponding to the minimum cost. That is, the number of candidate quantization bits for each layer in the quantization strategy corresponding to the minimum cost is taken as the target quantization bit number for each layer, and each layer of the benchmark text generation model is quantized based on the target quantization bit number for each layer.

[0088] Based on any of the above embodiments, an objective function is constructed based on the sensitivity of each layer to each candidate quantization bit number, and the objective function is solved to determine the target quantization bit number for each layer from each candidate quantization bit number, including:

[0089] Based on the sensitivity of each layer to each candidate number of quantized bits, and the sensitivity of each layer to the maximum candidate number of quantized bits, the profit function in the objective function is constructed.

[0090] With the goal of maximizing the return corresponding to the return function, the target number of quantization bits for each layer is determined from the candidate number of quantization bits.

[0091] Specifically, for cases where hardware storage space is unlimited, embodiments of the present invention can also provide a better mixed precision strategy, that is, to construct a profit function using the method of the above embodiments, and to determine the target quantization bit number of each layer from each candidate quantization bit number with the goal of maximizing the profit corresponding to the profit function.

[0092] Based on any of the above embodiments, quantization is performed on each layer of the benchmark text generation model based on each quantization bit count, and prior to this, the following steps are also included:

[0093] Obtain the original text generation model after training;

[0094] The original text generation model is quantized based on the initial quantization bit count to obtain the baseline text generation model; the initial quantization bit count is greater than the number of candidate quantization bits.

[0095] Specifically, the trained original text generation model refers to the model trained based on sample text and sample generated labels. The initial quantization bit count has a higher quantization precision than each candidate quantization bit count; that is, the initial quantization bit count is greater than each candidate quantization bit count. For example, if the candidate quantization bit counts include 4 bits, 3 bits, and 2 bits, then the initial quantization bit count can be 16 bits.

[0096] Based on any of the above embodiments, the present invention also provides a method for quantizing large-scale text generation models, the method comprising:

[0097] First, obtain the baseline text generation model M with 16-bit precision. 16bit Its layer number is l, and M is based on the number of candidate quantization bits (4 bits, 3 bits, and 2 bits). 16bit Quantization is performed to obtain M. 16bit The three corresponding single-precision quantization models are M 4bit M 3bit M 2bit .

[0098] Next, obtain the text data X related to the text generation task, and simultaneously calculate M. 16bit The perplexity index, for a text sequence X = (x0, x1, ..., x... t ), input it into M 16bit Then we get each element x i output probability The corresponding perplexity value is

[0099] For M 16bitThe parameters of the i-th layer model are replaced with M. nbit The i-th layer model parameters (n∈{2,3,4}) are used to obtain the candidate quantization model M. {i,n} M {i,n} The corresponding perplexity value is Finally, the perplexity index set {PPL} for each candidate quantization model is obtained. {i,n}}, where i∈{0,1,…,l-1}, n∈{2,3,4}. Next, the sensitivity delta_ppl of the i-th layer to each candidate quantization bit number n is calculated. {i,n} =PPL {i,n} -PPL base , where i∈{0,1,…,l-1},n∈{2,3,4}.

[0100] The payoff function is defined based on the sensitivity of each layer to the number of candidate quantization bits as follows:

[0101] profit {i,n} =(n-4)(delta_ppl {i,n} -delta_ppl {i,4} )

[0102] Where i∈{0,1,…,l-1},n∈{2,3,4}.

[0103] The cost function is defined as follows:

[0104] Define storage resource constraints as constraint = MEM total -MEM baae , of which MEM total The space representing the parameters of the hardware storage model, MEM base This represents the space occupied by the unquantized portion of the model.

[0105] A multi-objective optimization algorithm is used to determine the target number of quantization bits for each layer, i.e., maximizing profit. {i,n} And the corresponding cost {i,n} The model parameters of each layer in the baseline text generation model are updated based on the target quantization bit count of each layer in the corresponding single-precision quantization model, with the result less than or equal to the constraint. This yields the quantized target text generation model, i.e., the mixed-precision model M. mixed .

[0106] Furthermore, in cases where there are no limitations on hardware storage space, embodiments of the present invention can also maximize profits. {i,n} The objective is to select the candidate number of quantized bits that maximizes the benefit of each layer as the target number of bits: {argmax}n (profit {i,n} )}, where i∈{0,1,…,l-1},n∈{2,3,4}.

[0107] Therefore, it can be seen that, in cases where hardware storage space is limited, the embodiments of the present invention can automatically adjust the mixed precision quantization strategy to provide better model performance while ensuring that the model parameters are stored completely; in cases where hardware is unlimited, it can provide the optimal mixed precision strategy to save hardware resources to the greatest extent while minimizing the loss of model performance.

[0108] also, Figure 4 The present invention demonstrates the test results on an OPT model with 1.3 billion parameters. Figure 5 This is a test result of the present invention on an OPT model with 30 billion parameters. Figure 4 and Figure 5 The horizontal axis represents the average number of bits required for each element, where 2.00, 3.00, and 4.00 correspond to three single-precision quantization models, and other values ​​correspond to mixed-precision quantization models. The vertical axis represents the perplexity metric tested on the wikitext-2 dataset, where a smaller value indicates a better model performance.

[0109] in, Figure 4 and Figure 5 The dots outside the broken line represent the quantization model effect obtained by the random strategy, the dots on the broken line represent the quantization model effect obtained under the condition of satisfying hardware storage space limitations in the embodiments of the present invention, and the boxes represent the quantization model effect obtained under the condition of unlimited hardware storage space provided in the embodiments of the present invention. Figure 4 and Figure 5 It can be seen that, except for the more extreme cases (average bit count of around 2.25), the quantization model effect obtained under the condition of unlimited hardware storage space provided by the embodiments of the present invention is optimal, and the quantization model effect obtained under the condition of unlimited hardware storage space provided by the embodiments of the present invention can provide a better suggestion for balancing hardware storage resources and model effect.

[0110] The large text generation model quantization device provided by the present invention is described below. The large text generation model quantization device described below and the large text generation model quantization method described above can be referred to in correspondence.

[0111] Based on any of the above embodiments, the present invention also provides a large-scale text generation model quantization device, such as... Figure 6 As shown, the device includes:

[0112] Candidate quantization unit 610 is used to determine multiple candidate quantization bit numbers and quantize the benchmark text generation model based on each candidate quantization bit number to obtain multiple single-precision quantization models;

[0113] Replacement unit 620 is used to replace the i-th layer model parameters of the benchmark text generation model with the i-th layer model parameters of each single-precision quantization model to obtain multiple candidate quantization models corresponding to the i-th layer.

[0114] Sensitivity calculation unit 630 is used to determine the sensitivity of the i-th layer to each candidate quantization bit number based on the performance index of the benchmark text generation model and the performance index of each candidate quantization model corresponding to the i-th layer.

[0115] The mixed-precision configuration solver unit 640 is used to construct an objective function based on the sensitivity of each layer to each candidate number of quantization bits, and solve the objective function to determine the target number of quantization bits for each layer from each candidate number of quantization bits.

[0116] The quantization model construction unit 650 is used to update the model parameters of each layer in the reference text generation model with the model parameters in the corresponding single-precision quantization model based on the target quantization bit number of each layer, so as to obtain the quantized target text generation model.

[0117] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 7As shown, the electronic device may include: a processor 710, a memory 720, a communications interface 730, and a communications bus 740, wherein the processor 710, the memory 720, and the communications interface 730 communicate with each other through the communications bus 740. The processor 710 can call logic instructions in the memory 720 to execute a large-scale text generation model quantization method. This method includes: determining multiple candidate quantization bit counts, and quantizing a benchmark text generation model based on each candidate quantization bit count to obtain multiple single-precision quantization models; replacing the i-th layer model parameters of the benchmark text generation model with the i-th layer model parameters of each single-precision quantization model to obtain multiple candidate quantization models corresponding to the i-th layer; determining the sensitivity of the i-th layer to each candidate quantization bit count based on the performance metrics of the benchmark text generation model and the performance metrics of each candidate quantization model corresponding to the i-th layer; constructing an objective function based on the sensitivity of each layer to each candidate quantization bit count, solving the objective function, and determining the target quantization bit count for each layer from the candidate quantization bit counts; updating the model parameters of each layer in the benchmark text generation model with the model parameters of the corresponding single-precision quantization model based on the target quantization bit count of each layer to obtain the quantized target text generation model.

[0118] Furthermore, the logical instructions in the aforementioned memory 720 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0119] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the large text generation model quantization method provided by the above methods, the method comprising: determining a plurality of candidate quantization bit numbers, and quantizing a benchmark text generation model based on each candidate quantization bit number to obtain a plurality of single-precision quantization models; replacing the i-th layer model parameters of the benchmark text generation model with the i-th layer model parameters of each single-precision quantization model to obtain a plurality of candidate quantization models corresponding to the i-th layer; determining the sensitivity of the i-th layer to each candidate quantization bit number based on the performance index of the benchmark text generation model and the performance index of each candidate quantization model corresponding to the i-th layer; constructing an objective function based on the sensitivity of each layer to each candidate quantization bit number, and solving the objective function to determine the target quantization bit number of each layer from each candidate quantization bit number; updating the model parameters of each layer in the benchmark text generation model with the model parameters of the corresponding single-precision quantization model based on the target quantization bit number of each layer to obtain the quantized target text generation model.

[0120] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the aforementioned methods for quantizing large text generation models. The method includes: determining multiple candidate quantization bit counts, and quantizing a benchmark text generation model based on each candidate quantization bit count to obtain multiple single-precision quantization models; replacing the i-th layer model parameters of the benchmark text generation model with the i-th layer model parameters of each single-precision quantization model to obtain multiple candidate quantization models corresponding to the i-th layer; determining the sensitivity of the i-th layer to each candidate quantization bit count based on the performance metrics of the benchmark text generation model and the performance metrics of each candidate quantization model corresponding to the i-th layer; constructing an objective function based on the sensitivity of each layer to each candidate quantization bit count, solving the objective function, and determining the target quantization bit count for each layer from the candidate quantization bit counts; updating the model parameters of each layer in the benchmark text generation model with the model parameters of the corresponding single-precision quantization model based on the target quantization bit count of each layer to obtain the quantized target text generation model.

[0121] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0122] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A quantization method for large-scale text generation models, characterized in that, include: Multiple candidate quantization bit numbers are determined, and the benchmark text generation model is quantized based on each candidate quantization bit number to obtain multiple single-precision quantization models; The first of each single-precision quantization model The layer model parameters respectively replace the first layer of the baseline text generation model. The layer model parameters are obtained to get the first layer. Multiple candidate quantization models corresponding to the layer; Based on the performance metrics of the benchmark text generation model and the first The performance metrics of each candidate quantization model corresponding to the layer are used to determine the first layer. The sensitivity of each layer to the number of candidate quantization bits; the sensitivity is used to characterize the degree of influence of the number of candidate quantization bits on each layer in the model, using the [missing information - likely a specific parameter or characteristic]. The difference between the performance metrics of each candidate quantization model corresponding to the layer and the performance metrics of the benchmark text generation model is represented; the performance metrics are used to measure the perplexity of the model when predicting the next word or token in a given text; Based on the sensitivity of each layer to each candidate number of quantization bits, an objective function is constructed and the objective function is solved to determine the target number of quantization bits for each layer from each candidate number of quantization bits. The model parameters of each layer in the baseline text generation model are updated using the model parameters of the corresponding single-precision quantization model based on the target quantization bit count of each layer, resulting in the quantized target text generation model.

2. The quantization method for large-scale text generation models according to claim 1, characterized in that, The objective function, constructed based on the sensitivity of each layer to the number of candidate quantization bits, includes: Based on the sensitivity of each layer to each candidate quantization bit number, and the sensitivity of each layer to the maximum candidate quantization bit number, the profit function in the objective function is constructed. The cost function in the objective function is constructed based on the number of candidate quantization bits and the number of bits in the basic storage unit.

3. The quantization method for large-scale text generation models according to claim 2, characterized in that, Solving the objective function and determining the target quantization bit number for each layer from the candidate quantization bit numbers includes: Maximize the benefit corresponding to the benefit function, and ensure that the cost corresponding to the benefit is less than or equal to the storage resource limit. Determine the target number of quantized bits for each layer from the candidate number of quantized bits. The storage resource limit is determined based on the space of the hardware storage model parameters and the space occupied by the unquantized part of the model. The cost corresponding to the benefit is determined based on the cost function.

4. The quantization method for large-scale text generation models according to claim 3, characterized in that, Maximizing the return corresponding to the return function, where the cost corresponding to the return satisfies storage resource constraints, and determining the target quantization bit number for each layer from each candidate quantization bit number, includes: To maximize the return corresponding to the return function, multiple quantization strategies of the benchmark text generation model are determined, wherein the quantization strategy refers to the number of candidate quantization bits corresponding to each layer in the benchmark text generation model; Based on the cost function, the cost corresponding to each quantization strategy is determined, and the number of candidate quantization bits corresponding to each layer in the quantization strategy with the minimum cost is taken as the target number of quantization bits for each layer.

5. The quantization method for large-scale text generation models according to claim 2, characterized in that, The profit function is: ； in, Indicates the first The layer uses the candidate quantization bit number Quantified returns Indicates the first Number of candidate quantization bits per layer Sensitivity This represents the maximum number of candidate quantization bits; The cost function is: ； in, Indicates the first The layer uses the candidate quantization bit number The cost after quantification This indicates the number of bits in the basic storage unit.

6. The method for quantizing large-scale text generation models according to any one of claims 1 to 5, characterized in that, The process of constructing an objective function based on the sensitivity of each layer to each candidate quantization bit number, solving the objective function, and determining the target quantization bit number for each layer from each candidate quantization bit number includes: Based on the sensitivity of each layer to each candidate number of quantized bits, and the sensitivity of each layer to the maximum candidate number of quantized bits, the profit function in the objective function is constructed. With the goal of maximizing the return corresponding to the aforementioned return function, the target number of quantized bits for each layer is determined from the candidate number of quantized bits.

7. The method for quantizing large-scale text generation models according to any one of claims 1 to 5, characterized in that, The baseline text generation model is determined based on the following steps: Obtain the original text generation model after training; The original text generation model is quantized based on the initial number of quantization bits to obtain the baseline text generation model; the initial number of quantization bits is greater than the number of candidate quantization bits.

8. A quantization device for a large-scale text generation model, characterized in that, include: Candidate quantization units are used to determine multiple candidate quantization bit numbers and quantize the benchmark text generation model based on each candidate quantization bit number to obtain multiple single-precision quantization models. Replacement unit, used to quantize the model with each single precision The layer model parameters respectively replace the first layer of the baseline text generation model. The layer model parameters are obtained to get the first layer. Multiple candidate quantization models corresponding to the layer; The sensitivity calculation unit is used to calculate the performance metrics of the benchmark text generation model and the first... The performance metrics of each candidate quantization model corresponding to the layer are used to determine the first layer. The sensitivity of each layer to the number of candidate quantization bits; the sensitivity is used to characterize the degree of influence of the number of candidate quantization bits on each layer in the model, using the [missing information - likely a specific parameter or characteristic]. The difference between the performance metrics of each candidate quantization model corresponding to the layer and the performance metrics of the benchmark text generation model is represented; the performance metrics are used to measure the perplexity of the model when predicting the next word or token in a given text; A mixed-precision configuration solver unit is used to construct an objective function based on the sensitivity of each layer to each candidate number of quantization bits, and solve the objective function to determine the target number of quantization bits for each layer from each candidate number of quantization bits. The quantization model construction unit is used to update the model parameters of each layer in the baseline text generation model based on the target quantization bit number of each layer in the corresponding single-precision quantization model, so as to obtain the quantized target text generation model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the large text generation model quantization method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the large text generation model quantization method as described in any one of claims 1 to 7.

Images