Off-line large model soft quantization method and device, electronic equipment, storage medium and program product

Abstract

The invention provides an offline large model soft quantization method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence algorithms. According to the embodiment of the invention, the error between the quantized output result and the original output result is minimum while the validity of the quantized data is kept. In the embodiment of the invention, the original data of the model is alternately subjected to value domain softening and numerical value quantization, so that the data quantization is completed after the original value is softened to a proper range, and the quantized data almost contains the effective information of the original data. Meanwhile, model node input and weights are combined into a whole, the node input and the weights are not independently quantized, and the error between a final quantized input and weight calculation result and a node original data calculation result after inverse quantization is minimum through alternate soft quantization of the model input and the weights; therefore, the problem of poor quantization effect of the final calculation result caused by respective quantization input and weight is eliminated.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence algorithm technology, and in particular to an offline large model soft quantization method, device, electronic device, storage medium and program product. Background Art

[0002] As large models have achieved excellent results in various fields, they have received more and more attention, and offline large model deployment has become an increasingly important topic. However, offline large models consume too much hardware resources when deployed, so how to reduce hardware consumption has become the key to model deployment. Model quantization can effectively reduce the model and compress the model's hardware resource consumption, so quantization has become the key to model deployment.

[0003] The distribution of parameter data of offline large models is relatively smooth, but there are some very abrupt abnormal values ​​at some key points. Some of these values ​​have a great impact on the model effect. If such data is directly quantized, most of the data will be quantized to near zero, leaving only a small part of the data, and the quantization effect is extremely poor. At the same time, the more popular way of model quantization is to quantize the node input and weight separately. Using this quantization method may cause a large error between the quantization result and the node output after dequantization, and ultimately fail to quantize.

[0004] It can be seen that a new offline large model quantization method is urgently needed. Summary of the invention

[0005] Embodiments of the present invention provide an offline large-model soft quantization method, device, electronic device, storage medium and program product to at least partially solve the above-mentioned problems.

[0006] A first aspect of an embodiment of the present invention provides an offline large model soft quantization system, the method comprising:

[0007] Traverse and extract multiple nodes of the offline large model and save them to the node pool;

[0008] For any node in the node pool, if the node has no weight, the node input of the node is soft-quantized based on the input softening factor and the input quantization factor; if the node has weight, the node input and the node weight are jointly soft-quantized based on the input quantization factor and the softening factor as well as the weight softening factor and the weight quantization factor; after completing the soft quantization of the node, the soft quantization node parameters of the node are saved;

[0009] It is determined whether all nodes in the node pool have completed soft quantization. When all nodes have completed soft quantization, the parameters of each soft quantization node are saved as a quantization model.

[0010] Optionally, when the node has no weight, soft quantization is performed on the node input of the node based on the input softening factor and the input quantization factor, including:

[0011] Add input softening factor to the node input of this node;

[0012] Obtaining a first node input according to the node input to which an input softening factor is added;

[0013] quantizing the first node input according to the input quantization factor;

[0014] With the quantization error being less than a first threshold as a goal, the input softening factor and the input quantization factor are alternately iteratively updated to obtain a first input softening factor and a first input quantization factor; in any iterative process of the alternate iterative update, a value of one of the input softening factor and the input quantization factor obtained in the previous iterative process is fixed, and the value of the other is updated;

[0015] Save the soft quantization node parameters of the node, including:

[0016] A first input softening factor, a first quantization factor, and a first soft quantization node input obtained based on the first input softening factor and the first input quantization factor are saved.

[0017] Optionally, the method further comprises:

[0018] Dequantizing the first soft quantization node input to obtain a first dequantization result;

[0019] determining a first difference value based on the first inverse quantization result, a first input softening factor, and the node input;

[0020] When the first difference value is not less than the second threshold, continue to iterate the input softening factor and the input quantization factor with the goal of making the quantization error less than the first threshold, until the first difference value obtained after the soft quantization node input obtained by the iterative update is dequantized is less than the second threshold, thereby obtaining a second input softening factor and a second input quantization factor;

[0021] The step of storing the soft quantization node parameters of the node includes:

[0022] The second input softening factor, the second input quantization factor, and a second soft quantization node input obtained based on the second softening factor and the second quantization factor are saved.

[0023] Optionally, when the node has a weight, the node input and the node weight are jointly soft-quantized based on the input quantization factor and the softening factor and the weight softening factor and the weight quantization factor, including:

[0024] Add an input softening factor to the node input of the node, and add a weight softening factor to the node weight;

[0025] Obtaining a second node input according to the node input to which the weight softening factor is added and the input softening factor is added;

[0026] quantizing the second node input according to the input quantization factor;

[0027] The weight softening factor is fixed unchanged, and the quantization error is less than a first threshold as a goal, and the input softening factor and the input quantization factor are alternately iteratively updated to obtain a third input softening factor and a third input quantization factor;

[0028] Obtaining a first node weight according to the node weight added with the third input softening factor and the node weight added with the weight softening factor;

[0029] quantizing the weight of the first node according to the weight quantization factor;

[0030] The third input softening factor obtained by the fixed alternating iterative update remains unchanged, and the weight softening factor and the weight input quantization factor are alternately iteratively updated with the goal of making the quantization error less than the first threshold to obtain the first weight softening factor and the first weight quantization factor.

[0031] Optionally, the method further comprises:

[0032] obtaining a third node input based on the first weight softening factor, the third input softening factor and the node input;

[0033] Dequantizing the input soft quantization result to obtain a second dequantization result; the input soft quantization result is obtained according to the third input softening factor, the first weight softening factor, the third input quantization factor and the node input;

[0034] Dequantizing the weight soft quantization result to obtain a third dequantization result; the weight soft quantization result is obtained according to the first weight softening factor, the third input softening factor, the first weight quantization factor and the node weight;

[0035] Based on the second inverse quantization result, the third inverse quantization result, the node input and the node weight, a second difference value is inputted to determine;

[0036] When the difference value is not less than the second threshold, continue to soft quantize the node input and node weight jointly with the goal of making the quantization error less than the first threshold, until the second difference value obtained after inverse quantization of the input soft quantization result and the weight soft quantization result obtained by alternating iterative updating is less than the second threshold, and the fourth input softening factor, the fourth input quantization factor, the second weight softening factor, and the second weight quantization factor are obtained.

[0037] Optionally, save soft quantization node parameters, including:

[0038] Saving the fourth input softening factor, the fourth input quantization factor, the second weight softening factor, and the second weight quantization factor;

[0039] Saving an input soft quantization result obtained according to the fourth input softening factor, the fourth input quantization factor and the node input;

[0040] The weight soft quantization result obtained according to the second weight softening factor, the second weight quantization factor and the node weight is saved.

[0041] A second aspect of an embodiment of the present invention provides an offline large model soft quantization device, the offline large model soft quantization device comprising:

[0042] The offline large model node extraction module is used to traverse and extract multiple nodes of the offline large model and save them to the node pool;

[0043] A node soft quantization module is used for, for any node in the node pool, if the node has no weight, to soft quantize the node input of the node based on the input softening factor and the input quantization factor; if the node has weight, to soft quantize the node input and the node weight jointly based on the input quantization factor and the softening factor as well as the weight softening factor and the weight quantization factor; after completing the soft quantization of the node, the soft quantization node parameters of the node are saved;

[0044] The quantization model saving module is used to determine whether all nodes in the node pool have completed soft quantization. When all nodes have completed soft quantization, the parameters of each soft quantization node are saved as a quantization model.

[0045] A third aspect of an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the offline large model soft quantization method as described in the first aspect of the present invention is implemented.

[0046] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the offline large model soft quantization method as described in the first aspect of the present invention.

[0047] A fifth aspect of an embodiment of the present invention provides a computer program product, including a computer program / instruction, which is used by a processor to implement the steps in the offline large model soft quantization method as described in the first aspect of the present invention.

[0048] The embodiment of the present invention minimizes the error between the quantized output result and the original output result while retaining the validity of the quantized data. In the embodiment of the present invention, the original data of the model are alternately softened in the range and quantized in the numerical value, ensuring that the data quantization is completed after the original value is softened to a suitable range, and ensuring that the quantized data almost completely contains the valid information of the original data. At the same time, the model node input and weight are combined into a whole, and the node input and weight are no longer quantized independently. By alternately soft-quantizing the model input and weight, the error between the final quantized input and weight calculation result after dequantization and the node original data calculation result is minimized, thereby eliminating the problem of poor quantization effect of the final calculation result caused by separately quantizing the input and weight. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0050] Figure 1 is a flowchart of the steps of the offline large model soft quantization method provided by an embodiment of the present invention;

[0051] Figure 2 This is an example diagram of large model data value anomalies and quantization results in the offline large model soft quantization method provided in an embodiment of the present invention.

[0052] Figure 3 It is a flowchart of the overall steps of soft quantization of nodes in the offline large model soft quantization method provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] The present invention provides a flowchart of the steps of an offline large model soft quantization method. Figure 1 As shown, specifically, the offline large model soft quantization method includes the following steps:

[0055] S101, traverse and extract multiple nodes of the offline large model, and save them to the node pool.

[0056] S102, for any node in the node pool, when the node has no weight, the node input of the node is soft quantized based on the input softening factor and the input quantization factor; when the node has weight, the node input and the node weight are jointly soft quantized based on the input quantization factor and the softening factor as well as the weight softening factor and the weight quantization factor; after completing the soft quantization of the node, the soft quantization node parameters of the node are saved.

[0057] S103, judging whether all nodes in the node pool have completed soft quantization, and if all nodes have completed soft quantization, saving each soft quantization node parameter as a quantization model.

[0058] In the embodiment of the present invention, the offline large model refers to a deep network model with a huge number of parameters based on the transformer structure after training.

[0059] In the embodiment of the present invention, the multiple nodes in the offline large model may include: fully connected nodes, activated nodes, etc.

[0060] In the embodiment of the present invention, in step S101, all nodes of the offline large model may be extracted, and these nodes may be formed into a node pool, and all subsequent nodes that need to be soft quantized may be extracted from the node pool.

[0061] In the embodiment of the present invention, quantization refers to compressing floating point data into integer data.

[0062] In the embodiment of the present invention, in step S103, the soft quantization node parameters after soft quantization may be saved as a quantization model for subsequent model inference operations.

[0063] In the embodiment of the present invention, the value range of the original data can be effectively optimized to obtain a suitable value range for quantization by alternately optimizing the softening factor and the quantization factor. The optimized and adjusted data value range can make the quantized data basically retain all the valid information of the original data, effectively avoiding the influence of data outliers on the quantization results.

[0064] In the embodiment of the present invention, when the node has weight, the node input and the node weight are jointly soft-quantized, the node input value and the weight value are connected into a whole through the softening factor, and the node input and weight are alternately soft-quantized so that the error between the quantized input and weight calculation results and the original value calculation results after dequantization is minimized, and an optimal quantized output effect is obtained. The problem of independently quantizing the input and weight but the final calculation result is not good is eliminated.

[0065] The technical solution provided by the embodiment of the present invention is used to soft-quantize the nodes of the offline large model to solve the problem of quantization failure when there are abnormal values ​​in the quantized data. Figure 2As shown, it shows an example diagram of large model data value anomalies and quantization results in the offline large model soft quantization method provided by an embodiment of the present invention.

[0066] In the embodiment of the present invention, when the node has no weight, soft quantization is performed on the node input of the node based on the input softening factor and the input quantization factor, specifically including the following steps:

[0067] S201, adding an input softening factor to the node input of the node;

[0068] S202, obtaining a first node input according to the node input to which an input softening factor is added;

[0069] S203, quantizing the first node input according to the input quantization factor;

[0070] S204, with the quantization error being less than the first threshold as a goal, the input softening factor and the input quantization factor are alternately iteratively updated to obtain a first input softening factor and a first input quantization factor; in any iterative process of the alternate iterative update, a value of one of the input softening factor and the input quantization factor obtained in the previous iterative process is fixed, and the value of the other is updated;

[0071] S205 , save the first input softening factor, the first quantization factor, and the first soft quantization node input obtained based on the first input softening factor and the first input quantization factor.

[0072] For ease of understanding, combined Figure 3 The offline large model soft quantization method provided by the embodiment of the present invention is further explained. Figure 3 The overall step flow chart of soft quantization of nodes in the offline large model soft quantization method provided by an embodiment of the present invention is shown.

[0073] like Figure 3 As shown, in the embodiment of the present invention, a softening factor is introduced to alternately perform range softening and quantization operations on the original data. Specifically, the node input X of the current node can be obtained first, and a softening factor is added to X. Adaptively adjust the value range of X. Where X is a set of n values, X m is the mean of the input X, and λ is the input softening factor.

[0074] Further determine whether the current node has a weight value W.

[0075] In the absence of a weight value W, the quantization parameter s and the scaling factor λ are alternately optimized to perform soft quantization on the node input X, and the first soft quantized node input X is obtained. q .

[0076] Specifically, in the embodiment of the present invention, when a node does not include a weight, Let the first node input X' is quantified. In the embodiment of the present invention, the quantization formula is: in float32V is the first node input X', int8V is the quantized data, and s is the input quantization factor to be optimized. When the quantized data exceeds a range, the data is limited to the maximum value of the range. This embodiment is int8 quantization, and all quantized data must be less than 2 8 , and is limited to 2 after exceeding 8 -1.

[0077] In the embodiment of the present invention, the quantization error is calculated based on the first node input X' and the input quantization factor. The calculation formula of the quantization error is:

[0078] In the embodiment of the present invention, it is necessary to optimize the input quantization factor s and the input softening factor λ to minimize the quantization error L. s and λ are optimized alternately. First, s is updated. The update formula is: where s k is the current quantization factor, λ k is the current input softening factor, and k is the current iteration number. Then the softening factor is updated using the updated quantization factor. The update formula is: where s k+1 is the updated input quantization factor. It is determined whether the quantization error L is less than the specified first threshold. When it is less than the first threshold, the first node input X′ completes the soft quantization, which is recorded as X q .

[0079] When it is greater than the first threshold, the updated input softening factor λ is used k+1 The input quantization factor is updated again, and the input softening factor is updated using the updated input quantization factor, and the quantization factor and the softening factor are alternately optimized until the quantization error L is less than the first threshold to obtain a first input softening factor and a first soft quantization node input obtained based on the first input softening factor and the first input quantization factor.

[0080] In the embodiment of the present invention, when the node has no weight, performing soft quantization on the node input of the node based on the input softening factor and the input quantization factor may further include the following steps:

[0081] S206, performing inverse quantization on the first soft quantization node input to obtain a first inverse quantization result;

[0082] S207, determining a first difference value based on the first inverse quantization result, a first input softening factor, and a first node input;

[0083] S208, when the first difference value is not less than the second threshold, continue to iterate and update the input softening factor and the input quantization factor alternately with the quantization error being less than the first threshold as the goal, until the first difference value obtained after the soft quantization node input obtained by the alternate iterative update is dequantized is less than the second threshold, thereby obtaining a second input softening factor and a second input quantization factor;

[0084] S209: Save the second input softening factor, the second input quantization factor, and a second soft quantization node input obtained based on the second softening factor and the second quantization factor.

[0085] like Figure 3 As shown, the inverse quantization X q Note: X d , that is, input X to the first soft quantization node q Perform inverse quantization to obtain the first inverse quantization result X d .

[0086] In the case that the current node does not have a weight value, the first difference value is determined according to the first inverse quantization result, the first input softening factor and the node input. Specifically, Determine whether diff is less than the specified second threshold. If it is less than the specified second threshold, complete the soft quantization of the input of the current node and save the relevant optimization factors. If it is greater than the second threshold, determine whether the current node contains weights. If not, continue to alternately optimize and solve the quantization parameter s and the scaling factor λ until the node is quantized.

[0087] In the embodiment of the present invention, when the node has a weight, the node input and the node weight are jointly soft-quantized based on the input quantization factor and the softening factor and the weight softening factor and the weight quantization factor, specifically including the following steps:

[0088] S301, adding an input softening factor to the node input of the node, and adding a weight softening factor to the node weight;

[0089] S302, obtaining a second node input according to the node input to which the weight softening factor and the input softening factor are added;

[0090] S303, quantizing the second node input according to the input quantization factor;

[0091] S304, fixing the weight softening factor unchanged, taking the quantization error less than the first threshold as the goal, performing alternating iterative updates on the input softening factor and the input quantization factor to obtain a third input softening factor and a third input quantization factor;

[0092] S305, obtaining a first node weight according to the node weight added with the third input softening factor and the node weight added with the weight softening factor;

[0093] S306, quantizing the first node weight according to the weight quantization factor;

[0094] S307, the third input softening factor obtained by the fixed alternating iterative update remains unchanged, and the weight softening factor and the weight input quantization factor are alternately iteratively updated with the goal of making the quantization error less than the first threshold, to obtain a first weight softening factor and a first weight quantization factor.

[0095] like Figure 3 As shown, when the current node has a weight value W, the node weight W of the current node is obtained and a softening factor is added to W. Adaptively adjust the value range of W. Where W is a set of n values, W m is the mean of the node weight W, and β is the weight softening factor to be optimized.

[0096] In the case of weights, the calculation formula of the current node is Y = X × W. After the adaptive factor is added, the node calculation formula becomes Can remember

[0097] In the embodiment of the present invention, X′ is quantized, and the quantization formula is: The quantization error formula is: Here, we need to optimize the input quantization factor s and the input softening factor λ to minimize the quantization error L. Since the quantization operation is not differentiable, we optimize s and λ alternately. First, we update s. The update formula is: where s k is the current input quantization factor, λ k is the current input softening factor, k represents the current iteration number, X is the node input value,

[0098] Then use the updated input quantization factor to update the input softening factor. The update formula is: Among them, s k+1 is the updated quantization factor, λ k is the current softening factor. Determine whether the quantization error L is less than the specified first threshold. If it is less than the first threshold, complete the input softening factor and input quantization factor update. If it is greater than the first threshold, use the updated input softening factor λ k+1The input quantization factor is updated again, and the softening factor is updated using the updated quantization factor. The quantization factor and the softening factor are optimized alternately until the quantization error L is less than a specified threshold, thereby obtaining a third input softening factor and a third input quantization factor.

[0099] In the embodiment of the present invention, during the alternating iterative updating process, the initial value of each parameter may be an empirical value or a random value within the range of the empirical value.

[0100] After completing the soft quantization of X′, fix the updated input softening factor λ and recalculate the weight W range, that is, Where W is the current weight, W′ is the weight softened by the softening factor, β is the current weight softening factor, and λ is the third input softening factor obtained by updating in the previous step.

[0101] Then, W′ is quantized, and the quantization formula is: The quantization error formula is: Here we need to optimize the weight quantization factor r and the weight softening factor β to minimize the quantization error L. Since the quantization operation is not differentiable, we alternately optimize r and β. First, we update r. The update formula is: where r k is the current weight quantization factor, β k is the current weight softening factor, k is the current number of iterations, W is the node weight, Then use the updated quantization factor to update the softening factor. The update formula is: where r k+1 is the updated quantization factor, β k is the current weight softening factor. Determine whether the quantization error L is less than the specified first threshold. If it is less than the first threshold, W′ completes the update of the first weight softening factor and the first weight quantization factor. If it is greater than the first threshold, use the updated softening factor β k+1 The quantization factor is updated again, and the softening factor is updated using the updated quantization factor. The quantization factor and the softening factor are optimized alternately until the quantization error L is less than a specified threshold, thereby obtaining a first weighted softening factor and a first weighted quantization factor.

[0102] The softening factor is then updated using the updated quantization factor, and the quantization factor and the softening factor are alternately optimized until the quantization error L is less than a specified threshold, thereby obtaining a first weighted softening factor and a first weighted quantization factor.

[0103] In the embodiment of the present invention, when the node has a weight, the node input and the node weight are jointly soft-quantized based on the input quantization factor and the softening factor and the weight softening factor and the weight quantization factor, and the following steps may also be included:

[0104] S308, obtaining a third node input based on the first weight softening factor, the third input softening factor and the node input;

[0105] S309, dequantizing the input soft quantization result to obtain a second dequantization result; the input soft quantization result is based on the third input softening factor and the first weight softening factor 、 The third input quantization factor and node input are obtained;

[0106] S310, dequantizing the weight soft quantization result to obtain a third dequantization result; the weight soft quantization result is based on the first weight softening factor, the third input softening factor 、 The first weight quantization factor and the node weight are obtained;

[0107] S311, based on the second inverse quantization result, the third inverse quantization result, the node input and the node weight, input and determine a second difference value;

[0108] S312, when the difference value is not less than the second threshold, continue to soft quantize the node input and the node weight jointly with the goal of making the quantization error less than the first threshold, until the second difference value obtained after inverse quantization of the input soft quantization result and the weight soft quantization result obtained by alternating iterative updating is less than the second threshold, and obtain the fourth input softening factor, the fourth input quantization factor, the second weight softening factor, and the second weight quantization factor.

[0109] In the embodiment of the present invention, the updated first weight softening factor β is fixed, and the input X value range is recalculated, that is, Where X is the current input, X′ is the input softened by the softening factor, λ is the third input softening factor obtained by alternating iterative updates, and β is the first weight softening factor obtained by alternating iterative updates.

[0110] Use the third input quantization factor s obtained by alternating iteration to quantize X′ to obtain the input soft quantization result X q ,Right now Dequantization input soft quantization result X q Note: X d , wherein the input soft quantization result is the final input soft quantization result obtained in the alternating iterative update process according to the third input softening factor, the third input quantization factor, the first weight softening factor and the node input. The first weight quantization factor r obtained by the alternating iterative update is used to quantize W′ to obtain the input soft quantization result W q ,Right now Then the dequantization weight soft quantization result W q W d, wherein the weight soft quantization result is the final weight soft quantization result obtained according to the first weight softening factor, the first weight quantization factor, the third input softening factor and the node weight during the alternating iterative update process.

[0111] Calculate the error between the matrix multiplication result of the inverse quantized data and the matrix multiplication result of the node input data, diff = |X d W d -XW|, determine whether the second difference value diff is less than the second threshold. If it is less than the second threshold, complete the input and weight value quantization of this node and save the relevant optimization factors.

[0112] If the current node contains weights when it is greater than the second threshold, the quantization error is less than the first threshold as the goal, and the node input and node weight are soft-quantized together until |X d W d -XW| is less than the second threshold.

[0113] In the embodiment of the present invention, when the node has a weight, the node input and the node weight are jointly soft-quantized based on the input quantization factor and the softening factor and the weight softening factor and the weight quantization factor, and the following steps may also be included:

[0114] S313, saving the fourth input softening factor, the fourth input quantization factor, the second weight softening factor, and the second weight quantization factor;

[0115] S314, saving an input soft quantization result obtained according to the fourth input softening factor, the fourth input quantization factor and the node input;

[0116] S315, saving a weight soft quantization result obtained according to the second weight softening factor, the second weight quantization factor and the node weight.

[0117] In an embodiment of the present invention, after completing the quantization of the node input and weight (parameters), the relevant quantization parameters and data can be included, which can specifically include: a fourth input softening factor, a fourth input quantization factor, a second weight softening factor, a second weight quantization factor, the final input soft quantization result and the final weight soft quantization result.

[0118] Therefore, in an embodiment of the present invention, soft quantization is first performed on the node input, and then soft quantization is performed on the node weight. In the soft quantization process of the node input, softening is performed first, and then quantization is performed. With the quantization error being less than the first threshold as the goal, the input quantization factor and the input softening factor are solved. In the solution process, the input quantization factor and the input softening factor are alternately optimized.

[0119] Specifically, the input quantization factor and the input softening factor can be given initial values ​​at the beginning of the alternating iterative update, and the soft quantization result can be obtained. It is further determined whether the quantization error is less than the first threshold, and the input softening factor is finally fixed, and the input quantization factor is updated. Then, the input quantization factor is fixed and the input softening factor is updated until the quantization error is less than the first threshold. The alternating iterative update is performed in the same way in the soft quantization process of the node input.

[0120] Finally, based on the second difference value, the node input soft quantization result and the node weight soft quantization result are further checked to see whether they meet the preset threshold. If they do not meet the threshold, the alternating iterative update is continued.

[0121] Therefore, in the embodiment of the present invention, the value range of the original data is effectively optimized to obtain a suitable value range through alternating optimization of the softening factor and the quantization factor, and then quantized. The optimized and adjusted data value range can make the quantized data basically retain all the valid information of the original data, and effectively avoid the influence of data outliers on the quantization results. The soft quantization method proposed in the embodiment of the present invention connects the node input value and the weight value into a whole through the softening factor, and through alternating soft quantization of the node input and weight, the error between the quantized input and weight calculation results and the original value calculation results after dequantization is minimized, and an optimal quantization output effect is obtained. The problem of independently quantizing the input and weight separately but the final calculation result is not good

[0122] Based on the same inventive concept, an embodiment of the present invention further provides an offline large model soft quantization device, the offline large model soft quantization device comprising:

[0123] The offline large model node extraction module is used to traverse and extract multiple nodes of the offline large model and save them to the node pool;

[0124] A node soft quantization module is used for, for any node in the node pool, if the node has no weight, to soft quantize the node input of the node based on the input softening factor and the input quantization factor; if the node has weight, to soft quantize the node input and the node weight jointly based on the input quantization factor and the softening factor as well as the weight softening factor and the weight quantization factor; after completing the soft quantization of the node, the soft quantization node parameters of the node are saved;

[0125] The quantization model saving module is used to determine whether all nodes in the node pool have completed soft quantization. When all nodes have completed soft quantization, the parameters of each soft quantization node are saved as a quantization model.

[0126] Optionally, the node soft quantization module is used to:

[0127] Add input softening factor to the node input of this node;

[0128] Obtaining a first node input according to the node input to which an input softening factor is added;

[0129] quantizing the first node input according to the input quantization factor;

[0130] With the quantization error being less than a first threshold as a goal, the input softening factor and the input quantization factor are alternately iteratively updated to obtain a first input softening factor and a first input quantization factor; in any iterative process of the alternate iterative update, a value of one of the input softening factor and the input quantization factor obtained in the previous iterative process is fixed, and the value of the other is updated;

[0131] A first input softening factor, a first quantization factor, and a first soft quantization node input obtained based on the first input softening factor and the first input quantization factor are saved.

[0132] Optionally, the node soft quantization module is used to:

[0133] Dequantizing the first soft quantization node input to obtain a first dequantization result;

[0134] determining a first difference value based on the first inverse quantization result, a first input softening factor, and the node input;

[0135] When the first difference value is not less than the second threshold, continue to iterate the input softening factor and the input quantization factor with the goal of making the quantization error less than the first threshold, until the first difference value obtained after the soft quantization node input obtained by the iterative update is dequantized is less than the second threshold, thereby obtaining a second input softening factor and a second input quantization factor;

[0136] The second input softening factor, the second input quantization factor, and a second soft quantization node input obtained based on the second softening factor and the second quantization factor are saved.

[0137] Optionally, the node soft quantization module is used to:

[0138] Add an input softening factor to the node input of the node, and add a weight softening factor to the node weight;

[0139] Obtaining a second node input according to the node input to which the weight softening factor is added and the input softening factor is added;

[0140] quantizing the second node input according to the input quantization factor;

[0141] The weight softening factor is fixed unchanged, and the quantization error is less than a first threshold as a goal, and the input softening factor and the input quantization factor are alternately iteratively updated to obtain a third input softening factor and a third input quantization factor;

[0142] Obtaining a first node weight according to the node weight added with the third input softening factor and the node weight added with the weight softening factor;

[0143] quantizing the weight of the first node according to the weight quantization factor;

[0144] The third input softening factor obtained by the fixed alternating iterative update remains unchanged, and the weight softening factor and the weight input quantization factor are alternately iteratively updated with the goal of making the quantization error less than the first threshold to obtain the first weight softening factor and the first weight quantization factor.

[0145] Optionally, the node soft quantization module is used to:

[0146] obtaining a third node input based on the first weight softening factor, the third input softening factor and the node input;

[0147] Dequantizing the input soft quantization result to obtain a second dequantization result; the input soft quantization result is obtained according to the third input softening factor, the third input quantization factor, the first weight softening factor and the node input;

[0148] Dequantizing the weight soft quantization result to obtain a third dequantization result; the weight soft quantization result is obtained according to the first weight softening factor, the first weight quantization factor, the third input softening factor and the node weight;

[0149] Based on the second inverse quantization result, the third inverse quantization result, the node input and the node weight, a second difference value is inputted to determine;

[0150] When the difference value is not less than the second threshold, continue to soft quantize the node input and node weight jointly with the goal of making the quantization error less than the first threshold, until the second difference value obtained after inverse quantization of the input soft quantization result and the weight soft quantization result obtained by alternating iterative updating is less than the second threshold, and the fourth input softening factor, the fourth input quantization factor, the second weight softening factor, and the second weight quantization factor are obtained.

[0151] Optionally, the node soft quantization module is used to:

[0152] Saving the fourth input softening factor, the fourth input quantization factor, the second weight softening factor, and the second weight quantization factor;

[0153] Saving an input soft quantization result obtained according to the fourth input softening factor, the fourth input quantization factor and the node input;

[0154] The weight soft quantization result obtained according to the second weight softening factor, the second weight quantization factor and the node weight is saved.

[0155] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes, the steps in the offline large model soft quantization method described in any of the above embodiments are implemented.

[0156] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the offline large model soft quantization method described in any of the above embodiments are implemented.

[0157] Based on the same inventive concept, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the offline large model soft quantization method described in any of the above embodiments.

[0158] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0159] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0160] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (apparatus), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0164] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0165] The above is a detailed introduction to an offline large model soft quantization method provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An offline large model soft quantization method, characterized in that: The method comprises: Traverse and extract multiple nodes of the offline large model and save them to the node pool; For any node in the node pool, if the node has no weight, the node input of the node is soft-quantized based on the input softening factor and the input quantization factor; if the node has weight, the node input and the node weight are jointly soft-quantized based on the input quantization factor and the softening factor as well as the weight softening factor and the weight quantization factor; after completing the soft quantization of the node, the soft quantization node parameters of the node are saved; It is determined whether all nodes in the node pool have completed soft quantization. When all nodes have completed soft quantization, the parameters of each soft quantization node are saved as a quantization model.

2. The off-line large model soft quantization method according to claim 1, characterized in that: When the node has no weight, the node input of the node is soft-quantized based on the input softening factor and the input quantization factor, including: Add input softening factor to the node input of this node; Obtaining a first node input according to the node input to which an input softening factor is added; quantizing the first node input according to the input quantization factor; With the quantization error being less than a first threshold as a goal, the input softening factor and the input quantization factor are alternately iteratively updated to obtain a first input softening factor and a first input quantization factor; in any iterative process of the alternate iterative update, a value of one of the input softening factor and the input quantization factor obtained in the previous iterative process is fixed, and the value of the other is updated; Save the soft quantization node parameters of the node, including: A first input softening factor, a first quantization factor, and a first soft quantization node input obtained based on the first input softening factor and the first input quantization factor are saved.

3. The off-line large model soft quantization method according to claim 2, characterized in that: The method further comprises: Dequantizing the first soft quantization node input to obtain a first dequantization result; determining a first difference value based on the first inverse quantization result, a first input softening factor, and the node input; When the first difference value is not less than the second threshold, continue to iterate the input softening factor and the input quantization factor with the goal of making the quantization error less than the first threshold, until the first difference value obtained after the soft quantization node input obtained by the iterative update is dequantized is less than the second threshold, thereby obtaining a second input softening factor and a second input quantization factor; The step of storing the soft quantization node parameters of the node includes: The second input softening factor, the second input quantization factor, and a second soft quantization node input obtained based on the second softening factor and the second quantization factor are saved.

4. The off-line large model soft quantization method according to claim 1, characterized in that: When the node has a weight, the node input and the node weight are jointly soft-quantized based on the input quantization factor and the softening factor as well as the weight softening factor and the weight quantization factor, including: Add an input softening factor to the node input of the node, and add a weight softening factor to the node weight; Obtaining a second node input according to the node input to which the weight softening factor is added and the input softening factor is added; quantizing the second node input according to the input quantization factor; The weight softening factor is fixed unchanged, and the quantization error is less than a first threshold as a goal, and the input softening factor and the input quantization factor are alternately iteratively updated to obtain a third input softening factor and a third input quantization factor; Obtaining a first node weight according to the node weight added with the third input softening factor and the node weight added with the weight softening factor; quantizing the weight of the first node according to the weight quantization factor; The third input softening factor obtained by the fixed alternating iterative update remains unchanged, and the weight softening factor and the weight input quantization factor are alternately iteratively updated with the goal of making the quantization error less than the first threshold to obtain the first weight softening factor and the first weight quantization factor.

5. The off-line large model soft quantization method according to claim 4, characterized in that: The method further comprises: obtaining a third node input based on the first weight softening factor, the third input softening factor and the node input; Dequantizing the input soft quantization result to obtain a second dequantization result; the input soft quantization result is obtained according to the third input softening factor, the third input quantization factor, the first weight softening factor and the node input; Dequantizing the weight soft quantization result to obtain a third dequantization result; the weight soft quantization result is obtained according to the first weight softening factor, the first weight quantization factor, the third input softening factor and the node weight; Based on the second inverse quantization result, the third inverse quantization result, the node input and the node weight, a second difference value is inputted to determine; When the difference value is not less than the second threshold, continue to soft quantize the node input and node weight jointly with the goal of making the quantization error less than the first threshold, until the second difference value obtained after inverse quantization of the input soft quantization result and the weight soft quantization result obtained by alternating iterative updating is less than the second threshold, and the fourth input softening factor, the fourth input quantization factor, the second weight softening factor, and the second weight quantization factor are obtained.

6. The off-line large-mode soft quantization method according to claim 5, characterized in that: Save soft quantization node parameters, including: Saving the fourth input softening factor, the fourth input quantization factor, the second weight softening factor, and the second weight quantization factor; Saving an input soft quantization result obtained according to the fourth input softening factor, the fourth input quantization factor and the node input; The weight soft quantization result obtained according to the second weight softening factor, the second weight quantization factor and the node weight is saved.

7. An off-line large model soft quantization device, characterized in that: The offline large model soft quantization device comprises: The offline large model node extraction module is used to traverse and extract multiple nodes of the offline large model and save them to the node pool; A node soft quantization module is used for, for any node in the node pool, if the node has no weight, to soft quantize the node input of the node based on the input softening factor and the input quantization factor; if the node has weight, to soft quantize the node input and the node weight jointly based on the input quantization factor and the softening factor as well as the weight softening factor and the weight quantization factor; after completing the soft quantization of the node, the soft quantization node parameters of the node are saved; The quantization model saving module is used to determine whether all nodes in the node pool have completed soft quantization. When all nodes have completed soft quantization, the parameters of each soft quantization node are saved as a quantization model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the off-line large model soft quantization method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the off-line large model soft quantization method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps in the off-line large model soft quantization method according to any one of claims 1 to 6 are implemented.

Images