Parameter determination method, apparatus and device

Abstract

This application provides a parameter determination method, apparatus, and device. The method includes: processing input data through a target layer in a model to obtain a calculation result; determining multiple quantization results of the calculation result under multiple quantization methods, and determining the quantization error of the corresponding quantization method based on the quantization results; determining multiple memory-saving amounts corresponding to the multiple quantization methods, and determining a target quantization method among the multiple quantization methods based on the quantization error and memory-saving amount of each quantization method; and determining the target quantization parameter of the target layer based on the target quantization method. The above method can reduce the memory space occupied by the calculation result.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to a parameter determination method, apparatus and device. Background Technology

[0002] Key-value caching (KV-Cache) technology can be used to reduce the computational cost of models, thereby improving model processing efficiency.

[0003] Key-value caching technology can store the model's computation results in video memory to reduce redundant calculations during model processing. However, as the model decoding length gradually increases, the video memory space occupied by the computation results also increases. This conflicts with limited video memory resources. Therefore, how to reduce the video memory space occupied by the computation results is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a parameter determination method, apparatus, and device to reduce the memory space occupied by the calculation results.

[0005] In a first aspect, embodiments of this application provide a parameter determination method, including:

[0006] The target layer in the model processes the input data to obtain the calculation result;

[0007] Determine the quantization results of the operation under multiple quantization methods, and determine the quantization error of the corresponding quantization method based on the quantization results;

[0008] Determine multiple memory savings corresponding to the various quantization methods, and determine the target quantization method among the various quantization methods based on the quantization error and memory savings of each quantization method;

[0009] The target quantization parameters of the target layer are determined according to the target quantization method.

[0010] In one possible implementation, determining the calculation result under multiple quantization methods includes:

[0011] Determine the various quantization methods and the corresponding quantization parameters for each quantization method;

[0012] The calculation result is quantized using the quantization parameters corresponding to each quantization method to obtain the quantization calculation result corresponding to each quantization method.

[0013] The multiple quantization results include: the quantization results corresponding to each quantization method.

[0014] In one possible implementation, the computation result includes a key tensor and a value tensor; determining the multiple quantization methods includes:

[0015] Determine the multiple key quantization methods corresponding to the key tensor and the multiple value quantization methods corresponding to the value tensor;

[0016] The multiple key quantization methods and the multiple value quantization methods are combined to obtain the multiple quantization methods, which include one key quantization method and one value quantization method.

[0017] In one possible implementation, determining the quantization parameters corresponding to each quantization method includes:

[0018] Based on the key tensor, determine the various key quantization parameters corresponding to the various key quantization methods;

[0019] Based on the value tensor, determine the various value quantization parameters corresponding to the various value quantization methods;

[0020] For any quantization method, the key quantization parameters corresponding to the key quantization method included in the quantization method and the value quantization parameters corresponding to the value quantization method included in the quantization method are determined as the quantization parameters corresponding to the quantization method.

[0021] In one possible implementation, based on the key tensor, determining multiple key quantization parameters corresponding to the multiple key quantization methods includes:

[0022] Determine the quantization type for each of the key quantization methods, wherein the quantization type includes at least integer quantization and / or floating-point quantization;

[0023] Based on the key tensor and the quantization type of each key quantization method, determine the key quantization parameters corresponding to each key quantization method;

[0024] The various key quantization parameters include key quantization parameters corresponding to each key quantization method.

[0025] In one possible implementation, for any key quantization method, the key quantization parameters corresponding to the key quantization method are determined based on the key tensor and the quantization type of the key quantization method, including:

[0026] If the quantization type is integer quantization, determine the first space occupancy corresponding to the key quantization method. After quantizing the key tensor using the key quantization method, the size of the quantized key tensor is equal to the first space occupancy.

[0027] The key scaling factor corresponding to the key quantization method is determined based on the key tensor and the first space occupancy, and the key offset corresponding to the key quantization method is determined based on the key tensor.

[0028] The key quantization parameters corresponding to the key quantization method include: the key scaling factor corresponding to the key quantization method and the key offset corresponding to the key quantization method.

[0029] In one possible implementation, determining the key scaling factor corresponding to the key quantization method based on the key tensor and the first space occupancy includes:

[0030] Determine the first difference between the maximum and minimum elements in the key tensor;

[0031] Determine a second difference between the function value of a preset exponential function and a first preset value, wherein the base of the preset exponential function is the second preset value and the exponent is the first space occupancy;

[0032] The ratio of the second difference to the first difference is determined as the key scaling factor corresponding to the key quantization method.

[0033] In one possible implementation, determining the key offset corresponding to the key quantization method based on the key tensor includes:

[0034] Determine the minimum element in the key tensor and set the minimum element as the key offset.

[0035] In one possible implementation, for any key quantization method, the key quantization parameters corresponding to the key quantization method are determined based on the key tensor and the quantization type of the key quantization method, including:

[0036] If the quantization type is floating-point quantization, determine the first exponent bit value range of the current data format of the key tensor, where the first exponent bit value range is the exponent range corresponding to the exponent bit of the current data format;

[0037] Determine the range of the second exponent bit values ​​for the data format corresponding to the key quantization method, wherein the range of the second exponent bit values ​​is the range of the exponent bits corresponding to the exponent bits of the data format corresponding to the key quantization method;

[0038] Based on the first exponent bit value range and the second exponent bit value range, determine the key scaling hyperparameter corresponding to the key quantization method;

[0039] The key quantization parameters corresponding to the key quantization method include: the key scaling hyperparameter corresponding to the key quantization method.

[0040] In one possible implementation, determining the key scaling hyperparameter corresponding to the key quantization method based on the first exponent bit value range and the second exponent bit value range includes:

[0041] Determine the first minimum and first maximum values ​​of the first exponent value range, and determine the second minimum and second maximum values ​​of the second exponent value range;

[0042] Determine the third difference between the first minimum value and the second minimum value, and determine the fourth difference between the first maximum value and the second maximum value;

[0043] Based on the third difference and the fourth difference, a plurality of candidate key scaling hyperparameters are determined. The candidate key scaling hyperparameters are the third preset value raised to the power of i, where the minimum value of i is the third difference and the maximum value of i is the fourth difference.

[0044] The key scaling hyperparameters are determined based on the key tensor and the plurality of candidate key scaling hyperparameters.

[0045] In one possible implementation, determining the key scaling hyperparameters based on the key tensor and the plurality of candidate key scaling hyperparameters includes:

[0046] The key tensor is quantized using the scaling hyperparameter of each candidate key to obtain multiple candidate key tensors;

[0047] Calculate the error between each candidate key tensor and the key tensor to obtain multiple first errors;

[0048] The candidate key scaling hyperparameter corresponding to the minimum value among the plurality of first errors is determined as the key scaling hyperparameter.

[0049] In one possible implementation, determining the quantization error of the corresponding quantization method based on the quantization calculation result includes:

[0050] Attention is calculated on the results of the calculation to obtain a standard value;

[0051] Attention is calculated for each quantization result to obtain the candidate value corresponding to each quantization result;

[0052] Calculate the error between the candidate value and the standard value corresponding to each quantization operation result, and determine the error between the candidate value and the standard value corresponding to the quantization operation result as the quantization error of the quantization method corresponding to the quantization operation result.

[0053] In one possible implementation, for any quantization method, determining the memory savings of the quantization method includes:

[0054] The quantization method includes the candidate key quantization method and the candidate value quantization method;

[0055] Determine the key memory saving amount corresponding to the candidate key quantization method, and determine the value memory saving amount corresponding to the candidate value quantization method;

[0056] The memory saving amount of the quantization method is determined based on the key memory saving amount and the value memory saving amount.

[0057] In one possible implementation, determining a target quantization method among the multiple quantization methods based on the quantization error and memory savings of each quantization method includes:

[0058] Determine the first weight corresponding to the quantization error and the second weight corresponding to the memory saving amount;

[0059] Based on the quantization error and memory savings of each quantization method, the first weight, and the second weight, a weighting value corresponding to each quantization method is determined, resulting in multiple weighting values.

[0060] Based on the multiple weighted values, the target quantization method is determined among the multiple quantization methods.

[0061] In one possible implementation, for any quantization method, a weighted value corresponding to the quantization method is determined based on the quantization error and memory savings of the quantization method, the first weight, and the second weight, including:

[0062] Determine the first product of the quantization error of the quantization method and the first weight, and determine the second product of the memory saving amount of the quantization method and the second weight;

[0063] The sum of the first product and the second product is determined as the weighted value corresponding to the quantization method.

[0064] In one possible implementation, determining the target quantization method among the multiple quantization methods based on the plurality of weighted values ​​includes:

[0065] Determine the minimum value among the multiple weighted values, and determine the quantization method corresponding to the minimum value as the target quantization method.

[0066] In one possible implementation, determining multiple value quantization parameters corresponding to the multiple value quantization methods based on the value tensor includes:

[0067] Determine the quantization type for each of the value quantization methods, wherein the quantization type includes at least integer quantization and / or floating-point quantization;

[0068] Based on the value tensor and the quantization type of each value quantization method, determine the value quantization parameters corresponding to each value quantization method;

[0069] The multiple value quantization parameters include: value quantization parameters corresponding to each value quantization method.

[0070] In one possible implementation, for any value quantization method, the value quantization parameters corresponding to the value quantization method are determined based on the value tensor and the quantization type of the value quantization method, including:

[0071] If the quantization type is integer quantization, determine the second space occupancy corresponding to the value quantization method. After quantizing the value tensor by the value quantization method, the size of the quantized value tensor is equal to the second space occupancy.

[0072] The value scaling factor corresponding to the value quantization method is determined based on the value tensor and the second space occupancy, and the value offset corresponding to the value quantization method is determined based on the value tensor.

[0073] The value quantization parameters corresponding to the value quantization method include: the value scaling factor corresponding to the value quantization method and the value offset corresponding to the value quantization method.

[0074] In one possible implementation, determining the value scaling factor corresponding to the value quantization method based on the value tensor and the second space occupancy includes:

[0075] Determine the fifth difference between the maximum and minimum elements in the value tensor;

[0076] Determine the sixth difference between the function value of the preset exponential function and the first preset value, wherein the base of the preset exponential function is the second preset value and the exponent is the second space occupancy;

[0077] The ratio of the sixth difference to the fifth difference is determined as the value scaling factor corresponding to the value quantization method.

[0078] In one possible implementation, determining the value offset corresponding to the value quantization method based on the value tensor includes:

[0079] Determine the minimum element in the value tensor and set the minimum element as the value offset.

[0080] In one possible implementation, for any value quantization method, the value quantization parameters corresponding to the value quantization method are determined based on the value tensor and the quantization type of the value quantization method, including:

[0081] If the quantization type is floating-point quantization, determine the range of the third exponent bit value of the current data format of the value tensor, where the range of the third exponent bit value is the exponent range corresponding to the exponent bit of the current data format of the value tensor.

[0082] Determine the range of the fourth exponent bit values ​​for the data format corresponding to the value quantization method, wherein the range of the fourth exponent bit values ​​is the exponent range corresponding to the exponent bit of the data format corresponding to the value quantization method.

[0083] Based on the range of values ​​for the third exponent and the range of values ​​for the fourth exponent, determine the value scaling hyperparameter corresponding to the value quantization method;

[0084] The value quantization parameters corresponding to the value quantization method include: the value scaling hyperparameter corresponding to the value quantization method.

[0085] In one possible implementation, determining the value scaling hyperparameter corresponding to the value quantization method based on the third exponent value range and the fourth exponent value range includes:

[0086] Determine the third minimum and third maximum values ​​of the third exponent value range, and determine the fourth minimum and fourth maximum values ​​of the fourth exponent value range;

[0087] Determine the seventh difference between the third minimum value and the fourth minimum value, and determine the eighth difference between the third maximum value and the fourth maximum value;

[0088] Based on the seventh difference and the eighth difference, a plurality of candidate value scaling hyperparameters are determined. The candidate value scaling hyperparameters are the j-th power of the third preset value, the minimum value of j is the seventh difference, and the maximum value of j is the eighth difference.

[0089] The value scaling hyperparameters are determined based on the value tensor and the plurality of candidate value scaling hyperparameters.

[0090] In one possible implementation, determining the value scaling hyperparameters based on the value tensor and the plurality of candidate value scaling hyperparameters includes:

[0091] The value tensor is quantized using the scaling hyperparameter of each candidate value to obtain multiple candidate value tensors;

[0092] Calculate the error between each of the candidate value tensors and the value tensor to obtain multiple second errors;

[0093] The candidate value scaling hyperparameter corresponding to the minimum value among the plurality of second errors is determined as the value scaling hyperparameter.

[0094] Secondly, embodiments of this application provide a parameter determination device, the device comprising a processing module and a determination module, wherein,

[0095] The processing module is used to perform calculations on the input data through the target layer in the model to obtain the calculation results;

[0096] The determining module is used to determine the calculation result under multiple quantization methods, and to determine the quantization error of the corresponding quantization method based on the quantization result.

[0097] The determining module is further configured to determine multiple memory savings corresponding to the multiple quantization methods, and determine the target quantization method among the multiple quantization methods based on the quantization error and memory savings of each quantization method.

[0098] The determining module is further configured to determine the target quantization parameters of the target layer according to the target quantization method.

[0099] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0100] The memory stores computer-executed instructions;

[0101] The processor executes computer execution instructions stored in the memory, causing the electronic device to perform the method described in any of the first aspects.

[0102] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in any of the first aspects.

[0103] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the method shown in any of the first aspects.

[0104] This application provides a parameter determination method, apparatus, and device. The method can process input data through a target layer in a model to obtain calculation results; it can determine multiple quantization results under various quantization methods, and determine the quantization error of the corresponding quantization method based on the quantization results; it can determine multiple memory savings corresponding to the various quantization methods, and determine a target quantization method among the various quantization methods based on the quantization error and memory savings of each quantization method; and it can determine the target quantization parameters of the target layer based on the target quantization method. In the above method, the target quantization parameters of each layer of the model can be determined, so that the corresponding calculation results can be quantized according to the target quantization parameters of each layer during the inference process through the model. Through the above method, the memory space occupied by the calculation results can be reduced, thereby saving memory space. Furthermore, the above method also considers the quantization error of each layer of the model, achieving the goal of reducing the memory space occupied by the calculation results while ensuring the accuracy of the model. Attached Figure Description

[0105] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0106] Figure 1 A schematic diagram of an application scenario provided for an exemplary embodiment of this application;

[0107] Figure 2 A flowchart illustrating a parameter determination method provided for an exemplary embodiment of this application;

[0108] Figure 3 A flowchart illustrating another parameter determination method provided for an exemplary embodiment of this application;

[0109] Figure 4 A flowchart illustrating yet another parameter determination method provided for an exemplary embodiment of this application;

[0110] Figure 5 A flowchart illustrating yet another parameter determination method provided for an exemplary embodiment of this application;

[0111] Figure 6 A schematic diagram of a parameter determination device provided for an exemplary embodiment of this application;

[0112] Figure 7 This application provides a schematic diagram of the structure of an electronic device as an exemplary embodiment. Detailed Implementation

[0113] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0114] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0115] To facilitate understanding, the technical terms involved in the embodiments of this application will be explained first.

[0116] Large language (Transformer) models are deep learning models trained on massive text datasets, often with billions or even trillions of parameters. These models are capable of understanding, generating, and manipulating human language, thus demonstrating superior performance across a variety of natural language processing tasks.

[0117] Attention Mechanism: A key feature of Transformer models, it allows the model to give different attention to different parts of a sequence when processing it. In standard Transformer models, this is achieved by computing query, key, and value vectors. In sequence generation tasks, such as machine translation or text generation, the decoder generates the output sequence step by step. Each time a new element is generated, attention weights need to be computed and applied to all previous elements.

[0118] Key-value caching (KV-Cache): This generally refers to a technique used to optimize model performance and improve efficiency. Specifically, in large language models, especially in the decoder part, it involves caching previously computed key and value vectors to avoid recalculation when generating each new word. Using KV-Cache can significantly reduce the computational load of the model, effectively accelerating the decoding process by reducing redundant calculations, which is particularly useful when generating coherent and long sequences of text.

[0119] Head size: In large language models, K and V consist of several heads, each of which performs attention operations independently. Here, head size refers to the size of each head.

[0120] To facilitate understanding, the following will be combined with Figure 1 The application scenarios of the embodiments of this application will be described.

[0121] Figure 1 This is a schematic diagram illustrating an application scenario provided for an exemplary embodiment of this application. Please refer to [link / reference]. Figure 1 By processing the data to be processed through the model, corresponding operation results can be generated for each layer. The operation results can include key tensors and value tensors. For example, the model can be a large language model; after processing the data to be processed through the model, key tensors and value tensors of the first layer, the second layer, ..., and the Nth layer can be generated.

[0122] Currently, after generating the key and value tensors for each layer through the model, these tensors can be stored in GPU memory. This eliminates the need to repeatedly calculate the key and value tensors for each layer during model processing; they can be directly retrieved from GPU memory. However, with the continuous development of large language models, the decoding length of the models is constantly increasing, and the GPU memory space required to store the key and value tensors is also constantly increasing, which conflicts with limited GPU memory resources. Therefore, how to reduce the GPU memory space occupied by the computation results is an urgent problem to be solved.

[0123] In view of this, the embodiments of this application propose the following technical concept: while ensuring the processing accuracy of each layer of the model, the calculation results of each layer can be quantized separately. This achieves the goal of reducing the memory space occupied by the calculation results while maintaining the model's processing accuracy. Furthermore, quantizing the calculation results of each layer separately allows for the flexible determination of the optimization method for the calculation results of each layer based on the accuracy requirements of each layer of the model, further ensuring the processing accuracy of the model.

[0124] The technical solutions shown in this application will now be described in detail through specific embodiments. It should be noted that the following embodiments may exist independently or in combination with each other; for identical or similar content, the description will not be repeated in different embodiments.

[0125] It should be noted that the parameter determination method provided in this application can be used to determine the quantization parameters corresponding to any layer of the model, and the method for determining the quantization parameters corresponding to each layer of the model is consistent. In the following embodiments, only any layer of the model is used as an example to illustrate the method for determining the quantization parameters.

[0126] Figure 2 This is a flowchart illustrating a parameter determination method provided for an exemplary embodiment of this application. The method can be executed by an electronic device or a processor located within the electronic device. The following description uses an electronic device as the executing entity. Please refer to... Figure 2 The method may include:

[0127] S201. The input data is processed through the target layer in the model to obtain the calculation result.

[0128] The model can be any large language model. For example, the model can be the Llama2 model, etc.

[0129] The target layer in the model can be any layer of the model.

[0130] The input data can be the input data of the target layer.

[0131] The result of the operation can be the processing result obtained by the target layer after processing the input data. The result can include the key tensor and value tensor of the target layer.

[0132] The computational processing can refer to the operations performed on the input data by the target layer in the model. It should be understood that the computational processing corresponding to different target layers can be the same or different.

[0133] This embodiment can be applied to different models, making the parameter determination method provided in this embodiment applicable to a wide range of scenarios.

[0134] S202. Determine the quantization results under various quantization methods, and determine the quantization error of the corresponding quantization method based on the quantization results.

[0135] Quantization methods can include key quantization for key tensors and value quantization for value tensors. In this embodiment, there can be multiple key quantization methods, and there can also be multiple value quantization methods. Combining multiple key quantization methods with multiple value quantization methods can yield a variety of quantization methods.

[0136] For example, multiple key quantization methods, multiple value quantization methods, and multiple quantization methods can be shown in Table 1:

[0137] Table 1

[0138]

[0139] As shown in Table 1, it is assumed that the key quantization methods for key tensors can include key quantization method c and key quantization method d, and the value quantization methods for value tensors can include value quantization method a and value quantization method b. Combining key quantization methods with value quantization methods can yield multiple quantization methods, including: key quantization method c and value quantization method a, key quantization method c and value quantization method b, key quantization method d and value quantization method a, and key quantization method d and value quantization method b.

[0140] In this embodiment, for each quantization method, the calculation result can be quantized by the quantization method to obtain the quantized calculation result corresponding to the calculation result.

[0141] Specifically, a key tensor can be quantized using key quantization to obtain a quantized key tensor, and a value tensor can be quantized using value quantization to obtain a quantized value tensor. The quantization result can include both a quantized key tensor and a quantized value tensor.

[0142] For example, assuming the quantization methods are key quantization (c) and value quantization (a), a key tensor can be quantized using key quantization (c) to obtain a quantized key tensor, and a value tensor can be quantized using value quantization (a) to obtain a quantized value tensor. The quantization results can include both quantized key tensors and quantized value tensors.

[0143] In this embodiment, attention calculation can be performed on the calculation result to obtain a standard value; attention calculation can be performed on each quantization calculation result to obtain the candidate value corresponding to each quantization calculation result; the error between each candidate value and the standard value can be calculated; and the error between the candidate value and the standard value corresponding to the quantization calculation result can be determined as the quantization error of the quantization method corresponding to the quantization calculation result.

[0144] The error between the candidate value and the standard value can be the mean square error (MSE) or the root mean square error (RMSE).

[0145] S203. Determine multiple memory savings corresponding to various quantization methods, and determine the target quantization method among the various quantization methods based on the quantization error and memory savings of each quantization method.

[0146] Memory savings can include memory savings for key tensors and memory savings for value tensors.

[0147] The memory saving of a key tensor can be the memory space occupied by the quantized key tensor, or the amount of memory saving relative to the memory space occupied by the key tensor.

[0148] The memory saving of a value tensor can be the amount of memory space saved relative to the memory space occupied by the quantized value tensor.

[0149] It should be noted that the data format of key tensors and value tensors can be either half-precision floating-point (FP16) or single-precision floating-point (FP32), etc. It should be understood that storing FP16 format data requires 16 bits of storage space, and storing FP32 format data requires 32 bits of storage space. That is, storing key tensors requires either 16 bits or 32 bits of storage space, and storing value tensors also requires either 16 bits or 32 bits of storage space.

[0150] In this embodiment, the data format of the quantization key tensor and the quantization value tensor can be 8-bit floating-point (FP8) format, 8-bit integer (INT8) format, or 4-bit integer (INT4) format, etc. It should be understood that storing FP8 format data requires 8 bits of storage space, storing INT8 format data also requires 8 bits of storage space, and storing INT4 format data requires 4 bits of storage space. That is, storing the quantization key tensor requires either 8 bits or 4 bits of storage space, and storing the value tensor also requires either 8 bits or 4 bits of storage space.

[0151] If the key / value tensor data format is FP16 and the quantized key / value tensor data format is FP8, then the key / value tensor requires 16 bits of video memory, and the quantized key / value tensor requires 8 bits of video memory. Therefore, the video memory savings of the key tensor can be 50% compared to the value tensor.

[0152] If the key / value tensor data format is FP16 and the quantized key / value tensor data format is INT8, then the key / value tensor requires 16 bits of video memory, and the quantized key / value tensor requires 8 bits of video memory. Therefore, the video memory savings of the key tensor can be 50% compared to the value tensor.

[0153] If the data format of the key tensor / value tensor is FP16, and the data format of the quantized key tensor / quantized value tensor is INT4, then the video memory space required for the key tensor / value tensor is 16 bits, and the video memory space required for the quantized key tensor / quantized value tensor is 4 bits. Therefore, the video memory saving of the key tensor can be 75% compared to the video memory saving of the value tensor.

[0154] In this embodiment, the amount of video memory saved can be determined based on the data format of the key tensor, the data format of the value tensor, the data format of the quantized key tensor, and the data format of the quantized value tensor.

[0155] Specifically, the memory savings of key tensors can be determined based on their data format and the data format of quantized key tensors; the memory savings of value tensors can be determined based on their data format and the data format of quantized value tensors; and the total memory savings can be determined based on both key and value tensor memory savings. The memory savings from key tensors can account for 50% of the total memory savings, and the memory savings from value tensors can also account for 50%.

[0156] Specifically, the memory saving amount of the key tensor can be determined, and the third product of the memory saving amount of the key tensor and its proportion of the total memory saving amount can be determined; the memory saving amount of the value tensor can be determined, and the fourth product of the memory saving amount of the value tensor and its proportion of the total memory saving amount can be determined; and the sum of the third product and the fourth product can be determined as the memory saving amount.

[0157] For example, assuming the key tensor data format is FP16 and the quantized key tensor data format is FP8, the memory saving of the key tensor is 50%; assuming the value tensor data format is FP16 and the quantized value tensor data format is INT4, the memory saving of the value tensor is 75%; therefore, the memory saving = 50% * 50% + 75% * 50% = 62.5%.

[0158] In this embodiment, the quantization error of each quantization method and the memory saving amount of each quantization method can be determined by the above method.

[0159] In practice, a first weight for quantization error and a second weight for memory savings can be pre-set according to usage requirements. For any quantization method, a weighted value can be determined based on the quantization error, memory savings, first weight, and second weight. This allows for the determination of multiple weighted values ​​for various quantization methods, and the selection of a target quantization method based on these values. For example, the quantization method corresponding to the smallest weighted value can be chosen as the target quantization method.

[0160] S204. Determine the target quantization parameters of the target layer according to the target quantization method.

[0161] Quantization parameters are parameters used during the quantization process of the computation result. Quantization parameters can include key quantization parameters and value quantization parameters. Key quantization parameters are used when quantizing key tensors. Value quantization parameters are used when quantizing value tensors.

[0162] In this embodiment, each quantization method has corresponding quantization parameters. For any given quantization method, the corresponding quantization parameters include key quantization parameters and value quantization parameters.

[0163] In this embodiment, the quantization parameters corresponding to the target quantization method can be determined as the target quantization parameters of the target layer.

[0164] It should be noted that the method for determining the quantization parameters for each quantization method can be found in [reference needed]. Figure 4 Examples and Figure 5 Examples are not detailed here.

[0165] The parameter determination method provided in this embodiment can process input data through the target layer in the model to obtain the calculation result; it can determine multiple quantization results under various quantization methods, and determine the quantization error of the corresponding quantization method based on the quantization result; it can determine multiple memory savings corresponding to various quantization methods, and determine the target quantization method among the various quantization methods based on the quantization error and memory savings of each quantization method; and it can determine the target quantization parameter of the target layer based on the target quantization method. In the above method, the target quantization parameter of each layer of the model can be determined, so that the corresponding calculation result can be quantized according to the target quantization parameter of each layer during the inference process through the model. Through the above method, the memory space occupied by the calculation result can be reduced, thereby saving memory space. In addition, the above method also considers the quantization error of each layer of the model, achieving the goal of reducing the memory space occupied by the calculation result while ensuring the accuracy of the model.

[0166] Based on the above embodiments, the following is combined with Figure 3 The method for determining parameters provided in the embodiments of this application will be further explained.

[0167] Figure 3 This is a flowchart illustrating another parameter determination method provided for an exemplary embodiment of this application. The execution subject of this method can be an electronic device or a processor located within the electronic device. The following description uses an electronic device as an example. Please refer to... Figure 3 The method may include:

[0168] S301. The input data is processed through the target layer in the model to obtain the calculation result.

[0169] It should be noted that the specific implementation of S301 can be found in S201, and will not be repeated here.

[0170] S302. Determine multiple quantification methods.

[0171] In this embodiment, the calculation result may include a key tensor and a value tensor. When multiple quantization methods are determined, multiple key quantization methods corresponding to the key tensor and multiple value quantization methods corresponding to the value tensor can be determined. Multiple key quantization methods and multiple value quantization methods are combined to obtain multiple quantization methods, including one key quantization method and one value quantization method.

[0172] Key quantization can be performed using key tensor quantization methods. Key quantization methods include, but are not limited to, FP8 format quantization, INT8 format quantization, and INT4 format quantization.

[0173] Value quantization can be performed using value tensor quantization methods. Value quantization methods include, but are not limited to, FP8 format quantization, INT8 format quantization, and INT4 format quantization.

[0174] Among them, FP8 format quantization can include: FP8 format quantization with 5 exponent and 2 mantissa (FP8-E5M2 format quantization), FP8 format quantization with 4 exponent and 3 mantissa (FP8-E4M3 format quantization), and FP8 format quantization with 3 exponent and 4 mantissa (FP8-E3M4 format quantization).

[0175] In practice, for any given key quantization method, the key quantization method can be combined with each value quantization method. In this way, multiple quantization methods can be determined.

[0176] S303. Determine the quantization parameters corresponding to each quantization method.

[0177] The quantization parameters for each quantization method can include key quantization parameters and value quantization parameters.

[0178] In this embodiment, when determining the quantization parameters corresponding to each quantization method, multiple key quantization parameters corresponding to multiple key quantization methods can be determined based on the key tensor; multiple value quantization parameters corresponding to multiple value quantization methods can be determined based on the value tensor; for any quantization method, the key quantization parameters corresponding to the key quantization methods included in the quantization method and the value quantization parameters corresponding to the value quantization methods included in the quantization method are determined as the quantization parameters corresponding to the quantization method.

[0179] In this embodiment, depending on the quantization type of the key quantization method, the key quantization parameters can include at least the following two cases:

[0180] Case 1: The quantization type of the key quantization method is integer quantization.

[0181] Integer quantization is used to indicate that the data format of the quantization result is integer. For example, INT8 and INT4 quantization can be integer quantization.

[0182] In this case, the key quantization parameters may include the key scaling factor and the key offset.

[0183] Case 2: The quantization type of the key quantization method is floating-point quantization.

[0184] Floating-point quantization is used to indicate that the data format of the quantization result is floating-point. For example, the quantization type of FP8 format quantization can be floating-point quantization.

[0185] In this case, the key quantization parameters may include the key scaling hyperparameter.

[0186] In this embodiment, the quantization type of the value quantization method can also include integer quantization and floating-point quantization. If the quantization type of the value quantization method is integer quantization, the value quantization parameters can include a value scaling factor and a value offset. If the quantization type of the value quantization method is floating-point quantization, the value quantization parameters can include a value scaling hyperparameter.

[0187] It should be noted that methods for determining multiple key quantization parameters corresponding to various key quantization methods based on key tensors, and methods for determining multiple value quantization parameters corresponding to various value quantization methods based on value tensors, can be found in [reference needed]. Figure 4 Examples and Figure 5 Implementation examples. That is, methods for determining the key scaling factor and key offset corresponding to the key quantization method based on the key tensor, or methods for determining the key scaling hyperparameter corresponding to the key quantization method based on the key tensor, and methods for determining the value scaling factor and value offset corresponding to the value quantization method based on the value tensor, or methods for determining the value scaling hyperparameter corresponding to the value quantization method based on the value tensor, can be found in [reference needed]. Figure 4 Examples and Figure 5 Examples are not detailed here.

[0188] S304. Quantize the calculation results using the quantization parameters corresponding to each quantization method to obtain the quantization calculation results corresponding to each quantization method.

[0189] The result of quantization can be the result obtained after quantizing the calculation result.

[0190] In this embodiment, for any quantization method, the calculation result can be quantized using the quantization parameters corresponding to that quantization method to obtain the quantization calculation result corresponding to that quantization method. Thus, by quantizing the calculation result using the quantization parameters corresponding to multiple quantization methods, multiple quantization calculation results can be obtained.

[0191] It should be noted that, for any given quantization result, the memory space occupied by the quantization result is less than the memory space occupied by the calculation result. By quantizing the calculation result using the method provided in this embodiment, the goal of saving memory space can be achieved.

[0192] The results of quantization operations can include quantization key tensors and quantization value tensors.

[0193] It should be noted that the method for quantizing the calculation result using the quantization parameters corresponding to each quantization method is consistent. Below, we will use any one quantization method as an example to explain the method for quantizing the calculation result using the quantization parameters corresponding to that quantization method.

[0194] Specifically, for any quantization method, the candidate key quantization method and candidate value quantization method can be determined; the key tensor can be quantized according to the quantization parameters corresponding to the candidate key quantization method to obtain the quantized key tensor; the value tensor can be quantized according to the quantization parameters corresponding to the candidate value quantization method to obtain the quantized value tensor.

[0195] In this embodiment, if the quantization types of the candidate key quantization methods are different, the methods for quantizing the key tensor according to the quantization parameters corresponding to the candidate key quantization methods will also be different. The methods for quantizing the key tensor according to the quantization parameters corresponding to the candidate key quantization methods can include at least the following two cases:

[0196] Case 1: The quantization type of the candidate key quantization method is integer quantization.

[0197] In this case, the key quantization parameters can include key scaling factors and key offsets. The key tensor can be quantized using the following formula to obtain the quantized key tensor:

[0198] Quantization key tensor = (key tensor - key offset) × key scaling factor

[0199] Case 2: The quantization type of the candidate key quantization method is floating-point quantization.

[0200] In this case, the key quantization parameters may include key scaling hyperparameters. The key tensor can be quantized to obtain a quantized key tensor in the following manner:

[0201] 1) Determine the intermediate key tensor by multiplying the key tensor by the key scaling hyperparameter;

[0202] 2) Convert the data format of the intermediate key tensor to the data format corresponding to the quantization method of the candidate key to obtain the quantized key tensor.

[0203] For example, suppose the key quantization method to be selected is FP8-E5M2 format quantization. When quantizing the key tensor, the product of the key tensor and the key scaling hyperparameter can be used to determine the intermediate key tensor; and the data format of the intermediate key tensor can be converted to FP8-E5M2 data format to obtain the quantized key tensor.

[0204] Similarly, if the quantization types of the candidate value quantization methods are different, the methods for quantizing the value tensor according to the quantization parameters corresponding to the candidate value quantization methods will also be different. The methods for quantizing the value tensor according to the quantization parameters corresponding to the candidate value quantization methods can include at least the following two cases:

[0205] Case 1: The quantization type of the candidate value quantization method is integer quantization.

[0206] In this case, the value quantization parameters can include a value scaling factor and a value offset. The value tensor can be quantized using the following formula to obtain a quantized value tensor:

[0207] Quantized value tensor = (value tensor - value offset) × value scaling factor

[0208] Case 2: The quantization type of the candidate value quantization method is floating-point quantization.

[0209] In this case, the value quantization parameters may include value scaling hyperparameters. The value tensor can be quantized to obtain a quantized value tensor in this manner:

[0210] 1) Determine the intermediate value tensor by multiplying the value tensor by the value scaling hyperparameter;

[0211] 2) Convert the data format of the intermediate value tensor to the data format corresponding to the quantization method of the candidate value to obtain the quantized value tensor.

[0212] For example, suppose the quantization method for the candidate value is FP8-E3M4 format quantization. When quantizing a value tensor, the product of the value tensor and the value scaling hyperparameter can be used to determine the intermediate value tensor; and the data format of the intermediate value tensor can be converted to FP8-E3M4 data format to obtain the quantized value tensor.

[0213] S305. Determine the quantization error of the corresponding quantization method based on the quantization calculation results.

[0214] In this embodiment, when determining the quantization error of the corresponding quantization method based on the quantization calculation result, attention calculation can be performed on the calculation result to obtain a standard value; attention calculation can be performed on each type of quantization calculation result to obtain the candidate value corresponding to each type of quantization calculation result; the error between the candidate value and the standard value corresponding to each type of quantization calculation result can be calculated, and the error between the candidate value and the standard value corresponding to the quantization calculation result is determined as the quantization error of the quantization method corresponding to the quantization calculation result.

[0215] Specifically, the formula for calculating attention on the computation result can be as follows:

[0216] Standard value = softmax(q·k) T )v

[0217] Where q can be the query tensor corresponding to the input data, and k can be the key tensor corresponding to the input data. T It can be the transpose of k, and v can be the value tensor corresponding to the input data.

[0218] It should be noted that the result of the operation can also include the query tensor corresponding to the input data. That is, the result can include q, k, and v.

[0219] The formula for calculating attention based on the quantization results can be shown below:

[0220] Candidate value = softmax(q·k1) T v1

[0221] Where q can be the query tensor corresponding to the input data, and k1 can be the quantization key tensor. T It can be the transpose of k1, and v1 can be a quantized tensor.

[0222] In this embodiment, after determining the standard value and the candidate value using the above formula, the error between the candidate value and the standard value corresponding to each quantization result can be calculated separately, and the error between the candidate value and the standard value corresponding to the quantization result can be determined as the quantization error of the quantization method corresponding to the quantization result.

[0223] S306. Determine the memory savings for each quantization method.

[0224] It should be noted that the method for determining the memory savings for each quantization method is the same. Below, we will use any one quantization method as an example to explain the method for determining the memory savings for each quantization method.

[0225] When determining the memory saving amount of any quantization method, the candidate key quantization method and candidate value quantization method can be identified; the key memory saving amount corresponding to the candidate key quantization method and the value memory saving amount corresponding to the candidate value quantization method can be determined; and the memory saving amount of the quantization method can be determined based on the key memory saving amount and the value memory saving amount.

[0226] It should be noted that the methods for determining the key memory saving amount corresponding to the candidate key quantization method, the methods for determining the value memory saving amount corresponding to the candidate value quantization method, and the methods for determining the memory saving amount of the quantization method based on the key memory saving amount and the value memory saving amount can be found in S203, and will not be repeated here.

[0227] S307. Based on the quantization error and memory saving of each quantization method, determine the target quantization method among multiple quantization methods.

[0228] In this embodiment, when determining the target quantization method, the first weight corresponding to the quantization error and the second weight corresponding to the memory saving amount can be determined; based on the quantization error and memory saving amount, the first weight and the second weight of each quantization method, the weighted value corresponding to each quantization method is determined, and multiple weighted values ​​are obtained; based on the multiple weighted values, the target quantization method is determined among multiple quantization methods.

[0229] It should be noted that the method for determining the weighting value corresponding to any quantization method is consistent. Below, we will use any quantization method as an example to explain the method for determining the weighting value corresponding to that quantization method.

[0230] For any quantization method, when determining the weighting value corresponding to the quantization method, the quantization error of the quantization method and the first product of the first weight can be determined, and the memory saving amount of the quantization method and the second product of the second weight can be determined; the sum of the first product and the second product is determined as the weighting value corresponding to the quantization method.

[0231] For example, assuming the first weight is 60%, the second weight is 40%, the quantization error of the quantization method is 5%, and the memory saving of the quantization method is 62.5%, then the weighted value corresponding to the quantization method = 60% * 5% + 40% * 62.5% = 28%.

[0232] In this embodiment, the weighting value of each quantization method can be determined by the above method, resulting in multiple weighting values; the minimum value among the multiple weighting values ​​can be determined, and the quantization method corresponding to the minimum value can be determined as the target quantization method.

[0233] S308. Determine the target quantization parameters of the target layer according to the target quantization method.

[0234] It should be noted that the specific implementation of S308 can be found in S204, and will not be repeated here.

[0235] The parameter determination method provided in this embodiment can process input data through the target layer in the model to obtain the calculation result; it can determine multiple quantization methods; it can determine the quantization parameters corresponding to each quantization method; it can quantize the calculation result according to the quantization parameters corresponding to each quantization method to obtain the quantization calculation result corresponding to each quantization method; it can determine the quantization error of the corresponding quantization method based on the quantization calculation result; it can determine the memory saving of each quantization method; it can determine the target quantization method among multiple quantization methods based on the quantization error and memory saving of each quantization method; and it can determine the target quantization parameters of the target layer based on the target quantization method. In the above method, the target quantization parameters of each layer of the model can be determined so that the corresponding key tensors and value tensors can be quantized according to the target quantization parameters of each layer during the inference process through the model. Through the above method, the memory space occupied by key tensors and value tensors can be reduced to achieve the purpose of saving memory space. In addition, the above method also considers the quantization error of each layer of the model, achieving the goal of reducing the memory space occupied by key tensors and value tensors while ensuring the accuracy of the model.

[0236] Based on any of the above embodiments, the following is combined with Figure 4 The method for determining the various bond quantization parameters corresponding to different bond quantization methods is explained.

[0237] Figure 4 This is a flowchart illustrating another parameter determination method provided for an exemplary embodiment of this application. The method can be executed by an electronic device or a processor located within the electronic device. The following description uses an electronic device as the executing entity. Please refer to... Figure 4 The method may include:

[0238] S401. Determine the quantization type for each key quantization method.

[0239] The quantization type includes at least integer quantization and / or floating-point quantization.

[0240] In this embodiment, the quantization type of the key quantization method can be determined based on the quantized data format. Specifically, if the quantized data format is integer, the quantization type of the key quantization method can be integer quantization; if the quantized data format is floating-point, the quantization type of the key quantization method can be floating-point quantization. That is, if the data format of the quantized key tensor is integer, the quantization type of the key quantization method can be integer quantization; if the data format of the quantized key tensor is floating-point, the quantization type of the key quantization method can be floating-point quantization. For example, FP8 format can be floating-point format, while INT8 and INT4 formats can be integer formats.

[0241] If the quantization type is integer quantization, execute S402;

[0242] If the quantization type is floating-point quantization, execute S405.

[0243] S402, Determine the first space occupancy corresponding to the key quantization method.

[0244] It should be noted that if the quantization type of the key quantization method is integer quantization, then the key quantization parameters corresponding to the key quantization method include the key scaling factor and the key offset. In this embodiment, the methods for determining the key scaling factor and the key offset are described in steps S402-S404.

[0245] The first space occupancy corresponding to the key quantization method can be the first space occupancy of the quantized key tensor corresponding to the key quantization method. In other words, after the key tensor is quantized using the key quantization method, the size of the quantized key tensor is equal to the first space occupancy.

[0246] For any key quantization method, the first space occupancy corresponding to the key quantization method can be determined based on the data format of the quantized key tensor corresponding to the key quantization method.

[0247] For example, if the data format of the quantized key tensor corresponding to the key quantization method is FP8, then the first space occupancy of the key quantization method is 8 bits; if the data format of the quantized key tensor corresponding to the key quantization method is INT8, then the first space occupancy of the key quantization method is 8 bits; if the data format of the quantized key tensor corresponding to the key quantization method is INT4, then the first space occupancy of the key quantization method is 4 bits.

[0248] S403. Determine the key scaling factor corresponding to the key quantization method based on the key tensor and the first space occupancy.

[0249] In this embodiment, when determining the key scaling factor corresponding to the key quantization method, the first difference between the largest and smallest elements in the key tensor can be determined; the second difference between the function value of the preset exponential function and the first preset value can be determined, wherein the base of the preset exponential function is the second preset value and the exponent is the first space occupancy; and the ratio of the second difference to the first difference can be determined as the key scaling factor corresponding to the key quantization method.

[0250] It should be noted that the unit of elements in a key tensor can be the head. That is, a key tensor can include at least one element with the head as the basic unit.

[0251] The base of the preset exponential function can be set according to actual needs. Optionally, the base of the preset exponential function can be 2.

[0252] The first preset value can be set according to actual needs. Optionally, the first preset value can be 1.

[0253] If the base of the preset exponential function is 2, the first preset value is 1. For any key quantization method with integer quantization type, the key scaling factor corresponding to the key quantization method can be determined by the following formula:

[0254]

[0255] Where x can be the first space occupancy, and k max k can be the largest element in the key tensor. min It can be the smallest element in the key tensor.

[0256] S404. Determine the key offset corresponding to the key quantization method based on the key tensor.

[0257] In this embodiment, when determining the key offset corresponding to the key quantization method, the smallest element in the key tensor can be determined and the smallest element can be determined as the key offset.

[0258] Specifically, for any key quantization method with integer quantization type, the key offset corresponding to the key quantization method can be determined by the following formula:

[0259] Key offset = k min

[0260] Where, k min It can be the smallest element in the key tensor.

[0261] It should be noted that S403 can be executed before S404; or, S403 can be executed simultaneously with S404; or, S403 can be executed after S404. This embodiment does not limit this.

[0262] S405. Determine the range of the first exponent bit values ​​of the current data format of the key tensor.

[0263] It should be noted that if the quantization type of the key quantization method is floating-point quantization, then the key quantization parameters corresponding to the key quantization method include the key scaling hyperparameters corresponding to the key quantization method. In this embodiment, the method for determining the key scaling hyperparameters is described through S405-S407.

[0264] The range of the first exponent bit is the range of exponent bits corresponding to the current data format of the key tensor.

[0265] The current data format of the key tensor can be FP16 or FP32, etc.

[0266] It should be understood that the range of values ​​represented by the exponent in FP16 format can be 2^36.-14 ~2 15 The range of values ​​represented by the exponent in FP32 format can be 2^365. -126 ~2 127 .

[0267] In this embodiment, the range of values ​​represented by the exponent bits of the current data format of the key tensor can be represented by a first specific exponent form; and the range of exponent values ​​in the first specific exponent form can be determined as the range of values ​​of the first exponent bits; wherein, the base of the first specific exponent form is 2.

[0268] Specifically, if the current data format of the key tensor is FP16, then the value range represented by the exponent of the current data format of the key tensor can be 2. -14 ~2 15 The first exponent can range from -14 to 15; if the current data format of the key tensor is FP32, then the exponent in the current data format of the key tensor can represent a value ranging from 2. -126 ~2 127 The first exponent can be in the range of -126 to 127.

[0269] S406. Determine the range of the second exponent bit values ​​for the data format corresponding to the key quantization method.

[0270] The data format corresponding to the key quantization method can be the data format of the quantized key tensor corresponding to the key quantization method.

[0271] The range of the second exponent can be the range of exponents corresponding to the exponents of the data format corresponding to the key quantization method. Specifically, the data format corresponding to the key quantization method can be the data format of the quantized key tensor. In other words, the range of the second exponent can be the range of exponents corresponding to the exponents of the data format of the quantized key tensor.

[0272] In this embodiment, if the quantization type of the key quantization method is floating-point quantization, the data format of the quantized key tensor can be FP8-E5M2 format, FP8-E4M3 format, or FP8-E3M4 format.

[0273] It should be understood that the range of values ​​represented by the exponent in the FP8-E5M2 format can be 2^36. -14 ~2 15 The range of values ​​represented by the exponent in FP8-E4M3 format can be 2... -6 ~2 8 The range of values ​​represented by the exponent in FP8-E3M4 format can be 2... -3 ~2 3 .

[0274] In this embodiment, the range of values ​​represented by the exponent bits of the data format of the quantized key tensor can be represented by a second specific exponent form; and the range of exponent values ​​in the second specific exponent form can be determined as the range of values ​​of the second exponent bits; wherein, the base of the second specific exponent form is 2.

[0275] Specifically, if the data format of the quantization bond tensor is FP8-E5M2, then the range of values ​​represented by the exponent bits of the quantization bond tensor data format can be 2^35. -14 ~2 15 The second exponent can range from -14 to 15; if the quantization key tensor is in FP8-E4M3 format, then the exponent of the quantization key tensor can represent a value ranging from 2. -6 ~2 8 The second exponent can range from -6 to 8; if the quantization key tensor is in FP8-E3M4 format, then the exponent of the quantization key tensor can represent values ​​ranging from 2. -3 ~2 3 The second exponent can be in the range of -3 to 3.

[0276] S407. Determine the key scaling hyperparameter corresponding to the key quantization method based on the range of values ​​for the first and second exponent bits.

[0277] In this embodiment, when determining the key scaling hyperparameters corresponding to the key quantization method, the first minimum and first maximum values ​​of the first exponent range can be determined, and the second minimum and second maximum values ​​of the second exponent range can be determined; the third difference between the first minimum and the second minimum values ​​can be determined, and the fourth difference between the first maximum and the second maximum values ​​can be determined; based on the third and fourth differences, multiple candidate key scaling hyperparameters can be determined, where the candidate key scaling hyperparameters are the i-th power of the third preset value, the minimum value of i is the third difference, and the maximum value of i is the fourth difference; based on the key tensor and the multiple candidate key scaling hyperparameters, the key scaling hyperparameters are determined.

[0278] The third preset value can be set according to actual needs. Optionally, the third preset value can be 2.

[0279] Specifically, 2 i Multiple candidate key scaling hyperparameters were identified. Among them, the minimum value of i is the third difference, and the maximum value of i is the fourth difference.

[0280] For example, assuming the first minimum value of the first exponent's numerical range is -14 and the first maximum value is 15, and the second minimum value of the second exponent's numerical range is -7 and the second maximum value is 7, then the minimum value of i is -7 and the maximum value of i is 8. That is, the scaling hyperparameters for multiple candidate keys can be 2. -7 2-6 2 -5 2 -4 2 -3 2 -2 2 -1 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 and 2 8 .

[0281] In this embodiment, when determining the key scaling hyperparameter based on the key tensor and multiple candidate key scaling hyperparameters, the key tensor can be quantized using each candidate key scaling hyperparameter to obtain multiple candidate key tensors; the error between each candidate key tensor and the key tensor can be calculated to obtain multiple first errors; and the candidate key scaling hyperparameter corresponding to the minimum value among the multiple first errors can be determined as the key scaling hyperparameter.

[0282] Specifically, the key tensor can be quantized by each candidate key scaling hyperparameter to obtain multiple candidate key tensors; and the candidate key scaling hyperparameter corresponding to the candidate key tensor with the smallest error with the key tensor can be determined as the key scaling hyperparameter.

[0283] The parameter determination method provided in this embodiment can determine the quantization type of each key quantization method. If the quantization type is integer quantization, the first space occupancy corresponding to the key quantization method can be determined. The key scaling factor corresponding to the key quantization method can be determined based on the key tensor and the first space occupancy, and the key offset corresponding to the key quantization method can be determined based on the key tensor. If the quantization type is floating-point quantization, the first exponent value range of the current data format of the key tensor can be determined, the second exponent value range of the data format corresponding to the key quantization method can be determined, and the key scaling hyperparameter corresponding to the key quantization method can be determined based on the first exponent value range and the second exponent value range. Through the above method, the key quantization parameters corresponding to each key quantization method can be determined, achieving the purpose of determining the key quantization parameters.

[0284] Based on any of the above embodiments, the following is combined with Figure 5 This paper explains the method for determining the quantization parameters corresponding to various quantization methods.

[0285] Figure 5 This is a flowchart illustrating another parameter determination method provided for an exemplary embodiment of this application. The method can be executed by an electronic device or a processor located within the electronic device. The following description uses an electronic device as the executing entity. Please refer to... Figure 5 The method may include:

[0286] S501. Determine the quantization type for each value quantization method.

[0287] The quantization type includes at least integer quantization and / or floating-point quantization.

[0288] In this embodiment, the quantization type of the value quantization method can be determined based on the quantized data format. Specifically, if the quantized data format is integer, the quantization type of the value quantization method can be integer quantization; if the quantized data format is floating-point, the quantization type of the value quantization method can be floating-point quantization. That is, if the data format of the quantized value tensor is integer, the quantization type of the value quantization method can be integer quantization; if the data format of the quantized value tensor is floating-point, the quantization type of the value quantization method can be floating-point quantization. For example, FP8 format can be floating-point format, while INT8 and INT4 formats can be integer formats.

[0289] If the quantization type is integer quantization, execute S502;

[0290] If the quantization type is floating-point quantization, execute S505.

[0291] S502, the second space occupancy corresponding to the quantization method of the determined value.

[0292] It should be noted that if the quantization type of the value quantization method is integer quantization, then the value quantization parameters corresponding to the value quantization method include the value scaling factor and the value offset. In this embodiment, the methods for determining the value scaling factor and the methods for determining the value offset are explained through S502-S504.

[0293] The second space occupancy corresponding to the value quantization method can be the second space occupancy of the quantized value tensor. In other words, the size of the quantized value tensor after value quantization is equal to the second space occupancy.

[0294] For any value quantization method, the second space occupancy corresponding to the value quantization method can be determined based on the data format of the quantized value tensor corresponding to the value quantization method.

[0295] For example, if the data format of the quantized value tensor corresponding to the value quantization method is FP8, then the second space occupied by the value quantization method is 8 bits; if the data format of the quantized value tensor corresponding to the value quantization method is INT8, then the second space occupied by the value quantization method is 8 bits; if the data format of the quantized value tensor corresponding to the value quantization method is INT4, then the second space occupied by the value quantization method is 4 bits.

[0296] S503. Determine the value scaling factor corresponding to the value quantization method based on the value tensor and the second space occupancy.

[0297] In this embodiment, when determining the value scaling factor corresponding to the value quantization method, the fifth difference between the largest and smallest elements in the value tensor can be determined; the sixth difference between the function value of the preset exponential function and the first preset value can be determined, where the base of the preset exponential function is the second preset value and the exponent is the second space occupancy; the ratio of the sixth difference to the fifth difference is determined as the value scaling factor corresponding to the value quantization method.

[0298] It should be noted that the unit of elements in a value tensor can be the head. That is, a value tensor can include at least one element with the head as the basic unit.

[0299] The base of the preset exponential function can be set according to actual needs. Optionally, the base of the preset exponential function can be 2.

[0300] The first preset value can be set according to actual needs. Optionally, the first preset value can be 1.

[0301] If the base of the preset exponential function is 2, the first preset value is 1. For any value quantization method with integer quantization as the quantization type, the value scaling factor corresponding to the value quantization method can be determined by the following formula:

[0302]

[0303] Where y can be the amount of space occupied in the second space, and v max v can be the largest element in the value tensor. min It can be the smallest element in a value tensor.

[0304] S504. Determine the value offset corresponding to the value quantization method based on the value tensor.

[0305] In this embodiment, when determining the value offset corresponding to the value quantization method, the minimum element in the value tensor can be determined and the minimum element can be determined as the value offset.

[0306] Specifically, for any integer quantization method, the value offset corresponding to the value quantization method can be determined using the following formula:

[0307] Value offset = v min

[0308] Among them, v min It can be the smallest element in a value tensor.

[0309] It should be noted that S503 can be executed before S504; or S503 can be executed simultaneously with S504; or S503 can be executed after S504. This embodiment does not limit this.

[0310] S505. Determine the range of the third exponent value of the current data format of the value tensor.

[0311] It should be noted that if the quantization type of the value quantization method is floating-point quantization, then the value quantization parameters corresponding to the value quantization method include the value scaling hyperparameters corresponding to the value quantization method. In this embodiment, the method for determining the value scaling hyperparameters is described through S505-S507.

[0312] The range of the third exponent is the range of exponents corresponding to the exponents in the current data format of the value tensor.

[0313] The current data format of the value tensor can be FP16 or FP32, etc.

[0314] It should be understood that the range of values ​​represented by the exponent in FP16 format can be 2^36. -14 ~2 15 The range of values ​​represented by the exponent in FP32 format can be 2^365. -126 ~2 127 .

[0315] In this embodiment, the range of values ​​represented by the exponent bits of the current data format of the value tensor can be represented by a third specific exponent form; and the range of exponent values ​​in the third specific exponent form can be determined as the range of values ​​of the third exponent bits; wherein, the base of the third specific exponent form is 2.

[0316] Specifically, if the current data format of the value tensor is FP16, then the range of values ​​represented by the exponent in the current data format of the value tensor can be 2. -14 ~2 15 The third exponent can range from -14 to 15; if the current data format of the value tensor is FP32, then the exponent in the current data format of the value tensor can represent a value range of 2. -126 ~2 127 The value range of the third exponent can be -126 to 127.

[0317] S506, The range of the fourth exponent bit value of the data format corresponding to the quantization method of the determined value.

[0318] The data format corresponding to the value quantization method can be the data format of the quantized value tensor corresponding to the value quantization method.

[0319] The range of the fourth exponent bit can be the exponent range corresponding to the exponent bits of the data format corresponding to the value quantization method. Specifically, the data format corresponding to the value quantization method can be the data format of the quantized value tensor corresponding to the value quantization method. In other words, the range of the fourth exponent bit can be the exponent range corresponding to the exponent bits of the data format of the quantized value tensor.

[0320] In this embodiment, if the quantization type of the value quantization method is floating-point quantization, the data format of the quantized value tensor can be FP8-E5M2 format, FP8-E4M3 format, or FP8-E3M4 format.

[0321] It should be understood that the range of values ​​represented by the exponent in the FP8-E5M2 format can be 2^36. -14 ~2 15 The range of values ​​represented by the exponent in FP8-E4M3 format can be 2... -7 ~2 7 The range of values ​​represented by the exponent in FP8-E3M4 format can be 2... -3 ~2 3 .

[0322] In this embodiment, the range of values ​​represented by the exponent bits of the data format of the quantized tensor can be represented by the fourth specific exponent form; and the range of exponent values ​​in the fourth specific exponent form can be determined as the range of values ​​of the fourth exponent bits; wherein, the base of the fourth specific exponent form is 2.

[0323] Specifically, if the data format of the quantized tensor is FP8-E5M2, then the range of values ​​represented by the exponent bits of the quantized tensor data format can be 2^35. -14 ~2 15 The fourth exponent can range from -14 to 15; if the quantized tensor's data format is FP8-E4M3, then the exponent of the quantized tensor can represent a value range of 2. -6 ~2 8 The fourth exponent can range from -6 to 8; if the quantized tensor's data format is FP8-E3M4, then the exponent of the quantized tensor can represent a value range of 2. -3 ~2 3 The fourth exponent can range from -3 to 3.

[0324] S507. Determine the value scaling hyperparameter corresponding to the value quantization method based on the value range of the third exponent and the value range of the fourth exponent.

[0325] In this embodiment, when determining the value scaling hyperparameter corresponding to the value quantization method, the third minimum and third maximum values ​​of the third exponent range can be determined, and the fourth minimum and fourth maximum values ​​of the fourth exponent range can be determined; the seventh difference between the third minimum and the fourth minimum values ​​can be determined, and the eighth difference between the third maximum and the fourth maximum values ​​can be determined; based on the seventh and eighth differences, multiple candidate value scaling hyperparameters can be determined, where the candidate value scaling hyperparameter is the j-th power of the third preset value, the minimum value of j is the seventh difference, and the maximum value of j is the eighth difference; based on the value tensor and the multiple candidate value scaling hyperparameters, the value scaling hyperparameter is determined.

[0326] The third preset value can be set according to actual needs. Optionally, the third preset value can be 2.

[0327] Specifically, 2 j Multiple scaling hyperparameters were identified as candidate values. Among them, the minimum value of j is the seventh difference, and the maximum value of j is the eighth difference.

[0328] For example, suppose the third minimum value of the third exponent's numerical range is -14 and the third maximum value is 15, and the fourth minimum value of the fourth exponent's numerical range is -7 and the fourth maximum value is 7. Then the minimum value of j is -7 and the maximum value of j is 8. That is, the scaling hyperparameter for multiple candidate values ​​can be 2. -7 2 -6 2 -5 2 -4 2 -3 2 -2 2 -1 2 0 2 1 2 2 2 3 2 4 2 5 2 6 2 7 and 2 8 .

[0329] In this embodiment, when determining the value scaling hyperparameter based on the value tensor and multiple candidate value scaling hyperparameters, the value tensor can be quantized using each candidate value scaling hyperparameter to obtain multiple candidate value tensors; the error between each candidate value tensor and the value tensor can be calculated to obtain at least one second error; and the candidate value scaling hyperparameter corresponding to the minimum value among the at least one second error can be determined as the value scaling hyperparameter.

[0330] Specifically, the value tensor can be quantized by each candidate value scaling hyperparameter to obtain multiple candidate value tensors; and the candidate value scaling hyperparameter corresponding to the candidate value tensor with the smallest error with the value tensor can be determined as the value scaling hyperparameter.

[0331] The parameter determination method provided in this embodiment can determine the quantization type of each value quantization method. If the quantization type is integer quantization, the second space occupancy corresponding to the value quantization method can be determined. The value scaling factor corresponding to the value quantization method can be determined based on the value tensor and the second space occupancy, and the value offset corresponding to the value quantization method can also be determined based on the value tensor. If the quantization type is floating-point quantization, the range of the third exponent bits of the current data format of the value tensor can be determined. The range of the fourth exponent bits of the data format corresponding to the value quantization method can also be determined based on the range of the third and fourth exponent bits. Through the above method, the value quantization parameters corresponding to each value quantization method can be determined, achieving the purpose of determining the value quantization parameters.

[0332] The parameter determination method provided in this application can be applied to various models. For any model, the target quantization method corresponding to each layer of the model can be determined according to the parameter determination method provided in this application, so as to achieve the quantization of the entire model.

[0333] Figure 6 This is a schematic diagram of a parameter determination device provided for an exemplary embodiment of this application. The parameter determination device 10 includes a processing module 11 and a determination module 12, wherein...

[0334] The processing module 11 is used to perform calculations on the input data through the target layer in the model to obtain the calculation results;

[0335] The determining module 12 is used to determine the calculation result under multiple quantization methods, and to determine the quantization error of the corresponding quantization method based on the quantization result.

[0336] The determining module 12 is further configured to determine multiple memory savings corresponding to the multiple quantization methods, and determine the target quantization method among the multiple quantization methods based on the quantization error and memory savings of each quantization method.

[0337] The determining module 12 is further configured to determine the target quantization parameters of the target layer according to the target quantization method.

[0338] The parameter determination device provided in this application embodiment can execute the method shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0339] In one possible implementation, the determining module 12 is specifically used for:

[0340] Determine the various quantization methods and the corresponding quantization parameters for each quantization method;

[0341] The calculation result is quantized using the quantization parameters corresponding to each quantization method to obtain the quantization calculation result corresponding to each quantization method.

[0342] The multiple quantization results include: the quantization results corresponding to each quantization method.

[0343] In one possible implementation, the calculation result includes a key tensor and a value tensor; the determining module 12 is specifically used for:

[0344] Determine the multiple key quantization methods corresponding to the key tensor and the multiple value quantization methods corresponding to the value tensor;

[0345] The multiple key quantization methods and the multiple value quantization methods are combined to obtain the multiple quantization methods, which include one key quantization method and one value quantization method.

[0346] In one possible implementation, the determining module 12 is specifically used for:

[0347] Based on the key tensor, determine the various key quantization parameters corresponding to the various key quantization methods;

[0348] Based on the value tensor, determine the various value quantization parameters corresponding to the various value quantization methods;

[0349] For any quantization method, the key quantization parameters corresponding to the key quantization method included in the quantization method and the value quantization parameters corresponding to the value quantization method included in the quantization method are determined as the quantization parameters corresponding to the quantization method.

[0350] In one possible implementation, the determining module 12 is specifically used for:

[0351] Determine the quantization type for each of the key quantization methods, wherein the quantization type includes at least integer quantization and / or floating-point quantization;

[0352] Based on the key tensor and the quantization type of each key quantization method, determine the key quantization parameters corresponding to each key quantization method;

[0353] The various key quantization parameters include key quantization parameters corresponding to each key quantization method.

[0354] In one possible implementation, the determining module 12 is specifically used for:

[0355] If the quantization type is integer quantization, determine the first space occupancy corresponding to the key quantization method. After quantizing the key tensor using the key quantization method, the size of the quantized key tensor is equal to the first space occupancy.

[0356] The key scaling factor corresponding to the key quantization method is determined based on the key tensor and the first space occupancy, and the key offset corresponding to the key quantization method is determined based on the key tensor.

[0357] The key quantization parameters corresponding to the key quantization method include: the key scaling factor corresponding to the key quantization method and the key offset corresponding to the key quantization method.

[0358] In one possible implementation, the determining module 12 is specifically used for:

[0359] Determine the first difference between the maximum and minimum elements in the key tensor;

[0360] Determine a second difference between the function value of a preset exponential function and a first preset value, wherein the base of the preset exponential function is the second preset value and the exponent is the first space occupancy;

[0361] The ratio of the second difference to the first difference is determined as the key scaling factor corresponding to the key quantization method.

[0362] In one possible implementation, the determining module 12 is specifically used for:

[0363] Determine the minimum element in the key tensor and set the minimum element as the key offset.

[0364] In one possible implementation, the determining module 12 is specifically used for:

[0365] If the quantization type is floating-point quantization, determine the first exponent bit value range of the current data format of the key tensor, where the first exponent bit value range is the exponent range corresponding to the exponent bit of the current data format;

[0366] Determine the range of the second exponent bit values ​​for the data format corresponding to the key quantization method, wherein the range of the second exponent bit values ​​is the range of the exponent bits corresponding to the exponent bits of the data format corresponding to the key quantization method;

[0367] Based on the first exponent bit value range and the second exponent bit value range, determine the key scaling hyperparameter corresponding to the key quantization method;

[0368] The key quantization parameters corresponding to the key quantization method include: the key scaling hyperparameter corresponding to the key quantization method.

[0369] In one possible implementation, the determining module 12 is specifically used for:

[0370] Determine the first minimum and first maximum values ​​of the first exponent value range, and determine the second minimum and second maximum values ​​of the second exponent value range;

[0371] Determine the third difference between the first minimum value and the second minimum value, and determine the fourth difference between the first maximum value and the second maximum value;

[0372] Based on the third difference and the fourth difference, a plurality of candidate key scaling hyperparameters are determined. The candidate key scaling hyperparameters are the third preset value raised to the power of i, where the minimum value of i is the third difference and the maximum value of i is the fourth difference.

[0373] The key scaling hyperparameters are determined based on the key tensor and the plurality of candidate key scaling hyperparameters.

[0374] In one possible implementation, the determining module 12 is specifically used for:

[0375] The key tensor is quantized by scaling hyperparameters for each candidate key to obtain multiple candidate key tensors;

[0376] Calculate the error between each candidate key tensor and the key tensor to obtain multiple first errors;

[0377] The candidate key scaling hyperparameter corresponding to the minimum value among the plurality of first errors is determined as the key scaling hyperparameter.

[0378] In one possible implementation, the determining module 12 is specifically used for:

[0379] Attention is calculated on the results of the calculation to obtain a standard value;

[0380] Attention is calculated for each quantization result to obtain the candidate value corresponding to each quantization result;

[0381] Calculate the error between the candidate value and the standard value corresponding to each quantization operation result, and determine the error between the candidate value and the standard value corresponding to the quantization operation result as the quantization error of the quantization method corresponding to the quantization operation result.

[0382] In one possible implementation, for any quantization method, the determining module 12 is specifically used for:

[0383] The quantization method includes the candidate key quantization method and the candidate value quantization method;

[0384] Determine the key memory saving amount corresponding to the candidate key quantization method, and determine the value memory saving amount corresponding to the candidate value quantization method;

[0385] The memory saving amount of the quantization method is determined based on the key memory saving amount and the value memory saving amount.

[0386] In one possible implementation, the determining module 12 is specifically used for:

[0387] Determine the first weight corresponding to the quantization error and the second weight corresponding to the memory saving amount;

[0388] Based on the quantization error and memory savings of each quantization method, the first weight, and the second weight, a weighting value corresponding to each quantization method is determined, resulting in multiple weighting values.

[0389] Based on the multiple weighted values, the target quantization method is determined among the multiple quantization methods.

[0390] In one possible implementation, for any quantization method, the determining module 12 is specifically used for:

[0391] Determine the first product of the quantization error of the quantization method and the first weight, and determine the second product of the memory saving amount of the quantization method and the second weight;

[0392] The sum of the first product and the second product is determined as the weighted value corresponding to the quantization method.

[0393] In one possible implementation, the determining module 12 is specifically used for:

[0394] Determine the minimum value among the multiple weighted values, and determine the quantization method corresponding to the minimum value as the target quantization method.

[0395] The parameter determination device provided in this application embodiment can execute the method shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0396] Figure 7 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this application. Please refer to... Figure 7 The electronic device 20 may include a processor 21 and a memory 22. Exemplarily, the processor 21 and the memory 22 are interconnected via a bus 23.

[0397] The memory 22 stores computer-executed instructions;

[0398] The processor 21 executes the computer execution instructions stored in the memory 22, causing the processor 21 to perform the method as shown in the above method embodiment.

[0399] Figure 7The electronic device shown can be used to perform the methods shown in any of the above embodiments.

[0400] This application embodiment may also provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method shown in the above method embodiment.

[0401] This application embodiment may also provide a computer program product, including a computer program, which, when executed by a processor, can implement the methods shown in the above method embodiments.

[0402] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0403] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0404] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0405] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0406] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0407] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0408] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0409] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0410] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for determining parameters, characterized in that, include: The target layer in the model processes the input data to obtain the calculation result; Determine the quantization results of the operation under multiple quantization methods, and determine the quantization error of the corresponding quantization method based on the quantization results; Determine multiple memory savings corresponding to the various quantization methods, and determine the target quantization method among the various quantization methods based on the quantization error and memory savings of each quantization method; The target quantization parameters of the target layer are determined according to the target quantization method.

2. The method according to claim 1, characterized in that, Determine the quantization results of the operation under multiple quantization methods, including: Determine the various quantization methods and the corresponding quantization parameters for each quantization method; The calculation result is quantized using the quantization parameters corresponding to each quantization method to obtain the quantization calculation result corresponding to each quantization method. The multiple quantization results include: the quantization results corresponding to each quantization method.

3. The method according to claim 2, characterized in that, The operation result includes key tensors and value tensors; determining the multiple quantization methods includes: Determine the multiple key quantization methods corresponding to the key tensor and the multiple value quantization methods corresponding to the value tensor; The multiple key quantization methods and the multiple value quantization methods are combined to obtain the multiple quantization methods, which include one key quantization method and one value quantization method.

4. The method according to claim 3, characterized in that, Determine the quantization parameters for each quantization method, including: Based on the key tensor, determine the various key quantization parameters corresponding to the various key quantization methods; Based on the value tensor, determine the various value quantization parameters corresponding to the various value quantization methods; For any quantization method, the key quantization parameters corresponding to the key quantization method included in the quantization method and the value quantization parameters corresponding to the value quantization method included in the quantization method are determined as the quantization parameters corresponding to the quantization method.

5. The method according to claim 4, characterized in that, Based on the key tensor, determine various key quantization parameters corresponding to the various key quantization methods, including: Determine the quantization type for each of the key quantization methods, wherein the quantization type includes at least integer quantization and / or floating-point quantization; Based on the key tensor and the quantization type of each key quantization method, determine the key quantization parameters corresponding to each key quantization method; The various key quantization parameters include key quantization parameters corresponding to each key quantization method.

6. The method according to claim 5, characterized in that, For any key quantization method, based on the key tensor and the quantization type of the key quantization method, determine the key quantization parameters corresponding to the key quantization method, including: If the quantization type is integer quantization, determine the first space occupancy corresponding to the key quantization method. After quantizing the key tensor using the key quantization method, the size of the quantized key tensor is equal to the first space occupancy. The key scaling factor corresponding to the key quantization method is determined based on the key tensor and the first space occupancy, and the key offset corresponding to the key quantization method is determined based on the key tensor. The key quantization parameters corresponding to the key quantization method include: the key scaling factor corresponding to the key quantization method and the key offset corresponding to the key quantization method.

7. The method according to claim 6, characterized in that, Determining the key scaling factor corresponding to the key quantization method based on the key tensor and the first space occupancy includes: Determine the first difference between the maximum and minimum elements in the key tensor; Determine a second difference between the function value of a preset exponential function and a first preset value, wherein the base of the preset exponential function is the second preset value and the exponent is the first space occupancy; The ratio of the second difference to the first difference is determined as the key scaling factor corresponding to the key quantization method.

8. The method according to claim 6 or 7, characterized in that, Determining the key offset corresponding to the key quantization method based on the key tensor includes: Determine the minimum element in the key tensor and set the minimum element as the key offset.

9. The method according to claim 5, characterized in that, For any key quantization method, based on the key tensor and the quantization type of the key quantization method, determine the key quantization parameters corresponding to the key quantization method, including: If the quantization type is floating-point quantization, determine the first exponent bit value range of the current data format of the key tensor, where the first exponent bit value range is the exponent range corresponding to the exponent bit of the current data format; Determine the range of the second exponent bit values ​​for the data format corresponding to the key quantization method, wherein the range of the second exponent bit values ​​is the range of the exponent bits corresponding to the exponent bits of the data format corresponding to the key quantization method; Based on the first exponent bit value range and the second exponent bit value range, determine the key scaling hyperparameter corresponding to the key quantization method; The key quantization parameters corresponding to the key quantization method include: the key scaling hyperparameter corresponding to the key quantization method.

10. The method according to claim 9, characterized in that, Determining the key scaling hyperparameters corresponding to the key quantization method based on the first exponent value range and the second exponent value range includes: Determine the first minimum and first maximum values ​​of the first exponent value range, and determine the second minimum and second maximum values ​​of the second exponent value range; Determine the third difference between the first minimum value and the second minimum value, and determine the fourth difference between the first maximum value and the second maximum value; Based on the third difference and the fourth difference, a plurality of candidate key scaling hyperparameters are determined. The candidate key scaling hyperparameters are the third preset value raised to the power of i, where the minimum value of i is the third difference and the maximum value of i is the fourth difference. The key scaling hyperparameters are determined based on the key tensor and the plurality of candidate key scaling hyperparameters.

11. The method according to claim 10, characterized in that, Based on the key tensor and the plurality of candidate key scaling hyperparameters, the key scaling hyperparameters are determined, including: The key tensor is quantized by scaling hyperparameters for each candidate key to obtain multiple candidate key tensors; Calculate the error between each candidate key tensor and the key tensor to obtain multiple first errors; The candidate key scaling hyperparameter corresponding to the minimum value among the plurality of first errors is determined as the key scaling hyperparameter.

12. The method according to any one of claims 1-11, characterized in that, The quantization error of the corresponding quantization method is determined based on the quantization calculation result, including: Attention is calculated on the results of the calculation to obtain a standard value; Attention is calculated for each quantization result to obtain the candidate value corresponding to each quantization result; Calculate the error between the candidate value and the standard value corresponding to each quantization operation result, and determine the error between the candidate value and the standard value corresponding to the quantization operation result as the quantization error of the quantization method corresponding to the quantization operation result.

13. The method according to any one of claims 1-12, characterized in that, For any given quantization method, determine the memory savings achieved by that quantization method, including: The quantization method includes the candidate key quantization method and the candidate value quantization method; Determine the key memory saving amount corresponding to the candidate key quantization method, and determine the value memory saving amount corresponding to the candidate value quantization method; The memory saving amount of the quantization method is determined based on the key memory saving amount and the value memory saving amount.

14. The method according to any one of claims 1-13, characterized in that, Based on the quantization error and memory savings of each quantization method, a target quantization method is determined among the multiple quantization methods, including: Determine the first weight corresponding to the quantization error and the second weight corresponding to the memory saving amount; Based on the quantization error and memory savings of each quantization method, the first weight, and the second weight, a weighting value corresponding to each quantization method is determined, resulting in multiple weighting values. Based on the multiple weighted values, the target quantization method is determined among the multiple quantization methods.

15. The method according to claim 14, characterized in that, For any quantization method, based on the quantization error and memory savings of the quantization method, the first weight, and the second weight, a weighting value corresponding to the quantization method is determined, including: Determine the first product of the quantization error of the quantization method and the first weight, and determine the second product of the memory saving amount of the quantization method and the second weight; The sum of the first product and the second product is determined as the weighted value corresponding to the quantization method.

16. The method according to claim 14 or 15, characterized in that, Based on the multiple weighted values, the target quantization method is determined from among the multiple quantization methods, including: Determine the minimum value among the multiple weighted values, and determine the quantization method corresponding to the minimum value as the target quantization method.

17. A parameter determining device, characterized in that, The device includes a processing module and a determining module, wherein... The processing module is used to perform calculations on the input data through the target layer in the model to obtain the calculation results; The determining module is used to determine the calculation result under multiple quantization methods, and to determine the quantization error of the corresponding quantization method based on the quantization result. The determining module is further configured to determine multiple memory savings corresponding to the multiple quantization methods, and determine the target quantization method among the multiple quantization methods based on the quantization error and memory savings of each quantization method. The determining module is further configured to determine the target quantization parameters of the target layer according to the target quantization method.

18. An electronic device, characterized in that, include: Memory and processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to cause the electronic device to perform the method according to any one of claims 1-16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-16.

20. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-16.

Images