Attention mechanism operator optimization method and device, storage medium and program product

By quantizing the attention mechanism operator in the large language model, the memory usage is reduced, the problem of the attention mechanism operator's memory usage is solved, and the reasoning efficiency and performance are improved.

CN118982052APending Publication Date: 2024-11-19SHANGHAI BIREN TECH CO LTD
View PDF 0 Cites 3 Cited by

Patent Information

Application Number
CN202411053482.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The video memory usage of the Attention operator in the large language model is too large, resulting in low inference efficiency and high deployment cost. Although the existing key-value caching technology improves the inference speed, it increases the additional video memory usage.

Method used

By mapping the input tensor into a first key representation, a first value representation and a first query representation, and quantizing it using a quantized scaling factor, saving the quantized key and value representation in the video memory, and executing based on the quantized representation Self-attention mechanism calculation to reduce video memory usage.

Benefits of technology

Video memory only needs to save the quantized key-value data, which reduces the additional video memory usage brought by key-value cache and improves the performance of the Attention operator.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982052A_ABST
    Figure CN118982052A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an attention mechanism operator optimization method and device, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: employing a first quantization scaling factor to carry out the online quantization of a first key representation in a model reasoning process, and obtaining a second key representation; and performing online quantization on the first value representation by using a second quantization scaling factor to obtain a second value representation. The second key representation and the first quantization scaling factor are then saved in a key information set in the video memory, and the second value representation and the second quantization scaling factor are saved in a value information set in the video memory. And performing self-attention mechanism calculation based on the first query representation and a key information set and a value information set in the video memory to obtain an output tensor. Thus, in the large language model reasoning process, the video memory only needs to store quantized key value data, extra video memory occupation caused by key value cache is greatly reduced, and video memory occupation of attention mechanism reasoning is also reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an optimization method, device, storage medium and program product of an attention mechanism operator. Background Art

[0002] At present, the technology of large language models is developing very rapidly, and the number of model parameters is getting larger and larger. Therefore, the video memory occupied during the inference process is also getting larger and larger, resulting in problems such as low inference efficiency and high deployment cost. Therefore, how to solve the problem of large video memory occupation of model parameters in large language models has become a very popular research direction.

[0003] In related technologies, the attention mechanism (Attention) operator in large language model reasoning uses an autoregressive method to generate characters round by round. When generating characters in each round, not only the query (Q) representation, key (K) representation, and value (V) representation calculated in this round are required, but also the key (K) representation and value (V) representation calculated in each previous round are required. In order to avoid repeatedly calculating the key (K) representation and value (V) representation calculated in each previous round when generating characters in each round, the related technology uses the key-value cache (KVCache) technology to save the key (K) representation and value (V) representation calculated in each round in the video memory. In this way, when generating characters in each round, the key (K) representation and value (V) representation calculated in each previous round can be directly read from the video memory.

[0004] Although the above key-value cache technology can improve the inference speed, it also leads to a large amount of additional video memory usage. Summary of the invention

[0005] The embodiments of the present application provide an optimization method, device, storage medium and program product for an attention mechanism operator, which are used to reduce the additional video memory occupancy caused by key-value caching, thereby reducing the video memory occupancy of Attention reasoning.

[0006] On the one hand, an embodiment of the present application provides an optimization method for an attention mechanism operator, including:

[0007] Mapping the input tensor into a first key representation, a first value representation, and a first query representation;

[0008] quantizing the first key representation using a first quantization scaling factor to obtain a second key representation; and quantizing the first value representation using a second quantization scaling factor to obtain a second value representation;

[0009] storing the second key representation and the first quantization scaling factor in a key information set in a video memory, and storing the second value representation and the second quantization scaling factor in a value information set in a video memory;

[0010] A self-attention mechanism calculation is performed based on the first query representation, the key information set, and the value information set to obtain an output tensor.

[0011] On the one hand, an embodiment of the present application provides an optimization device for an attention mechanism operator, including:

[0012] A mapping module, configured to map an input tensor into a first key representation, a first value representation, and a first query representation;

[0013] a quantization module, configured to quantize the first key representation using a first quantization scaling factor to obtain a second key representation; and quantize the first value representation using a second quantization scaling factor to obtain a second value representation;

[0014] A saving module, configured to save the second key representation and the first quantization scaling factor in a key information set in a video memory, and to save the second value representation and the second quantization scaling factor in a value information set in a video memory;

[0015] A computing module is used to perform self-attention mechanism calculation based on the first query representation, the key information set and the value information set to obtain an output tensor.

[0016] Optionally, the quantization module is specifically used for:

[0017] Performing linear mapping on the input tensor through the offline quantized key linear layer to obtain the first key representation;

[0018] Performing linear mapping on the input tensor through an offline quantized value linear layer to obtain the first value representation;

[0019] The input tensor is linearly mapped through the offline quantized query linear layer to obtain the first query representation.

[0020] Optionally, the parameter information of the key linear layer after offline quantization includes: a first weight after offline quantization and a corresponding third quantization scaling factor;

[0021] The quantization module is specifically used for:

[0022] A matrix multiplication calculation is performed on the first weight and the input tensor, and the obtained calculation result is multiplied by the third quantization scaling factor to obtain the first key representation.

[0023] Optionally, the parameter information of the offline quantized value linear layer includes: the offline quantized second weight and the corresponding fourth quantization scaling factor;

[0024] The quantization module is specifically used for:

[0025] A matrix multiplication calculation is performed on the second weight and the input tensor, and the obtained calculation result is multiplied by the fourth quantization scaling factor to obtain the first value representation.

[0026] Optionally, the parameter information of the query linear layer after offline quantization includes: a third weight after offline quantization and a corresponding fifth quantization scaling factor;

[0027] The quantization module is specifically used for:

[0028] A matrix multiplication calculation is performed on the third weight and the input tensor, and the obtained calculation result is multiplied by the fifth quantization scaling factor to obtain the first query representation.

[0029] Optionally, the calculation module is specifically used for:

[0030] Performing a matrix multiplication calculation on the first query representation and the key representation in the key information set to obtain a first calculation result;

[0031] Dequantizing the first calculation result by using the quantization scaling factor in the key information set to obtain a second calculation result;

[0032] performing a normalization operation on the second calculation result, and multiplying the obtained normalization operation result by the quantization scaling factor in the value information set to obtain a third calculation result;

[0033] Performing a matrix multiplication calculation on the third calculation result and the value representation in the value information set to obtain a fourth calculation result;

[0034] The fourth calculation result is linearly mapped through the output linear layer after offline quantization to obtain the output tensor.

[0035] Optionally, the calculation module is specifically used for:

[0036] After performing a scaling operation on the first calculation result, the obtained scaling operation result is multiplied by the quantization scaling factor in the key information set to obtain the second calculation result.

[0037] Optionally, the parameter information of the output linear layer after offline quantization includes: a fourth weight after offline quantization and a corresponding sixth quantization scaling factor;

[0038] The calculation module is specifically used for:

[0039] A matrix multiplication calculation is performed on the fourth weight and the fourth calculation result, and the obtained calculation result is multiplied by the sixth quantization scaling factor to obtain the output tensor.

[0040] On the one hand, an embodiment of the present application provides a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements the steps of the above-mentioned optimization method of the attention mechanism operator when executing the computer program.

[0041] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned attention mechanism operator optimization method.

[0042] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the optimization method of the above-mentioned attention mechanism operator.

[0043] In an embodiment of the present application, during the large language model inference process, the first key representation is online quantized using a first quantization scaling factor to obtain a second key representation; and the first value representation is online quantized using a second quantization scaling factor to obtain a second value representation. The second key representation and the first quantization scaling factor are then saved in a key information set in the video memory, and the second value representation and the second quantization scaling factor are saved in a value information set in the video memory. Based on the first query representation, the key information set and the value information set in the video memory, a self-attention mechanism calculation is performed to obtain an output tensor. In this way, during the large language model inference process, the video memory only needs to store the quantized key-value data, which greatly reduces the additional video memory occupancy caused by the key-value cache, and also reduces the video memory occupancy of the Attention inference, thereby improving the performance of the Attention operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0045] Figure 1 A schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of the present application;

[0046] Figure 2 Schematic diagram of a method for optimizing an attention mechanism operator provided in an embodiment of the present application Figure 1 ;

[0047] Figure 3 Schematic diagram of a method for optimizing an attention mechanism operator provided in an embodiment of the present application Figure 2 ;

[0048] Figure 4 A schematic diagram of the structure of an optimization device for an attention mechanism operator provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] refer to Figure 1 , which is a structural diagram of an artificial intelligence chip applicable to an embodiment of the present application, the artificial intelligence chip 100 at least includes: a video memory 101, a plurality of execution units 102, wherein each execution unit 102 includes: a tensor calculation unit 103 and a vector calculation unit 104.

[0052] The video memory 101 may be a high bandwidth memory (HBM for short) or other types of memory.

[0053] The tensor computing unit 103 and the vector computing unit 104 are heterogeneous cores. The tensor computing unit 103, as an accelerator in the artificial intelligence chip, usually performs convolution calculations or matrix multiplication calculations, that is, the tensor computing unit 103 is usually used to execute matrix multiplication operators, convolution operators, etc.

[0054] The vector computing unit 104 usually has multiple single instruction multiple data (Single Instruction Multiple Data, referred to as SIMD) arithmetic logic units and / or multiple single instruction multiple threads (Single Instruction Multiple Threads, referred to as SIMT) arithmetic logic units, which are used to perform scalar and vector calculations. Therefore, the vector computing unit 104 is usually used to execute a normalization (softmax) operator, a random dropout (dropout) operator, etc.

[0055] The tensor calculation unit 103 and the vector calculation unit 104 can exchange data through the video memory 101. In addition to the above structure, the artificial intelligence chip 100 in this application may also include other structures, which are not specifically limited in this application.

[0056] The artificial intelligence chip 100 can be: a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a domain-specific architecture (DSA), etc.

[0057] In practical applications, the Attention operator in large language model reasoning uses an autoregressive method to generate characters in rounds. When generating characters in each round, not only the query (Q) representation, key (K) representation, and value (V) representation calculated in this round are required, but also the key (K) representation and value (V) representation calculated in each previous round are required. In order to avoid repeatedly calculating the key (K) representation and value (V) representation calculated in each round when generating characters in each round, the related technology uses the key-value cache (KVCache) technology to save the key (K) representation and value (V) representation calculated in each round in the video memory. In this way, when generating characters in each round, the key (K) representation and value (V) representation calculated in each previous round can be directly read from the video memory.

[0058] Although the above key-value cache technology can improve the inference speed, it also leads to a large amount of additional video memory usage.

[0059] In view of this, this application is based on Figure 1 The architecture diagram of the artificial intelligence chip shown in Figure 1 provides a flow chart of the optimization method of the attention mechanism operator, such as Figure 2 As shown, the process of the method is executed by an artificial intelligence chip and includes the following steps:

[0060] Step 201 , mapping the input tensor into a first key representation, a first value representation, and a first query representation.

[0061] Specifically, a tensor is a basic data structure for operator calculations, which can be a scalar or a one-dimensional or multi-dimensional array. Different tensors can have different data types and shapes.

[0062] Key representation refers to the specific content of the key (K) in the attention mechanism, which can be a scalar, vector, multi-dimensional array, etc. Similarly, value representation refers to the specific content of the value (V) in the attention mechanism, which can be a scalar, vector, multi-dimensional array, etc. Query representation refers to the specific content of the query (Q) in the attention mechanism, which can be a scalar, vector, multi-dimensional array, etc.

[0063] It should be noted that the descriptions such as “first”, “second”, and “third” in this application are only used to distinguish different data and are not used to limit the calculation order.

[0064] In the large language model, the Attention operator includes: the original key linear (KLinear) layer, the original value linear (VLinear) layer, and the original query linear (QLinear) layer. The original key linear layer, the original value linear layer, and the original query linear layer each correspond to an original weight (Weight), which is obtained through model training and saved in the video memory.

[0065] In some scenarios, the original key linear layer, the original value linear layer, and the original query linear layer each correspond to an original bias vector (Bias), which is also obtained through model training and saved in the video memory.

[0066] In order to reduce the memory usage of the original weights of the original key linear layer, the original value linear layer, and the original query linear layer, the present application obtains the original weights of the original key linear layer, the original value linear layer, and the original query linear layer from the weight file of the large language model after the training of the large language model is completed. Then, offline quantization operations are performed on each of the obtained original weights to obtain the key linear layer after offline quantization, the value linear layer after offline quantization, and the query linear layer after offline quantization. Of course, in addition to performing offline quantization operations on the original weights, offline quantization operations can also be further performed on the original bias vectors of the original key linear layer, the original value linear layer, and the original query linear layer. This application does not make specific limitations on this.

[0067] In this way, during the reasoning process of the large language model, the input tensor is linearly mapped through the key linear layer after offline quantization to obtain the first key representation; the input tensor is linearly mapped through the value linear layer after offline quantization to obtain the first value representation; the input tensor is linearly mapped through the query linear layer after offline quantization to obtain the first query representation.

[0068] In some embodiments, when the original weights of the original key linear layer are quantized offline, the following operations are specifically included: the original weights of the original key linear layer are quantized offline using a third quantization scaling factor to obtain a first weight after offline quantization, and the first weight is stored in a video memory.

[0069] In specific implementations, quantization refers to converting the data associated with the model from a high-precision data representation to a low-precision data representation, thereby reducing the video memory space occupied by the data representation. Offline quantization refers to the quantization operation performed before model inference. In this application, the type of quantization can be: INT8 quantization, INT4 quantization, etc.

[0070] The quantization scaling factor (Scale) refers to: a quantization scaling coefficient used in the quantization process, which is associated with the high-precision data representation before quantization and the low-precision data representation after quantization.

[0071] In some embodiments, the quantization scaling factor is calculated using the following formula (1):

[0072]

[0073] Wherein, S represents a quantization scaling factor, α represents the maximum value of the absolute values ​​of the original data in the data representation before quantization; and b represents the number of bits of the data type corresponding to the data representation after quantization.

[0074] The process of quantization using the quantization scaling factor can be expressed as the following formula (2):

[0075]

[0076] Among them, x represents the original data in the data representation before quantization, x q It represents the target data after quantizing the original data x. Round means: rounding to the nearest integer to represent the specific quantized integer type; Clip means: clipping the output result of the round function within the data range [-2 b-1 +1,2 b-1 -1] is truncated.

[0077] When performing dequantization, each quantized target data x q Multiplying by the quantization scaling factor S, we can get the inverse quantized data The specific formula is as follows:

[0078]

[0079] Taking INT8 quantization as an example, the maximum value of the absolute value of each original data in the FP32 (32-bit floating point) type data representation is set to α=63.5. Since the data type of the quantized data representation is INT8 (8-bit integer) type, b=8. According to the above formula (1), the quantization scaling factor S=0.5 can be obtained. Next, for an original data x=63.5 in the FP32 type data representation, the original data x can be truncated to [-127, 127] according to the above formula (2), and the quantized target data x can be obtained. q =127. The target data after quantization of other original data can be obtained in the same way. All the obtained target data constitute the quantized INT8 type data representation.

[0080] In the dequantization stage, that is, the process of converting INT8 type data representation into FP32 type data representation. For the target data x q =127, according to the above formula (3), the corresponding inverse quantization data can be calculated The same method can be used to obtain the dequantized data of other target data.

[0081] It should be noted that the above formulas (1) to (3) are merely a quantization and dequantization method exemplified in the present application. In practical applications, the embodiments of the present application may also adopt other quantization and dequantization methods, and the present application does not make any specific limitations on this.

[0082] In the present application, for the third quantization scaling factor, the third quantization scaling factor is: a quantization scaling coefficient between an original weight of the original key linear layer and the first weight.

[0083] In practical applications, the third quantization scaling factor can also be obtained by the above calculation (1). Of course, when the quantization method used is different, the specific method of obtaining the third quantization scaling factor is also different accordingly, and this application does not make any specific limitation on this.

[0084] Taking offline INT8 quantization as an example, after the large language model training is completed and before the large language model inference, the original weight of the FP32 (32-bit floating point) type of the original key linear layer is obtained, and the maximum value α is determined from the absolute value of each element value of the original weight. Substituting the maximum value α and the corresponding bit number b of the INT8 type into the above formula (1), the third quantization scaling factor can be obtained. Using the above formula (2), combined with the obtained third quantization scaling factor, the original weight of the FP32 type of the original key linear layer is converted into the first weight of the INT8 type, and the first weight is saved in the video memory.

[0085] It should be noted that when the original weight is a multidimensional array, a quantization scaling factor can be set for each dimension, and the quantization scaling factors corresponding to multiple dimensions can be the same or different. Then, according to the quantization scaling factor corresponding to each dimension, the data of the corresponding dimension is quantized respectively. The specific quantization process is the same as described above and will not be repeated here.

[0086] During the large language model inference process, a matrix multiplication is performed on the first weight and the input tensor, and the obtained calculation result is multiplied by the third quantization scaling factor to obtain a first key representation.

[0087] Specifically, the tensor calculation unit performs matrix multiplication calculation on the first weight and the input tensor to obtain a calculation result, and the vector calculation unit multiplies the obtained calculation result with the third quantization scaling factor to obtain a first key representation to achieve inverse quantization.

[0088] When the offline quantized key linear layer also corresponds to an original bias vector (that is, the original bias vector is not offline quantized), a matrix multiplication is performed on the first weight and the input tensor, and then the calculated result is multiplied by the third quantization scaling factor, and the multiplication result is added to the original bias vector to obtain the first key representation.

[0089] When the offline quantized key linear layer also corresponds to an offline quantized bias vector, a matrix multiplication is performed on the first weight and the input tensor, and then the obtained calculation result is added to the offline quantized bias vector, and the obtained addition result is multiplied by the third quantization scaling factor to obtain the first key representation.

[0090] For example, the first weight of type INT8 is read from the video memory, and then the first weight of type INT8 is matrix multiplied with the input tensor of type FP32 to obtain a calculation result of type FP32. Compared with the original weight of type FP32, the first weight of type INT8 actually lacks some data. Therefore, after performing matrix multiplication on the first weight of type INT8 and the input tensor of type FP32, the calculation result of type FP32 also lacks some data. In order to ensure the accuracy of the calculation result, it is also necessary to multiply the calculation result of type FP32 with the third quantization scaling factor, and add the obtained multiplication result to the original bias vector to obtain the first key representation of type FP32. This not only reduces the occupation of the video memory by the weight of the key linear layer, but also ensures the accuracy of the first key representation obtained during the model inference process.

[0091] In some embodiments, when the original weights of the original value linear layer are subjected to offline quantization, the following operations are specifically included: the original weights of the original value linear layer are subjected to offline quantization using a fourth quantization scaling factor, a second weight after offline quantization is obtained, and the second weight is stored in a video memory.

[0092] Specifically, the type of quantization may be: INT8 quantization, INT4 quantization, etc. The fourth quantization scaling factor is: the quantization scaling factor between the original weight of the original value linear layer and the second weight. In practical applications, the fourth quantization scaling factor may also be obtained by the above (1) calculation. Of course, when the quantization method used is different, the specific method of obtaining the fourth quantization scaling factor is also different accordingly, and this application does not make any specific limitation on this.

[0093] During the large language model inference process, matrix multiplication is performed on the second weight and the input tensor, and the obtained calculation result is multiplied by the fourth quantization scaling factor to obtain a first value representation.

[0094] Specifically, the tensor calculation unit performs matrix multiplication calculation on the second weight and the input tensor to obtain a calculation result, and the vector calculation unit multiplies the obtained calculation result with the fourth quantization scaling factor to obtain a first value representation to achieve inverse quantization.

[0095] When the offline quantized value linear layer also corresponds to an original bias vector (that is, the original bias vector is not offline quantized), a matrix multiplication is performed on the second weight and the input tensor, and then the calculated result is multiplied by the fourth quantization scaling factor, and the multiplication result is added to the original bias vector to obtain the first value representation.

[0096] When the offline quantized value linear layer also corresponds to an offline quantized bias vector, a matrix multiplication is performed on the second weight and the input tensor, and then the obtained calculation result is added to the offline quantized bias vector, and the obtained addition result is multiplied by the fourth quantization scaling factor to obtain the first value representation.

[0097] For example, the second weight of INT8 type is read from the video memory, and then the second weight of INT8 type is matrix multiplied with the input tensor of FP32 type to obtain the calculation result of FP32 type. In order to ensure the accuracy of the calculation result, the calculation result of FP32 type is further multiplied with the fourth quantization scaling factor, and the obtained multiplication result is added to the original bias vector to obtain the first value representation, which not only reduces the occupation of the video memory by the weight of the value linear layer, but also ensures the accuracy of the first value representation obtained during the model inference process.

[0098] In some embodiments, when the original weights of the original query linear layer are subjected to offline quantization, the following operations are specifically included: the original weights of the original query linear layer are subjected to offline quantization using a fifth quantization scaling factor, a third weight after offline quantization is obtained, and the third weight is stored in a video memory.

[0099] Specifically, the type of quantization can be: INT8 quantization, INT4 quantization, etc. The fifth quantization scaling factor is: the quantization scaling factor between the original weight of the original query linear layer and the third weight. In practical applications, the fifth quantization scaling factor can also be obtained by the above (1) calculation. Of course, when the quantization method used is different, the specific method of obtaining the fifth quantization scaling factor is also different accordingly, and this application does not make specific limitations on this.

[0100] During the large language model inference process, matrix multiplication is performed on the third weight and the input tensor, and the obtained calculation result is multiplied by the fifth quantization scaling factor to obtain the first query representation.

[0101] Specifically, the tensor calculation unit performs matrix multiplication calculation on the third weight and the input tensor to obtain a calculation result. The vector calculation unit multiplies the obtained calculation result with the fifth quantization scaling factor to obtain the first query representation to achieve inverse quantization.

[0102] When the offline quantized query linear layer also corresponds to an original bias vector (that is, the original bias vector is not offline quantized), a matrix multiplication is performed on the third weight and the input tensor, and then the calculated result is multiplied by the fifth quantization scaling factor, and the multiplication result is added to the original bias vector to obtain a first query representation.

[0103] When the offline quantized query linear layer also corresponds to an offline quantized bias vector, a matrix multiplication is performed on the third weight and the input tensor, and then the obtained calculation result is added to the offline quantized bias vector, and the obtained addition result is multiplied by the fifth quantization scaling factor to obtain the first query representation.

[0104] For example, the third weight of INT8 type is read from the video memory, and then the third weight of INT8 type is matrix multiplied with the input tensor of FP32 type to obtain the calculation result of FP32 type. In order to ensure the accuracy of the calculation result, the calculation result of FP32 type is further multiplied with the fifth quantization scaling factor, and the multiplication result is added to the original bias vector to obtain the first query representation, which not only reduces the occupation of the video memory by the weight of the query linear layer, but also ensures the accuracy of the first query representation obtained during the model inference process.

[0105] Step 202: quantize the first key representation using a first quantization scaling factor to obtain a second key representation; and quantize the first value representation using a second quantization scaling factor to obtain a second value representation.

[0106] Specifically, during the large language model inference process, the first key representation is online quantized using a first quantization scaling factor to obtain a second key representation, wherein the first quantization scaling factor is: a quantization scaling coefficient between the first key representation and the second key representation.

[0107] During the large language model inference process, the first value representation is online quantized using a second quantization scaling factor to obtain a second value representation, wherein the second quantization scaling factor is: a quantization scaling coefficient between the first value representation and the second value representation.

[0108] Step 203: Save the second key representation and the first quantization scaling factor in a key information set in the video memory, and save the second value representation and the second quantization scaling factor in a value information set in the video memory.

[0109] Specifically, the Attention operator generates characters in rounds by using an autoregressive method. When generating characters in each round, the key representation is quantized online according to the previous step 202, and then the quantized key representation and the corresponding quantization scaling factor are saved in the key information set in the video memory. Therefore, the key information set includes not only the quantized key representation and the corresponding quantization scaling factor obtained in this round, but also the quantized key representation and the corresponding quantization scaling factor obtained in each previous round.

[0110] Similarly, the value information set includes not only the quantized value representation and the corresponding quantization scaling factor obtained in this round, but also the quantized value representation and the corresponding quantization scaling factor obtained in each previous round.

[0111] Step 204, perform self-attention mechanism calculation based on the first query representation, the key information set and the value information set to obtain an output tensor.

[0112] Specifically, when the Attention operator generates characters in rounds using an autoregressive method, the output tensor is the characters generated in this round.

[0113] In an embodiment of the present application, during the large language model inference process, the first key representation is online quantized using a first quantization scaling factor to obtain a second key representation; and the first value representation is online quantized using a second quantization scaling factor to obtain a second value representation. The second key representation and the first quantization scaling factor are then saved in a key information set in the video memory, and the second value representation and the second quantization scaling factor are saved in a value information set in the video memory. Based on the first query representation, the key information set and the value information set in the video memory, a self-attention mechanism calculation is performed to obtain an output tensor. In this way, during the large language model inference process, the video memory only needs to store the quantized key-value data, which greatly reduces the additional video memory occupancy caused by the key-value cache, and also reduces the video memory occupancy of the Attention inference, thereby improving the performance of the Attention operator.

[0114] In some embodiments, when performing self-attention mechanism calculation based on the first query representation, the key information set, and the value information set, the following steps are specifically included:

[0115] A matrix multiplication calculation is performed on the first query representation and the key representation in the key information set to obtain a first calculation result. Then, the first calculation result is dequantized using the quantization scaling factor in the key information set to obtain a second calculation result.

[0116] Specifically, all quantized key representations (KCache) in the key information set are read from the video memory, and a matrix multiplication calculation is performed on the first query representation and the read key representation through the tensor calculation unit to obtain a first calculation result. Since the key representations in the key information set are all quantized key representations with some data missing, after the first query representation and the read key representation are matrix multiplied to obtain the first calculation result, it is necessary to use the quantization scaling factors of all the key representations in the key information set to dequantize the first calculation result to obtain a second calculation result.

[0117] In some embodiments, after performing a scaling operation on the first calculation result, the obtained scaling operation result is multiplied by the quantization scaling factor in the key information set to obtain a second calculation result.

[0118] Specifically, the vector calculation unit divides the first calculation result by a constant value to implement a scaling operation on the first calculation result. The vector calculation unit multiplies the obtained scaling operation result by the quantization scaling factors of all key representations in the key information set to obtain a second calculation result.

[0119] Next, the second calculation result is normalized, and the normalized result is multiplied by the quantization scaling factor in the value information set to obtain a third calculation result. Then, the third calculation result and the value representation in the value information set are matrix multiplied to obtain a fourth calculation result.

[0120] Specifically, the second calculation result is normalized by the vector calculation unit, and the normalized result is multiplied by the quantization scaling factor of each of the quantized value representations in the value information set to obtain a third calculation result. The third calculation result and all the quantized value representations in the value information set are matrix multiplied by the tensor calculation unit to obtain a fourth calculation result.

[0121] In some embodiments, when the Attention operator includes a mask operation, the second calculation result is masked by the vector calculation unit, and the obtained mask operation result is normalized to obtain a normalized operation result. The normalized operation result is then multiplied by the quantization scaling factor in the value information set to obtain a third calculation result. The third calculation result and the value representation in the value information set are matrix multiplied by the tensor calculation unit to obtain a fourth calculation result.

[0122] The fourth calculation result is linearly mapped through the output linear layer after offline quantization to obtain the output tensor.

[0123] Specifically, the Attention operator includes the original key linear layer, original value linear layer, and original query linear layer described above, as well as the original output linear (OLinear) layer. The original output linear layer corresponds to an original weight, which is obtained through model training and stored in the video memory. In some scenarios, the original output linear layer also corresponds to an original bias vector.

[0124] In order to reduce the memory usage of the original weights of the original output linear layer, after the training of the large language model is completed, the present application obtains the original weights of the original output linear layer from the large language model weight file, and then performs an offline quantization operation on the original weights of the original output linear layer to obtain the output linear layer after offline quantization. Of course, in addition to performing an offline quantization operation on the original weights of the original output linear layer, an offline quantization operation can be further performed on the original bias vector of the original output linear layer, and this application does not make specific limitations on this.

[0125] In some embodiments, when the original weights of the original output linear layer are subjected to an offline quantization operation, the following operations are specifically included: the original weights of the original output linear layer are subjected to offline quantization using a sixth quantization scaling factor to obtain a fourth weight after offline quantization, and the fourth weight is stored in a video memory.

[0126] Specifically, the type of quantization may be: INT8 quantization, INT4 quantization, etc. The sixth quantization scaling factor is: the quantization scaling factor between the original weight of the original output linear layer and the fourth weight. In practical applications, the sixth quantization scaling factor may also be obtained by the above (1) calculation. Of course, when the quantization method used is different, the specific method of obtaining the sixth quantization scaling factor is also different accordingly, and this application does not make any specific limitation on this.

[0127] During the large language model inference process, matrix multiplication is performed on the fourth weight and the fourth calculation result, and the obtained calculation result is multiplied by the sixth quantization scaling factor to obtain an output tensor.

[0128] Specifically, the tensor calculation unit performs matrix multiplication calculation on the fourth weight and the fourth calculation result to obtain a calculation result. The vector calculation unit multiplies the obtained calculation result with the sixth quantization scaling factor to obtain an output tensor.

[0129] When the offline quantized output linear layer also corresponds to an original bias vector (that is, the original bias vector is not offline quantized), a matrix multiplication is performed on the fourth weight and the fourth calculation result, and then the obtained calculation result is multiplied by the sixth quantization scaling factor, and the obtained multiplication result is added to the original bias vector to obtain the output tensor.

[0130] When the output linear layer after offline quantization also corresponds to an offline quantized bias vector, a matrix multiplication is performed on the fourth weight and the fourth calculation result, and then the obtained calculation result is added to the offline quantized bias vector, and the obtained addition result is multiplied by the sixth quantization scaling factor to obtain the output tensor.

[0131] For example, see Figure 3 , is a schematic diagram of a calculation process of an Attention operator provided in an embodiment of the present application, comprising the following steps:

[0132] Step 301, offline quantization is performed on the original key linear layer.

[0133] The original weight of the key linear layer of type FP32 is converted into the first weight of type INT8 by using the third quantization scaling factor, and the third quantization scaling factor and the first weight are saved in the video memory.

[0134] Step 302: Offline quantization is performed on the original value linear layer.

[0135] The fourth quantization scaling factor is used to convert the original weight of the FP32 type of the value linear layer into the second weight of the INT8 type, and the fourth quantization scaling factor and the second weight are saved in the video memory.

[0136] Step 303: Offline quantization is performed on the original query linear layer.

[0137] The fifth quantization scaling factor is used to convert the original weight of the FP32 type of the query linear layer into the third weight of the INT8 type, and the fifth quantization scaling factor and the third weight are saved in the video memory.

[0138] Step 304: Offline quantization is performed on the original output linear layer.

[0139] The sixth quantization scaling factor is used to convert the original weight of the FP32 type of the output linear layer into the fourth weight of the INT8 type, and the sixth quantization scaling factor and the fourth weight are saved in the video memory.

[0140] After the offline quantization process of steps 301 to 304 is completed, the following large language model inference process is started:

[0141] Step 305 , linearly map the input tensor in the key linear layer after offline quantization to obtain a first key representation.

[0142] Specifically, a matrix multiplication calculation is performed on the first weight of the INT8 type and the input tensor of the FP32 type to obtain a calculation result of the FP32 type. The obtained calculation result of the FP32 type is multiplied by the third quantization scaling factor to obtain a first key representation of the FP32 type.

[0143] Step 306: Online quantization is performed on the first key representation.

[0144] Specifically, a first quantization scaling factor is used to convert a first key representation of the FP32 type into a second key representation of the INT8 type, and the first quantization scaling factor and the second key representation of the INT8 type are stored in a key information set in a video memory.

[0145] Step 307 , linearly map the input tensor in the value linear layer after offline quantization to obtain a first value representation.

[0146] Specifically, a matrix multiplication calculation is performed on the second weight of the INT8 type and the input tensor of the FP32 type to obtain a calculation result of the FP32 type. The obtained calculation result of the FP32 type is multiplied by the fourth quantization scaling factor to obtain a first value representation of the FP32 type.

[0147] Step 308: Online quantization is performed on the first value representation.

[0148] Specifically, the first value representation of the FP32 type is converted into a second value representation of the INT8 type using the second quantization scaling factor, and the second quantization scaling factor and the second value representation of the INT8 type are stored in a value information set in a video memory.

[0149] Step 309 , linearly map the input tensor in the offline quantized query linear layer to obtain a first query representation.

[0150] Specifically, a matrix multiplication is performed on the third weight of the INT8 type and the input tensor of the FP32 type to obtain a calculation result of the FP32 type. The obtained calculation result of the FP32 type is multiplied by the fifth quantization scaling factor to obtain a first query representation of the FP32 type.

[0151] Step 310, perform matrix multiplication calculation.

[0152] Specifically, a tensor computing unit performs matrix multiplication calculation on a first query representation of the FP32 type and all key representations of the INT8 type in the key information set to obtain a first calculation result of the FP32 type.

[0153] Step 311, performing a scaling operation, a multiplication by a quantization coefficient operation and a mask operation.

[0154] Specifically, the first calculation result of the FP32 type is divided by a constant value through a vector calculation unit to obtain a scaling operation result, and then the scaling operation result is multiplied by the quantization scaling factor of each key representation in the key information set to obtain a second calculation result of the FP32 type. A mask operation is then performed on the second calculation result of the FP32 type to obtain a mask operation result of the FP32 type.

[0155] Step 312, performing a normalization operation and a multiplication operation by a quantization coefficient.

[0156] Specifically, the vector calculation unit performs a normalization operation on the FP32 type mask operation result to obtain a FP32 type normalized operation result. The FP32 type normalized operation result is multiplied by the quantization scaling factor in the value information set to obtain a third calculation result of the FP32 type.

[0157] Step 313, perform matrix multiplication calculation.

[0158] Specifically, the tensor computing unit performs matrix multiplication calculation on the third calculation result of the FP32 type and all value representations of the INT8 type in the value information set to obtain a fourth calculation result of the FP32 type.

[0159] Step 314: linearly map the fourth calculation result in the output linear layer after offline quantization to obtain an output tensor.

[0160] Specifically, a matrix multiplication is performed on the fourth weight of the INT8 type and the fourth calculation result of the FP32 type to obtain a calculation result of the FP32 type. The obtained calculation result of the FP32 type is multiplied by the sixth quantization scaling factor to obtain an output tensor of the FP32 type.

[0161] In an embodiment of the present application, before the large language model inference process, the key linear layer, the value linear layer, the query linear layer, and the output linear layer are offline quantized, and during the large language model inference process, the key representation and the value representation involved in the key-value cache are online quantized. In this way, the video memory only needs to store the weights of each linear layer after offline quantization and the key-value data of online quantization, which greatly reduces the additional video memory occupancy caused by the key-value cache, and also reduces the video memory occupancy of Attention inference, thereby improving the performance of the Attention operator.

[0162] Secondly, taking into account the characteristics of the tensor computing unit in the artificial intelligence chip that is specifically used to perform matrix multiplication, the present application proposes to adjust the multiplication operation by the quantization coefficient outside the matrix multiplication calculation, and integrate it with the scaling operation, mask operation and normalization operation performed by the vector computing unit (such as steps 311 and 312 described above). This avoids performing matrix multiplication calculations in the middle, thereby avoiding additional memory access operations and calculations, which is highly feasible and efficient.

[0163] Based on the same technical concept, the present application embodiment provides a structural diagram of an optimization device for an attention mechanism operator, such as Figure 4 As shown, the optimization device 400 of the attention mechanism operator includes:

[0164] A mapping module 401, configured to map an input tensor into a first key representation, a first value representation, and a first query representation;

[0165] A quantization module 402, configured to quantize the first key representation using a first quantization scaling factor to obtain a second key representation; and quantize the first value representation using a second quantization scaling factor to obtain a second value representation;

[0166] A saving module 403, configured to save the second key representation and the first quantization scaling factor in a key information set in a video memory, and to save the second value representation and the second quantization scaling factor in a value information set in a video memory;

[0167] The computing module 404 is used to perform self-attention mechanism calculation based on the first query representation, the key information set and the value information set to obtain an output tensor.

[0168] Optionally, the quantization module 402 is specifically used for:

[0169] Performing linear mapping on the input tensor through the offline quantized key linear layer to obtain the first key representation;

[0170] Performing linear mapping on the input tensor through an offline quantized value linear layer to obtain the first value representation;

[0171] The input tensor is linearly mapped through the offline quantized query linear layer to obtain the first query representation.

[0172] Optionally, the parameter information of the key linear layer after offline quantization includes: a first weight after offline quantization and a corresponding third quantization scaling factor;

[0173] The quantization module 402 is specifically used for:

[0174] A matrix multiplication calculation is performed on the first weight and the input tensor, and the obtained calculation result is multiplied by the third quantization scaling factor to obtain the first key representation.

[0175] Optionally, the parameter information of the offline quantized value linear layer includes: the offline quantized second weight and the corresponding fourth quantization scaling factor;

[0176] The quantization module 402 is specifically used for:

[0177] A matrix multiplication calculation is performed on the second weight and the input tensor, and the obtained calculation result is multiplied by the fourth quantization scaling factor to obtain the first value representation.

[0178] Optionally, the parameter information of the query linear layer after offline quantization includes: a third weight after offline quantization and a corresponding fifth quantization scaling factor;

[0179] The quantization module 402 is specifically used for:

[0180] A matrix multiplication calculation is performed on the third weight and the input tensor, and the obtained calculation result is multiplied by the fifth quantization scaling factor to obtain the first query representation.

[0181] Optionally, the calculation module 404 is specifically used for:

[0182] Performing a matrix multiplication calculation on the first query representation and the key representation in the key information set to obtain a first calculation result;

[0183] Dequantizing the first calculation result by using the quantization scaling factor in the key information set to obtain a second calculation result;

[0184] performing a normalization operation on the second calculation result, and multiplying the obtained normalization operation result by the quantization scaling factor in the value information set to obtain a third calculation result;

[0185] Performing a matrix multiplication calculation on the third calculation result and the value representation in the value information set to obtain a fourth calculation result;

[0186] The fourth calculation result is linearly mapped through the output linear layer after offline quantization to obtain the output tensor.

[0187] Optionally, the calculation module 404 is specifically used for:

[0188] After performing a scaling operation on the first calculation result, the obtained scaling operation result is multiplied by the quantization scaling factor in the key information set to obtain the second calculation result.

[0189] Optionally, the parameter information of the output linear layer after offline quantization includes: a fourth weight after offline quantization and a corresponding sixth quantization scaling factor;

[0190] The calculation module 404 is specifically used for:

[0191] A matrix multiplication calculation is performed on the fourth weight and the fourth calculation result, and the obtained calculation result is multiplied by the sixth quantization scaling factor to obtain the output tensor.

[0192] In an embodiment of the present application, during the large language model inference process, the first key representation is online quantized using a first quantization scaling factor to obtain a second key representation; and the first value representation is online quantized using a second quantization scaling factor to obtain a second value representation. The second key representation and the first quantization scaling factor are then saved in a key information set in the video memory, and the second value representation and the second quantization scaling factor are saved in a value information set in the video memory. Based on the first query representation, the key information set and the value information set in the video memory, a self-attention mechanism calculation is performed to obtain an output tensor. In this way, during the large language model inference process, the video memory only needs to store the quantized key-value data, which greatly reduces the additional video memory occupancy caused by the key-value cache, and also reduces the video memory occupancy of the Attention inference, thereby improving the performance of the Attention operator.

[0193] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Figure 5 As shown, it includes at least one artificial intelligence chip 100 and a memory 501 connected to the at least one artificial intelligence chip 100. The specific connection medium between the artificial intelligence chip 100 and the memory 501 is not limited in the embodiment of the present application. Figure 5 In the example, the artificial intelligence chip 100 and the memory 501 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0194] In an embodiment of the present application, the memory 501 stores instructions that can be executed by at least one artificial intelligence chip 100. At least one artificial intelligence chip 100 can execute the steps of the above-mentioned optimization method of the attention mechanism operator by executing the instructions stored in the memory 501.

[0195] Among them, the artificial intelligence chip 100 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and the optimization of the attention mechanism operator can be realized by running or executing instructions stored in the memory 501 and calling data stored in the memory 501. Optionally, the artificial intelligence chip 100 may include one or more processing units, and the artificial intelligence chip 100 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 501 can be implemented on the same chip, and in some embodiments, they can also be implemented separately on independent chips.

[0196] The artificial intelligence chip 100 can be a general-purpose processor, such as a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0197] Memory 501, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 501 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RandomAccess Memory, RAM), static random access memory (Static RandomAccess Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic storage, disk, optical disk, etc. Memory 501 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 501 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0198] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the optimization method of the above-mentioned attention mechanism operator.

[0199] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the optimization method of the above-mentioned attention mechanism operator.

[0200] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0202] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0203] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0204] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0205] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An optimization method for attention mechanism operators, It is characterized in that include: Mapping the input tensor into a first key representation, a first value representation, and a first query representation; quantizing the first key representation using a first quantization scaling factor to obtain a second key representation; and, quantizing the first value representation using a second quantization scaling factor to obtain a second value representation; storing the second key representation and the first quantization scaling factor in a key information set in a video memory, and storing the second value representation and the second quantization scaling factor in a value information set in a video memory; A self-attention mechanism calculation is performed based on the first query representation, the key information set, and the value information set to obtain an output tensor.

2. The method according to claim 1, It is characterized in that Mapping the input tensor into a first key representation, a first value representation, and a first query representation includes: Performing linear mapping on the input tensor through the offline quantized key linear layer to obtain the first key representation; Performing linear mapping on the input tensor through an offline quantized value linear layer to obtain the first value representation; The input tensor is linearly mapped through the offline quantized query linear layer to obtain the first query representation.

3. The method according to claim 2, It is characterized in that The parameter information of the key linear layer after offline quantization includes: the first weight after offline quantization and the corresponding third quantization scaling factor; The step of performing linear mapping on the input tensor through the offline quantized key linear layer to obtain the first key representation includes: A matrix multiplication calculation is performed on the first weight and the input tensor, and the obtained calculation result is multiplied by the third quantization scaling factor to obtain the first key representation.

4. The method according to claim 2, It is characterized in that The parameter information of the offline quantized linear layer includes: the offline quantized second weight and the corresponding fourth quantization scaling factor; The step of performing linear mapping on the input tensor through the offline quantized value linear layer to obtain the first value representation includes: A matrix multiplication calculation is performed on the second weight and the input tensor, and the obtained calculation result is multiplied by the fourth quantization scaling factor to obtain the first value representation.

5. The method according to claim 2, It is characterized in that The parameter information of the query linear layer after offline quantization includes: a third weight after offline quantization and a corresponding fifth quantization scaling factor; The step of performing linear mapping on the input tensor through the offline quantized query linear layer to obtain the first query representation includes: A matrix multiplication calculation is performed on the third weight and the input tensor, and the obtained calculation result is multiplied by the fifth quantization scaling factor to obtain the first query representation.

6. The method according to any one of claims 1 to 5, It is characterized in that The performing a self-attention mechanism calculation based on the first query representation, the key information set, and the value information set to obtain an output tensor includes: Performing a matrix multiplication calculation on the first query representation and the key representation in the key information set to obtain a first calculation result; Dequantizing the first calculation result by using the quantization scaling factor in the key information set to obtain a second calculation result; performing a normalization operation on the second calculation result, and multiplying the obtained normalization operation result by the quantization scaling factor in the value information set to obtain a third calculation result; Performing a matrix multiplication calculation on the third calculation result and the value representation in the value information set to obtain a fourth calculation result; The fourth calculation result is linearly mapped through the output linear layer after offline quantization to obtain the output tensor.

7. The method according to claim 6, It is characterized in that The step of using the quantization scaling factor in the key information set to dequantize the first calculation result to obtain the second calculation result includes: After performing a scaling operation on the first calculation result, the obtained scaling operation result is multiplied by the quantization scaling factor in the key information set to obtain the second calculation result.

8. The method according to claim 6, It is characterized in that The parameter information of the output linear layer after offline quantization includes: a fourth weight after offline quantization and a corresponding sixth quantization scaling factor; The output linear layer after offline quantization performs linear mapping on the fourth calculation result to obtain the output tensor, including: A matrix multiplication calculation is performed on the fourth weight and the fourth calculation result, and the obtained calculation result is multiplied by the sixth quantization scaling factor to obtain the output tensor.

9. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, It is characterized in that When the artificial intelligence chip executes the computer program, the steps of the method as claimed in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, It is characterized in that It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of the method according to any one of claims 1 to 8.

11. A computer program product, It is characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 8.

Citation Information

Cited By

  • Attention mechanism operator optimization method and device, storage medium and program product

    CN121212217A

  • Method, device, storage medium and program product for optimizing attention mechanism operator

    CN121212217B

  • Method for determining output result of model, and related apparatus

    WO2026113320A1