A method, device, medium, and program product for quantization after training a large model.

Abstract

This application relates to the field of artificial intelligence chip technology, and provides a quantization method, device, medium, and program product for large model training, to reduce the storage burden on video memory and improve the quantization speed of large models. The method includes: determining a smoother for a target operator in the large model; when there is no layer normalization operator between the target operator and its adjacent preceding operator, migrating the smoother of the target operator to the weights of the linear layer of the target operator, migrating the inverse of the smoother of the target operator to the weights of the linear layer of the preceding operator, and quantizing the target operator.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a quantization method, device, medium, and program product after training a large model. Background Technology

[0002] SmoothQuant is a post-training quantization (PTQ) technique designed for large language models, supporting 8-bit weight and activation quantization. The core idea of ​​SmoothQuant is to calculate a smoother using statistical methods and transfer it to the weights, thereby reducing the difficulty of activation quantization. This allows matrix operations in large models like Transformers to be performed with INT8 precision, significantly reducing memory usage and improving inference speed while maintaining accuracy.

[0003] The existing smoother transfer process includes the following steps: 1. Calculate the maximum activation value and maximum weight of each fully connected layer in the general dataset; 2. Based on the operator fusion granularity of the large model, fuse the corresponding linear layer weights (such as the linear layers in the query, key, value (QKV) projection of the attention mechanism, the linear layers in the up projection and gate projection of the feed-forward network (FFN), and calculate the smoother for each operator; 3. Transfer the smoother to the weights of the fully connected layer. If the operator before the linear layer is a root mean square (RMS) normalized layer, the inverse of the smoother can be transferred to the weights and biases of the RMS normalized layer.

[0004] Existing quantization methods for large models typically include four types: quantization of the attention QKV projection, quantization of the attention output projection, quantization of the FFN upward projection and gated projection, and quantization of the FFN downward projection. In some cases, when quantizing operators in a large model, if there is no layer normalization operator before the operator and its adjacent preceding operator, the inverse of the smoother cannot be transferred to the layer normalized weights, and a separate SmoothQuant operator needs to be inserted.

[0005] For example, when projecting the attention output, since there is no layer normalization between the attention value projection and the attention output projection, the inverse of the smoother cannot be transferred to the layer normalized weights, and a separate SmoothQuant operator needs to be inserted. Similarly, when projecting the FFN downward, since there is no layer normalization between the FFN upward projection, the gated projection, and the FFN downward projection, the inverse of the smoother cannot be transferred to the layer normalized weights, and a separate SmoothQuant operator needs to be inserted.

[0006] In summary, existing quantization methods for large models, when quantizing various operators in a large model, cannot transfer the inverse of the smoother to the layer normalization weights if there is no layer normalization operator between the operator and its adjacent preceding operator. A separate SmoothQuant operator needs to be inserted, which requires the smoother as a parameter. This leads to the need to store the smoother during quantization, increasing the GPU memory burden. Furthermore, the computation of the SmoothQuant operator requires reading the smoother and performing additional calculations, reducing the model quantization speed. Summary of the Invention

[0007] This application provides a quantization method, device, medium, and program product for large model training, which reduces the storage burden on video memory and improves the quantization speed of large models.

[0008] In a first aspect, embodiments of this application provide a quantization method after training a large model, the method comprising:

[0009] Determine the smoother for the target operator in the large model;

[0010] When there is no layer normalization operator between the target operator and the adjacent preceding operator, the smoother of the target operator is moved to the weights of the linear layer of the target operator, and the inverse of the smoother of the target operator is moved to the weights of the linear layer of the preceding operator to quantize the target operator.

[0011] Secondly, embodiments of this application provide a quantization apparatus after training a large model, the apparatus comprising:

[0012] The processing module is used to determine the smoother for the target operator in the large model;

[0013] The quantization module is used to quantize the target operator when there is no layer normalization operator between the target operator and the adjacent previous operator, by shifting the smoother of the target operator to the weights of the linear layer of the target operator, and shifting the inverse of the smoother of the target operator to the weights of the linear layer of the previous operator.

[0014] Optionally, the target operator is the output projection of an attention mechanism;

[0015] The quantization module is specifically used for:

[0016] The smoother of the output projection is moved to the weights of the output projection linear layer of the attention mechanism, and the inverse of the smoother of the output projection is moved to the weights of the value projection linear layer of the attention mechanism.

[0017] Optionally, the attention mechanism is a grouped query attention mechanism;

[0018] The quantization module is specifically used for:

[0019] Obtain the configuration parameters of the grouped query attention mechanism, including: the head dimension, the number of key-value heads, and the number of groups of the grouped query attention mechanism;

[0020] Based on the head dimension, the number of key-value heads, and the number of groups, the smoother of the output projection is processed to obtain a first smoother, the dimension of the first smoother being the same as the dimension of the attention mechanism value projection;

[0021] Based on the head dimension, the number of key-value heads, and the number of groups, the first smoother is processed to obtain a second smoother, the dimension of the second smoother being the same as the dimension of the output projection of the attention mechanism;

[0022] The inverse of the first smoother is transferred to the weights of the attention mechanism value projection linear layer, and the second smoother is transferred to the weights of the attention mechanism output projection linear layer.

[0023] Optionally, the quantization module is specifically used for:

[0024] The smoother of the output projection is reshaped into an initial three-dimensional tensor, wherein the dimension values ​​of the first, second, and third dimensions of the initial three-dimensional tensor are equal to the values ​​of the number of key-value heads, the number of groups, and the head dimension, respectively.

[0025] In the initial three-dimensional tensor, taking the second dimension of the initial three-dimensional tensor as a reference, the average value of multiple values ​​at the same position in the first dimension and the third dimension is calculated to obtain the initial two-dimensional tensor. The dimension values ​​of the first dimension and the second dimension of the initial two-dimensional tensor are equal to the number of key heads and the value of the head dimension, respectively.

[0026] The initial two-dimensional tensor is reshaped into an initial one-dimensional tensor to obtain the first smoother.

[0027] Optionally, the quantization module is specifically used for:

[0028] The first smoother is reshaped into an intermediate two-dimensional tensor, wherein the dimension values ​​of the first dimension and the second dimension of the intermediate two-dimensional tensor are equal to the number of key heads and the value of the head dimension, respectively.

[0029] The values ​​of the first and second dimensions of the intermediate two-dimensional tensor are expanded N times to obtain an intermediate three-dimensional tensor, where N is equal to the number of groups. The dimension values ​​of the first, second, and third dimensions of the intermediate three-dimensional tensor are equal to the number of key headers, the number of groups, and the value of the header dimension, respectively.

[0030] The intermediate three-dimensional tensor is reshaped into an intermediate one-dimensional tensor to obtain the second smoother.

[0031] Optionally, the target operator is a projection under a feedforward neural network;

[0032] The quantization module is specifically used for:

[0033] The smoother of the projection of the feedforward neural network is transferred to the weights of the projection linear layer of the feedforward neural network, and the inverse of the smoother of the projection of the feedforward neural network is transferred to the weights of the projection linear layer of the feedforward neural network.

[0034] Optionally, the quantization module is further configured to:

[0035] When there is a layer normalization operator between the target operator and the adjacent previous operator, the smoother of the target operator is moved to the weights of the linear layer of the target operator, and the inverse of the smoother of the target operator is moved to the weights of the layer normalization operator to quantize the target operator.

[0036] Thirdly, embodiments of this application provide a computer device, including a memory, a processor chip, and a computer program stored in the memory and executable on the processor chip. When the processor chip executes the program, it implements the steps of the quantization method described above after training a large model.

[0037] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the quantization method described above after training the large model.

[0038] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a computer-readable storage medium, the computer program including program instructions, which, when executed by a computer device, cause the computer device to perform the steps of the quantization method after the large model training described above.

[0039] In this embodiment, after determining the smoother of the target operator in the large model, when quantizing the target operator, if it is determined that there is no layer normalization operator between the target operator and the adjacent preceding operator, the smoother of the target operator is moved to the weights of the linear layer of the target operator, and the inverse of the smoother of the target operator is moved to the weights of the linear layer of the preceding operator. This achieves quantization of the target operator, solving the problem that the inverse of the smoother cannot be moved to the weights of the layer normalization layer when there is no layer normalization operator between the target operator and the adjacent preceding operator, requiring the insertion of an independent SmoothQuant operator. Therefore, there is no need to store the smoother, reducing the memory burden, and there is no need to calculate the SmoothQuant operator, improving the quantization speed of the model. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application;

[0042] Figure 2 A flowchart illustrating a quantization method after training a large model, provided in an embodiment of this application;

[0043] Figure 3 A schematic diagram illustrating the principle of calculating the first smoother based on output projection, provided for an embodiment of this application;

[0044] Figure 4 This is a schematic diagram illustrating the principle of calculating a second smoother based on a first smoother, provided in an embodiment of this application.

[0045] Figure 5A schematic diagram illustrating the overall quantization process of the attention mechanism provided in the embodiments of this application;

[0046] Figure 6 A schematic diagram illustrating the overall quantization process of FFN provided in this application embodiment;

[0047] Figure 7 A schematic diagram of the structure of a quantization device after training a large model, provided in an embodiment of this application;

[0048] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0050] The following describes some of the concepts involved in the embodiments of this application.

[0051] 1. A token, or lexical unit, in large language models usually refers to a discrete unit of text. It can be a word, punctuation mark, number, or other language element, which is used as the basic unit for training and generating text.

[0052] 2. Large-scale models, also known as large-scale language models, refer to artificial intelligence language models generated by training on large-scale datasets with massive parameters (usually reaching billions or even hundreds of billions) based on deep learning technology. They can understand and generate human language and can handle various natural language processing tasks, such as text generation, translation, and question answering.

[0053] Large models are typically based on deep learning architectures, especially Transformer architectures, which can include multiple Transformers. Each Transformer can typically adopt the following structure: Layer Normalization operator + Attention mechanism operator + Layer Normalization & Residual Connection (add) operator + Feedforward Neural Network (FFN) operator.

[0054] 3. Grouped-Query Attention (GQA) is a compromise between Multi-Head Attention (MHA) and Multi-Query Attention (MQA). In MHA, each query head has its own independent query Q, key K, and value V projection. In MQA, multiple query heads share the same K and V projections. GQA, however, divides the query heads into G groups, with each group sharing the same K and V projections.

[0055] When attention is set to GQA, the number of query heads and output heads are the same, and the number of key heads and value heads are the same, but the number of query heads (or output heads) is a multiple of the number of value heads (or key heads) by the number of groups.

[0056] 4. Smoother, or smoother, is a key scaling factor in large models, especially in SmoothQuant quantization techniques. It can adjust the numerical distribution of activation values ​​and weights in each layer of the model, making activation values ​​that are originally difficult to quantize easier to quantize. Thus, while maintaining model accuracy, it can achieve efficient INT8 quantization inference.

[0057] Specifically, in actual Transformer models, activation values ​​often contain outliers, and direct quantization can lead to a significant drop in accuracy. The smoother model scales the activation values ​​and descales the weights, making the distribution of activation values ​​smoother. While this slightly increases the difficulty of weight quantization, it reduces the overall quantization error.

[0058] By applying smoother, large models can quantize both weights and activations to INT8 with almost no loss of accuracy, thus significantly reducing model storage and computational overhead. During quantization deployment, smoother can be integrated into adjacent operators (such as layer normalization) to avoid introducing additional computational overhead while maintaining model equivalence.

[0059] The design concept of the embodiments of this application is briefly introduced below:

[0060] SmoothQuant is a post-training quantization technique designed for large language models, supporting 8-bit weight and activation quantization. The core idea of ​​SmoothQuant is to calculate the smoother using statistical methods and transfer it to the weights, thereby reducing the difficulty of activation quantization. This allows matrix operations in large models like Transformers to be performed with INT8 precision, significantly reducing memory usage and improving inference speed while maintaining accuracy.

[0061] The existing smoother transfer process includes the following steps: 1. Calculate the maximum activation value and maximum weight of each fully connected layer in the general dataset; 2. Based on the operator fusion granularity of the large model, fuse the corresponding linear layer weights (such as the linear layers in QKV projection of attention, the linear layers in up projection and gate projection of FFN), and calculate the smoother for each operator; 3. Transfer the smoother to the weights of the fully connected layer. If the operator before the linear layer is an RMS normalized layer, the inverse of the smoother can be transferred to the weights and biases of the RMS normalized layer.

[0062] Existing quantization methods for large models typically include four types: quantization of the attention QKV projection, quantization of the attention output projection, quantization of the FFN upward projection and gated projection, and quantization of the FFN downward projection. In some cases, when quantizing operators in a large model, if there is no layer normalization operator before the operator and its adjacent preceding operator, the inverse of the smoother cannot be transferred to the layer normalized weights, and a separate SmoothQuant operator needs to be inserted.

[0063] For example, when projecting the attention output, since there is no layer normalization between the attention value projection and the attention output projection, the inverse of the smoother cannot be transferred to the layer normalized weights, and a separate SmoothQuant operator needs to be inserted. Similarly, when projecting the FFN downward, since there is no layer normalization between the FFN upward projection, the gated projection, and the FFN downward projection, the inverse of the smoother cannot be transferred to the layer normalized weights, and a separate SmoothQuant operator needs to be inserted.

[0064] In summary, existing quantization methods for large models, when quantizing various operators in a large model, cannot transfer the inverse of the smoother to the layer normalization weights if there is no layer normalization operator between the operator and its adjacent preceding operator. A separate SmoothQuant operator needs to be inserted, which requires the smoother as a parameter. This leads to the need to store the smoother during quantization, increasing the GPU memory burden. Furthermore, the computation of the SmoothQuant operator requires reading the smoother and performing additional calculations, reducing the model quantization speed.

[0065] In view of this, embodiments of this application provide a quantization method, device, medium, and program product after training a large model. After determining the smoother of the target operator in the large model, when quantizing the target operator, if it is determined that there is no layer normalization operator between the target operator and the adjacent preceding operator, the smoother of the target operator is moved to the weights of the linear layer of the target operator, and the inverse of the smoother of the target operator is moved to the weights of the linear layer of the preceding operator. This achieves quantization of the target operator, solving the problem that when there is no layer normalization operator between the target operator and the adjacent preceding operator, it is impossible to move the inverse of the smoother to the weights of the layer normalization layer, requiring the insertion of an independent SmoothQuant operator. Therefore, there is no need to store the smoother, reducing the memory storage burden, and there is no need to calculate the SmoothQuant operator, improving the model quantization speed.

[0066] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0067] refer to Figure 1 This is a structural diagram of an artificial intelligence chip applicable to an embodiment of this application. The artificial intelligence chip 100 includes at least: video memory 101 and multiple computing units 102, wherein the computing units 102 may be streaming processing clusters (SPCs). The video memory 101 may be high-bandwidth memory (HBM) or other types of memory.

[0068] In this embodiment, the large model may include multiple sub-networks, each deployed in a computing unit 102, and the parameters of each sub-network stored in the video memory 101. Multiple sub-networks may be deployed in a single computing unit 102 within an AI chip 100, or they may be deployed in multiple computing units 102 within multiple AI chips 100; this application does not specifically limit this deployment.

[0069] When performing quantization after training a large model, after determining the smoother of the target operator in the large model, the smoother of the target operator can be sent to memory 101. When it is determined that there is no layer normalization operator between the target operator and the adjacent previous operator, the smoother of the target operator is transferred to the weights of the linear layer of the target operator, and the inverse of the smoother of the target operator is transferred to the weights of the linear layer of the previous operator, thereby realizing the quantization of the target operator.

[0070] Compared to traditional quantization schemes (which require reading the stored `smoother`, calculating the `SmoothQuant` operator, and inserting the `SmoothQuant` operator between the target operator and the preceding operator when there is no layer normalization operator between them), this embodiment eliminates the need to insert the `SmoothQuant` operator and thus eliminates the need to store the `smoother` parameter, reducing the storage burden on GPU memory. Furthermore, this embodiment eliminates the need to read the `smoother` parameter, reducing data copying between the CPU and the AI ​​chip 100, and eliminating the need to calculate the `SmoothQuant` operator, thereby reducing CPU computational complexity and improving model quantization efficiency.

[0071] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.

[0072] Artificial intelligence chips 100 can be: Graphics Processing Unit (GPU), General-purpose computing on graphics processing units (GPGPU), Domain Specific Architecture (DSA), etc.

[0073] Specifically, the large model mentioned in the embodiments of this application can be a large model in various application scenarios, such as image processing scenarios, speech processing scenarios, text processing scenarios, etc. The large model can be used to process various tasks, and the quantization operation after the large model is trained can also be performed in the process of processing various tasks.

[0074] For example, in a text processing scenario, the input tensor of the large model can be text data used in tasks such as text generation and text recognition. After receiving the text data, during the processing of the text data using the large model, the parameters in the large model are quantized using the quantization method provided in the embodiments of this application after the large model is trained, and the quantized large model is used to perform inference processing on the text data to obtain the output result. This reduces the memory usage while maintaining the output accuracy of the large model, and improves the inference efficiency and speed of the large model.

[0075] For example, in a speech processing scenario, the input tensor of a large model can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis. After receiving the speech data, during the processing of the speech data using the large model, the parameters in the large model are quantized using the quantization method provided in the embodiments of this application after the large model is trained. The quantized large model is then used to perform inference processing on the speech data to obtain the output result. This reduces the memory usage while maintaining the output accuracy of the large model, and improves the inference efficiency and speed of the large model.

[0076] For example, in image processing scenarios, the input tensor of a large model can be image data used in tasks such as image preprocessing, image segmentation, and object detection. After receiving the image data, during the processing of the image data using the large model, the parameters in the large model are quantized using the quantization method provided in the embodiments of this application after the large model is trained, and the quantized large model is used to perform inference processing on the image data to obtain the output result. This reduces the memory usage while maintaining the output accuracy of the large model, and improves the inference efficiency and speed of the large model.

[0077] The following is based on Figure 1 The diagram shows the architecture of the artificial intelligence chip. A detailed description of the flow of a quantization method after training a large model, as provided in this application embodiment, is available in [reference needed]. Figure 2 This method is performed by a computer device, which includes... Figure 1 The artificial intelligence chip shown includes the following steps:

[0078] S201, determines the smoother for the target operator in the large model.

[0079] In specific implementation, when determining the smoother for each operator in the large model, the embodiments of this application may adopt methods from related technologies, and the embodiments of this application do not limit this. Among them, the QKV projection of the attention mechanism, the output projection of the attention mechanism, the up projection and gated projection of the feedforward neural network, and the down projection of the feedforward neural network are all operators in the large model, and the target operator can be any one of the above operators.

[0080] For example, when determining the smoother for each operator in a large model, we can first count the maximum activation value and the maximum weight value of each fully connected layer in the large model, and then, based on the operator fusion granularity of the large model, fuse the corresponding linear layer weights (such as the linear layers in the QKV projection of attention, the linear layers in the up projection and gate projection of FFN), and calculate the smoother for each operator.

[0081] S202, when there is no layer normalization operator between the target operator and the adjacent previous operator, the smoother of the target operator is moved to the weights of the linear layer of the target operator, and the inverse of the smoother of the target operator is moved to the weights of the linear layer of the previous operator to quantize the target operator.

[0082] In practical applications, considering that there is no layer normalization operator between the target operator and the adjacent previous operator, when quantizing the target operator, it is impossible to transfer the inverse of the smoother to the weights of the layer normalization. In this embodiment, after transferring the smoother of the target operator to the weights of the linear layer of the target operator, the inverse of the smoother of the target operator is transferred to the weights of the linear layer of the previous operator to achieve the quantization of the target operator.

[0083] When there is a layer normalization operator between the target operator and the adjacent previous operator, when quantizing the target operator, the smoother of the target operator can be moved to the weights of the linear layer of the target operator, and the inverse of the smoother of the target operator can be moved to the weights of the layer normalization operator, so as to realize the quantization of the target operator.

[0084] In practical applications, large models are usually based on deep learning architectures, especially the Transformer architecture, which can include multiple Transformers. Each Transformer can typically adopt the following structure: Layer normalization (LayerNorm or RMSNorm) operator + attention mechanism operator + Layer normalization (LayerNorm or RMSNorm) & residual connection (add) operator + feedforward neural network (FFN) operator. Among them, the attention operator includes: attention QKV projection + attention output projection, and the FFN operator includes: FFN down projection, FFN up projection and gated projection.

[0085] In the architecture mentioned above, there is no layer normalization operator between the attention output projection and the attention QKV projection, and there is no layer normalization operator between the FFN down projection, the FFN up projection, and the gated projection. The preceding operator of the attention QKV projection, the FFN up projection, and the gated projection is a layer normalization operator.

[0086] For the aforementioned large model architecture, in this embodiment of the application, when the target operator is attention output projection or FFN downward projection, the smoother of the target operator needs to be transferred to the weights of the linear layer of the target operator, and the inverse of the smoother of the target operator needs to be transferred to the weights of the linear layer of the previous operator; when the target operator is attention QKV projection or FFN upward projection and gated projection, the smoother of the target operator needs to be transferred to the weights of the linear layer of the target operator, and the inverse of the smoother of the target operator needs to be transferred to the weights of the normalization operator of the adjacent layer of the target operator.

[0087] The following detailed descriptions, using specific examples, illustrate the quantization process of the target operator when it is attention QKV projection, attention output projection, FFN up projection and gated projection, or FFN down projection.

[0088] Case 1: The target operator is attention QKV projection.

[0089] In this case, if the preceding operator of the attention QKV projection is a layer normalization operator, then the smoother of the attention QKV projection is transferred to the weights of the linear layer of the attention QKV projection, and the inverse of the smoother of the attention QKV projection is transferred to the weights of the layer normalization operator of the attention QKV projection.

[0090] Case 2: The target operator is the attention output projection.

[0091] In this case, the preceding operator adjacent to the attention output projection is the attention QKV projection, and there is no layer normalization operator between the attention output projection and the attention QKV projection. Then, the smoother of the attention output projection is moved to the weights of the attention output projection linear layer, and the inverse of the smoother of the attention output projection is moved to the weights of the attention V projection linear layer.

[0092] In one example, suppose the smoother for the attention output projection is... Before the attention output is projected and quantized, the output of the attention mechanism satisfies the following formula (1):

[0093] (1)

[0094] in, The output of the attention operator, As the input to the attention operator, V is the projection matrix in the attention operator. This is the output projection matrix in the attention operator.

[0095] After projection quantization of the attention output, the output of the attention mechanism satisfies the following formula (2):

[0096] (2)

[0097] From the above formulas (1) and (2), it can be determined that the output of the attention mechanism can remain mathematically equivalent before and after the attention output projection quantization in the embodiments of this application, thereby realizing the quantization of the attention output projection without inserting the SmoothQuant operator.

[0098] In practical applications, when attention is set to GQA, the number of query heads and output heads are the same, as are the number of key-value heads and value heads, but the number of output heads is a Group multiple of the number of value heads. In this case, due to the dimension mismatch, the inverse of the smoother for output projection cannot be directly transferred to the weights of the linear layer for value projection.

[0099] For the GQA-based attention mechanism, according to the calculation principle of GQA, the number of key headers and value headers are actually stretched by a group multiple in the header dimension, and then MHA is performed with the number of query headers. Therefore, when quantizing the attention output projection, the smoother also needs to be averaged over each group to obtain the smoother belonging to the value projection. Then, following the same stretching method, the smoother corresponding to the output projection is obtained, and then the transfer quantization operation is performed.

[0100] In specific implementation, after determining the smoother for the attention output projection, this application embodiment obtains the configuration parameters of GQA, including: the head dimension, the number of key-value heads, and the number of groups for the grouped query attention mechanism. Then, based on the head dimension, the number of key-value heads, and the number of groups, the smoother for the output projection is processed to obtain a first smoother. The dimension of the first smoother is the same as the dimension of the attention mechanism value projection. Based on the head dimension, the number of key-value heads, and the number of groups, the first smoother is processed to obtain a second smoother. The dimension of the second smoother is the same as the dimension of the attention mechanism output projection. Finally, the inverse of the first smoother is transferred to the weights of the linear layer of the attention mechanism value projection, and the second smoother is transferred to the weights of the linear layer of the attention mechanism output projection.

[0101] Specifically, based on the head dimension, the number of key-value heads, and the number of groups, the smoother of the output projection is processed to obtain the first smoother. First, the smoother of the output projection is reshaped into an initial three-dimensional tensor, where the dimension values ​​of the first, second, and third dimensions of the initial three-dimensional tensor are equal to the values ​​of the number of key-value heads, the number of groups, and the head dimension, respectively. Then, in the initial three-dimensional tensor, using the second dimension of the initial three-dimensional tensor as a reference, the average of multiple values ​​at the same position in the first and third dimensions is calculated to obtain an initial two-dimensional tensor, where the dimension values ​​of the first and second dimensions of the initial two-dimensional tensor are equal to the values ​​of the number of key-value heads and the head dimension, respectively. Finally, the initial two-dimensional tensor is reshaped into an initial one-dimensional tensor to obtain the first smoother.

[0102] The following is combined Figure 3 and Figure 4 The process of calculating the first and second smoothers based on output projection is explained in detail with specific examples.

[0103] In one example, assuming the hidden layer dimension in the GQA model is 4096, the feature dimension of each attention head is 128, the number of attention heads (or query heads q heads, output heads o heads) is 32, and the number of key heads and value heads is 8, then the hidden layer dimension for key projection and value projection is 8. 128 = 1024, the hidden layer dimension of the query projection and output projection is 32. 128=4096, Group number=4096 / / 1024=4.

[0104] In this example, the dimension of the smoother for the output projection is determined to be 4096, while the dimension of the value projection is 1024. Due to the dimension mismatch, the inverse of the smoother for the output projection cannot be directly migrated to the value projection. In this embodiment, a first smoother (with the same dimension as the value projection, migrated to the linear layer of the value projection) and a second smoother (with the same dimension as the output projection, migrated to the linear layer of the output projection) are determined based on the smoother for the output projection.

[0105] When calculating the first smoother based on the output projection smoother, first obtain the head dimension of GQA (128), the number of key-value heads (8), and the number of groups (4).

[0106] like Figure 3 As shown, the smoother of the determined output projection is assumed to be a one-dimensional tensor with the shape shown in Tensor 30, containing 4096 values, namely A0~A4095.

[0107] The smoother of the output projection is reshaped into an initial 3D tensor. The dimensions of the first, second, and third dimensions of this initial 3D tensor are equal to the values ​​of the number of key headers, the number of groups, and the head dimension, respectively. That is, the first dimension is 8, the second dimension is 4, and the third dimension is 128. Its shape is shown in Tensor 31, which is divided into 4 groups, each containing 8. 128 data points.

[0108] In the initial three-dimensional tensor, using the second dimension of the initial three-dimensional tensor as a reference, the average of multiple values ​​at the same position in the first and third dimensions is calculated to obtain the initial two-dimensional tensor. The dimension values ​​of the first and second dimensions of the initial two-dimensional tensor are equal to the number of key-value heads and the value of the head dimension, respectively. Its shape is shown in Tensor 32, including values ​​B0 to B1023. Among them, value B0 is the average of values ​​A0, A1024, A2048, and A3072; value B1 is the average of values ​​A1, A1025, A2049, and A3073; and so on. Value B1023 is the average of values ​​A1023, A2047, A3071, and A4095.

[0109] Finally, the initial two-dimensional tensor is reshaped into an initial one-dimensional tensor to obtain the first smoother, which has the shape shown in Tensor 33 and includes 1024 values, namely B0~B1023.

[0110] After calculating the first smoother, in order to keep the accuracy of the data before and after quantization unchanged, it is necessary to recalculate the smoother actually applied to the output projection based on the first smoother, that is, the second smoother mentioned in the embodiments of this application.

[0111] Specifically, based on the head dimension, the number of key-value heads, and the number of groups, the first smoother is processed to obtain the second smoother. First, the first smoother is reshaped into an intermediate two-dimensional tensor, where the dimensions of the first and second dimensions of the intermediate two-dimensional tensor are equal to the values ​​of the number of key-value heads and the head dimension, respectively. Then, the values ​​of the first and second dimensions of the intermediate two-dimensional tensor are expanded N times to obtain an intermediate three-dimensional tensor, where N is equal to the number of groups. The dimensions of the first, second, and third dimensions of the intermediate three-dimensional tensor are equal to the values ​​of the number of key-value heads, the number of groups, and the head dimension, respectively. Finally, the intermediate three-dimensional tensor is reshaped into an intermediate one-dimensional tensor to obtain the second smoother.

[0112] Using the example above, such as Figure 4 As shown, the first smoother has the shape of tensor 40 and contains 1024 values, namely B0~B1023.

[0113] When calculating the second smoother, the first smoother is first reshaped into an intermediate two-dimensional tensor. The dimensions of the first and second dimensions of the intermediate two-dimensional tensor are equal to the number of key heads and the head dimension, respectively. That is, the first dimension is 8 and the second dimension is 128. The reshaped intermediate two-dimensional tensor is shown in Tensor 41.

[0114] Then, the values ​​of the first and second dimensions of the intermediate two-dimensional tensor are expanded 4 times (the number of groups) to obtain the intermediate three-dimensional tensor. The intermediate three-dimensional tensor has a first dimension of 8, a second dimension of 4, and a third dimension of 128. Its shape is shown in Tensor 42, which includes 4 groups, each containing 8. 128 data points, which is equivalent to stretching tensor 41 into 4 sets of data.

[0115] Finally, the intermediate three-dimensional tensor is reshaped into an intermediate one-dimensional tensor to obtain the second smoother, which has the shape shown in tensor 43 and contains 4096 data points.

[0116] Through the above process, the embodiments of this application can convert the smoother of the attention output projection into a first smoother and a second smoother when the attention is GQA, thereby realizing the quantization of the attention output projection and solving the problem that quantization cannot be performed when the attention is GQA.

[0117] Case 3: The target operator is FFN projection and gated projection.

[0118] In this case, if the preceding operator of the FFN upward projection and the gated projection is the layer normalization operator, then the smoothers of the FFN upward projection and the gated projection are transferred to the weights of the FFN upward projection linear layer and the gated projection linear layer, respectively, and the inverses of the smoothers of the FFN upward projection and the gated projection are transferred to the weights of the layer normalization operator that is adjacent to both the FFN upward projection and the gated projection.

[0119] Case 4: The target operator is the projection under FFN.

[0120] In this case, the preceding operator of the FFN downward projection is the FFN upward projection, and there is no layer normalization operator between the FFN downward projection and the FFN upward projection. Then, the smoother of the FFN downward projection is moved to the weights of the FFN downward projection linear layer, and the inverse of the FFN downward projection smoother is moved to the weights of the FFN upward projection linear layer.

[0121] The above describes the quantization process of each operator in the large model after training, as provided in the embodiments of this application. It should be noted that the quantization method provided in the embodiments of this application, since the quantization process does not require the participation of a smoother, can therefore degenerate into a normal quantization method, quantizing each token individually.

[0122] The following describes the quantization method after training of a large model provided in the embodiments of this application, and explains the overall quantization process of the attention mechanism and FFN in the large model.

[0123] like Figure 5 As shown in the embodiments of this application, the quantization method for large model training, when quantizing the attention mechanism in the large model, specifically involves the following process:

[0124] Assuming the input data for the attention mechanism is The accuracy is BF16.

[0125] During the inference process, a smoother for the QKV projection is used due to quantization requirements. The reciprocal of the factor is transferred to the weights of the layer normalization operator. Therefore, the weights of the layer normalization operator are first scaled. Perform calculation Then, the input data is processed using the new weights to obtain the output data. Its precision is BF16, and then a quantization operation (quantization of each token) is performed to convert the data with a precision of BF16. Convert to INT8 and include the scaling factor (scale), then input the quantized data into the QKV projection operator.

[0126] To maintain data accuracy before and after quantization, the QKV fusion weights are adjusted in the QKV projective linear layer. Perform calculation Weights of the projection value Perform calculation (in, For the smoother of the output projection, during output projection quantization, the smoother of the output projection is moved to the linear layer of the value projection. Then, the new weights are used to perform correlation calculations with the input data to obtain the projection results q, k, and ... All data are of INT8 precision. Before performing subsequent attention calculations, to ensure the accuracy of the attention calculation, dequantization is performed first, converting the projection results q, k, ... The data precision is converted to BF16, and then attention is calculated.

[0127] Assuming the attention calculation output data Its accuracy is BF16, for data Quantize the data, convert its precision to INT8, and include a scaling factor (scale). Then, input and output the quantized data using projection.

[0128] To maintain data accuracy before and after quantization, the weights of the output projection in the output projection linear layer are... Perform calculation Using new output projection weights and input data Perform relevant calculations to obtain output data. The data precision is INT8, and it can then be dequantized to obtain the data. The precision is converted to BF16 for subsequent inference calculations.

[0129] like Figure 6 As shown in the embodiment of this application, the quantization method after training a large model, when quantizing the FFN in the large model, specifically involves the following process:

[0130] Assume the input data of FFN is The accuracy is BF16.

[0131] During the inference process, due to quantization requirements, smoothers for upprojection and gated projection are used. The reciprocal of the factor is transferred to the weights of the layer normalization operator. Therefore, the weights of the layer normalization operator are first scaled. Perform calculation Then, the input data is processed using the new weights to obtain the output data. Its precision is BF16, and then a quantization operation (quantization of each token) is performed to convert the data with a precision of BF16. The data is converted to INT8 and includes a scaling factor (scale). The quantized data is then input into the gated projection operator and the upprojection operator, respectively.

[0132] To maintain data accuracy before and after quantization, the weights of the gated projection in the gated projection linear layer are... Perform calculation Then, the gated projection uses the new weights and input data to perform correlation calculations to obtain the projection result. Its precision is INT8. In the linear layer with upward projection, the weights of the upward projection... Perform calculation (in, The smoother for downward projection is then migrated to the linear layer of upward projection. Then, upward projection uses the new weights to perform correlation calculations with the input data to obtain the projection result. Its precision is INT8. Before calculating the activation function on the gated projection result and multiplying the activation function result (act) with the upper projection result, inverse quantization is performed first to ensure calculation accuracy. , Projection results The data precision is converted to BF16, then the activation function is calculated on the gated projection result to obtain act, and then act is compared with the upper projection result. Perform multiplication (mul) calculations.

[0133] Assuming the data output by mul is Its accuracy is BF16, for data Quantize the data to INT8 precision, including the scaling factor, and then project the quantized data into the input.

[0134] To maintain data accuracy before and after quantization, the weights of the downprojection are adjusted in the downprojection linear layer. Perform calculation Using new downprojection weights and input data Perform relevant calculations to obtain output data. The data precision is INT8, and it can then be dequantized to obtain the data. The precision is converted to BF16 for subsequent inference calculations.

[0135] The above describes the quantization method after training of a large model provided in the embodiments of this application, and the specific quantization steps for the attention mechanism and FFN during the large model inference process.

[0136] The quantization method provided in this application after training a large model can completely eliminate the storage and reading / writing of the smoother through a specific smoother transfer scheme, thereby accelerating the computation of SmoothQuant during the inference process.

[0137] If the prefill and decoding stages are separated (prefill uses SmoothQuant, while decoding does not), the quantization method provided in this application's embodiments eliminates the need for weight restoration during decoding. Furthermore, script code verification shows that the combined error after inference with the same input is below 1.

[0138] In addition, this application proposes a smoother transfer method for attention output projection in GQA, enabling GQA to also use SmoothQuant and improve model inference speed.

[0139] Based on the same technical concept, this application provides a schematic diagram of the structure of a quantization device after large model training, as shown in the following embodiment. Figure 7 As shown, the data processing device 700 includes:

[0140] Processing module 701 is used to determine the smoother for the target operator in a large model;

[0141] The quantization module 702 is used to quantize the target operator by shifting the smoother of the target operator to the weights of the linear layer of the target operator and shifting the inverse of the smoother of the target operator to the weights of the linear layer of the previous operator when there is no layer normalization operator between the target operator and the adjacent previous operator.

[0142] Optionally, the target operator is the output projection of the attention mechanism;

[0143] Quantization module 702 is specifically used for:

[0144] The smoother of the output projection is moved to the weights of the linear layer of the output projection of the attention mechanism, and the inverse of the smoother of the output projection is moved to the weights of the linear layer of the value projection of the attention mechanism.

[0145] Optionally, the attention mechanism is a grouped query attention mechanism;

[0146] Quantization module 702 is specifically used for:

[0147] Obtain the configuration parameters for the grouped query attention mechanism. The configuration parameters include: the head dimension, the number of key-value heads, and the number of groups for the grouped query attention mechanism.

[0148] Based on the head dimension, the number of key-value heads, and the number of groups, the smoother of the output projection is processed to obtain the first smoother. The dimension of the first smoother is the same as the dimension of the value projection of the attention mechanism.

[0149] Based on the head dimension, the number of key-value heads, and the number of groups, the first smoother is processed to obtain the second smoother. The dimension of the second smoother is the same as the dimension of the output projection of the attention mechanism.

[0150] The inverse of the first smoother is transferred to the weights of the attention mechanism value projection linear layer, and the second smoother is transferred to the weights of the attention mechanism output projection linear layer.

[0151] Optional, quantization module 702, specifically used for:

[0152] The smoother of the output projection is reshaped into an initial 3D tensor, where the dimensions of the first, second, and third dimensions of the initial 3D tensor are equal to the values ​​of the number of key headers, the number of groups, and the head dimension, respectively.

[0153] In the initial three-dimensional tensor, taking the second dimension of the initial three-dimensional tensor as the reference, the average of multiple values ​​at the same position in the first and third dimensions is calculated to obtain the initial two-dimensional tensor. The dimension values ​​of the first and second dimensions of the initial two-dimensional tensor are equal to the number of key heads and the head dimension, respectively.

[0154] The initial two-dimensional tensor is reshaped into an initial one-dimensional tensor to obtain the first smoother.

[0155] Optional, quantization module 702, specifically used for:

[0156] The first smoother is reshaped into an intermediate two-dimensional tensor, where the dimension values ​​of the first and second dimensions of the intermediate two-dimensional tensor are equal to the number of key heads and the head dimension, respectively.

[0157] Expand the values ​​of the first and second dimensions of the intermediate two-dimensional tensor N times to obtain the intermediate three-dimensional tensor, where N equals the number of groups. The dimension values ​​of the first, second, and third dimensions of the intermediate three-dimensional tensor are equal to the values ​​of the number of key headers, the number of groups, and the head dimension, respectively.

[0158] The intermediate three-dimensional tensor is reshaped into an intermediate one-dimensional tensor to obtain the second smoother.

[0159] Optionally, the target operator is a projection onto a feedforward neural network;

[0160] Quantization module 702 is specifically used for:

[0161] The smoother of the downward projection of the feedforward neural network is transferred to the weights of the downward projection linear layer of the feedforward neural network, and the inverse of the smoother of the downward projection of the feedforward neural network is transferred to the weights of the upward projection linear layer of the feedforward neural network.

[0162] Optionally, the quantization module 702 is also used for:

[0163] When there is a layer normalization operator between the target operator and the adjacent previous operator, the smoother of the target operator is moved to the weights of the linear layer of the target operator, and the inverse of the smoother of the target operator is moved to the weights of the layer normalization operator to quantize the target operator.

[0164] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 8 As shown, it includes at least one processor chip 801 and a memory 802 connected to at least one processor chip. In this embodiment, the specific connection medium between the processor chip 801 and the memory 802 is not limited. Figure 8 Taking the connection between the processor chip 801 and the memory 802 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0165] In this embodiment of the application, the memory 802 stores instructions that can be executed by at least one processor chip 801. By executing the instructions stored in the memory 802, the at least one processor chip 801 can perform the steps of the quantization method after the large model training described above.

[0166] The processor chip 801 serves as the control center of the computer device, connecting to various parts of the device via various interfaces and lines. It performs data processing by running or executing instructions stored in the memory 802 and accessing data stored in the memory 802. Optionally, the processor chip 801 may include one or more processing units. The processor chip 801 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor chip 801. In some embodiments, the processor chip 801 and the memory 802 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0167] The processor chip 801 can be a general-purpose processor, such as a graphics processing unit (GPU), general-purpose computing on graphics processing units (GPGPU), central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0168] Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 802 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 802 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. In the embodiments of this application, memory 802 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0169] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the quantization method after the large model training described above.

[0170] Based on the same inventive concept, this application provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions that, when executed by a computer device, cause the computer device to perform the steps of the quantization method after the large model training described above.

[0171] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0173] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0175] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0176] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for post-training quantization of a large model, comprising: The method comprises: determining a smoother of a target operator in the large model, network parameters of the large model being stored in a display memory; when there is no layer normalization operator between the target operator and a previous adjacent operator, migrating the smoother of the target operator into weights of a linear layer of the target operator, migrating an inverse of the smoother of the target operator into weights of a linear layer of the previous operator, quantizing the target operator, and storing network parameters of the quantized large model to the display memory.

2. The method of claim 1, wherein, The target operator is an attention mechanism output projection. The migrating the smoother of the target operator into the weights of the linear layer of the target operator and the migrating the inverse of the smoother of the target operator into the weights of the linear layer of the previous operator comprise: The migrating the smoother of the output projection into the weights of the linear layer of the attention mechanism output projection and the migrating the inverse of the smoother of the output projection into the weights of the linear layer of the attention mechanism value projection comprise:

3. The method of claim 2, wherein, The attention mechanism is a group query attention mechanism. The migrating the smoother of the output projection into the weights of the linear layer of the attention mechanism output projection and the migrating the inverse of the smoother of the output projection into the weights of the linear layer of the attention mechanism value projection comprise: obtaining configuration parameters of the group query attention mechanism, the configuration parameters comprising: a head dimension of the group query attention mechanism, a number of key-value heads, and a number of groups; processing the smoother of the output projection based on the head dimension, the number of key-value heads, and the number of groups to obtain a first smoother, a dimension of the first smoother being the same as a dimension of the attention mechanism value projection; processing the first smoother based on the head dimension, the number of key-value heads, and the number of groups to obtain a second smoother, a dimension of the second smoother being the same as a dimension of the attention mechanism output projection; migrating the inverse of the first smoother into the weights of the linear layer of the attention mechanism value projection and migrating the second smoother into the weights of the linear layer of the attention mechanism output projection.

4. The method of claim 3, wherein, The processing the smoother of the output projection based on the head dimension, the number of key-value heads, and the number of groups to obtain a first smoother comprises: reshaping the smoother of the output projection into an initial three-dimensional tensor, wherein a first dimension, a second dimension, and a third dimension of the initial three-dimensional tensor have dimension values equal to values of the number of key-value heads, the number of groups, and the head dimension, respectively; calculating, in the initial three-dimensional tensor, an average value of a plurality of values at the same position in the first dimension and the third dimension based on the second dimension of the initial three-dimensional tensor to obtain an initial two-dimensional tensor, the initial two-dimensional tensor having a first dimension and a second dimension with dimension values equal to values of the number of key-value heads and the head dimension, respectively; reshaping the initial two-dimensional tensor into an initial one-dimensional tensor to obtain the first smoother.

5. The method of claim 3, wherein, The processing the first smoother based on the head dimension, the number of key-value heads, and the number of groups to obtain a second smoother comprises: The first smoother is reshaped into an intermediate two-dimensional tensor, wherein the dimension values of the first dimension and the second dimension of the intermediate two-dimensional tensor are equal to the values of the key value head number and the head dimension, respectively; The values of the first dimension and the second dimension of the intermediate two-dimensional tensor are expanded N times to obtain an intermediate three-dimensional tensor, wherein the N is equal to the group number, and the dimension values of the first dimension, the second dimension and the third dimension of the intermediate three-dimensional tensor are equal to the values of the key value head number, the group number and the head dimension, respectively; The intermediate three-dimensional tensor is reshaped into an intermediate one-dimensional tensor to obtain the second smoother.

6. The method of claim 1, wherein, The target operator is a projection under a feedforward neural network; The migration of the smoother of the target operator to the weight of the linear layer of the target operator and the migration of the inverse of the smoother of the target operator to the weight of the linear layer of the previous operator include: The migration of the smoother of the projection under the feedforward neural network to the weight of the linear layer of the projection under the feedforward neural network and the migration of the inverse of the smoother of the projection under the feedforward neural network to the weight of the linear layer of the projection under the feedforward neural network.

7. The method of claim 1, wherein, The method further includes: When there is a layer normalization operator between the target operator and the adjacent previous operator, the smoother of the target operator is migrated to the weight of the linear layer of the target operator, the inverse of the smoother of the target operator is migrated to the weight of the layer normalization operator, and the target operator is quantized.

8. A computer device comprising a memory, a processor chip and a computer program stored on the memory and executable on the processor chip, characterized in that, The processor chip implements the steps of the method of any one of claims 1-7 when executing the program.

9. A computer-readable storage medium, characterized in that, The computer program product includes a computer program stored on a computer readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a computer device, the computer device executes the steps of the method of any one of claims 1-7.

10. A computer program product, characterised in that, The computer program product includes a computer program stored on a computer readable storage medium, and the computer program includes program instructions, and when the program instructions are executed by a computer device, the computer device executes the steps of the method of any one of claims 1-7.

Images