Quantization method, processing system and quantization unit of artificial intelligence model

Abstract

The invention discloses a quantification method of an artificial intelligence model, a processing system of the artificial intelligence model, a quantification unit and computing equipment. A quantization unit may divide a quantization matrix into a plurality of block quantization matrices applicable to processing granularities of respective computing units of a computing device running an AI model based on the processing granularities of the respective computing units. The quantization unit can perform quantization processing on the block quantization matrix to obtain the quantized block quantization matrix, and the quantized block quantization matrix is stored in the memory, so that the memory occupied by the AI model can be reduced. The quantization matrix can be a weight matrix or an activation matrix. In the quantization process of the quantization unit, quantization is carried out based on the processing granularity of each computing unit of the computing device running the AI model, outlier statistics does not need to be carried out based on the tensor level, and real-time quantization can be carried out on the weight matrix and the activation matrix by the AI model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a quantization method for an artificial intelligence model, a processing system for an artificial intelligence model, a quantization unit and a computing device. Background Art

[0002] Artificial intelligence (AI) models refer to the use of data and algorithms to model and solve problems in a certain field or task. Taking the transformer model as an example of an AI model, the transformer model is a neural network model based on the self-attention mechanism, which is used for natural language processing and other sequence-to-sequence tasks. The transformer model can be applied in fields such as machine translation, question-answering systems, speech recognition, and text generation. The transformer model can process long text sequences and has the ability to perform parallel computing, which allows for efficient training and prediction. With the development of artificial intelligence (AI) technology, the scale of transformer models is getting larger and larger. Large-scale transformer models have problems such as a large number of parameters and a large amount of memory occupied, which makes the computational complexity of the transformer model relatively high. Summary of the invention

[0003] In order to solve the above problems, a quantization method for an artificial intelligence model is provided in an embodiment of the present application. When the quantization matrix of the multi-head self-attention mechanism running in each encoding block and decoding block of the AI ​​model is stored, the quantization matrix can be divided into multiple block quantization matrices suitable for the processing granularity of each computing unit based on the processing granularity of each computing unit of the computing device running the AI ​​model. The AI ​​model can quantize the block quantization matrix to obtain the quantized block quantization matrix and store it in a memory, which can reduce the memory occupied by the AI ​​model. In addition, the present application also provides a processing system, a quantization unit and a computing device for an artificial intelligence model.

[0004] To this end, the following technical solutions are adopted in the embodiments of the present application:

[0005] In a first aspect, an embodiment of the present application provides a quantization method for an artificial intelligence AI model, the method being executed by a quantization unit, and the method comprising: obtaining a quantization matrix, the quantization matrix being a weight matrix or an activation matrix, the weight matrix being a matrix composed of weights connecting different neurons in the AI ​​model, and the activation matrix referring to a feature matrix obtained after the AI ​​model performs a linear transformation on an input sequence; based on the processing granularity of multiple computing units, dividing the quantization matrix into multiple block quantization matrices; for each block quantization matrix, extracting a first value and a second value, the first value referring to one or more elements of the maximum value in the block quantization matrix, and the second value referring to an element of the maximum value in the block quantization matrix except the first value; for each block quantization matrix, using the second value as a quantization coefficient, performing quantization calculation on the block quantization matrix excluding the first value to obtain a block quantization output matrix.

[0006] In this embodiment, the quantization unit can divide the quantization matrix into multiple block quantization matrices suitable for the processing granularity of each computing unit based on the processing granularity of each computing unit of the computing device running the AI ​​model. The quantization unit can quantize the block quantization matrix to obtain the quantized block quantization matrix and store it in the memory, which can reduce the memory occupied by the AI ​​model. The quantization matrix can be a weight matrix or an activation matrix. During the quantization process of the quantization unit, quantization is performed based on the processing granularity of each computing unit of the computing device running the AI ​​model, and there is no need to perform outlier statistics based on the tensor level, so that the AI ​​model can perform real-time quantization of both the weight matrix and the activation matrix.

[0007] In one embodiment, for each block quantization matrix, the second value is used as a quantization coefficient, and a quantization calculation is performed on the block quantization matrix except the first value to obtain a block quantization output matrix, which specifically includes: dividing each element of the block quantization matrix except the first value by the second value to obtain the block quantization output matrix corresponding to the block quantization matrix.

[0008] In this implementation, the quantization unit divides each element of the block quantization matrix except the first value by the quantization coefficient to quantize the block quantization matrix. The processing is simple to reduce the complexity of the calculation.

[0009] In one embodiment, the method further includes: constructing structured outliers corresponding to the multiple block quantization matrices, the structured outliers including a quantization information matrix and a low-precision matrix, the quantization information matrix including quantization coefficients and high-precision outliers, the low-precision matrix is ​​the quantization data corresponding to the block quantization matrix, the quantization coefficient is the second value, and the high-precision outliers include the first value and an index index of the first value.

[0010] In this implementation, after the quantization unit quantizes each block quantization matrix, the quantization result is formed into a structured outlier. The quantization unit stores the quantization matrix in the memory in the form of multiple structured outliers, which can reduce the memory occupied by the AI ​​model.

[0011] In one embodiment, extracting the first value and the second value for each block quantization matrix specifically includes: sorting the elements of the block quantization matrix according to the size of the value, setting a set number of elements at the front of the sort to the first value, and setting an element other than the set number at the front of the sort to the second value.

[0012] In one embodiment, the quantization unit is a hardware unit disposed in an AI processor.

[0013] In a second aspect, an embodiment of the present application provides a processing system for an artificial intelligence (AI) model, comprising: at least one computing unit, a memory, a quantization unit and a dequantization unit, wherein the first computing unit in the at least one computing unit is used to: obtain input data of the first computing unit; the dequantization unit is used to obtain a quantization weight matrix of the first computing unit from the memory, and perform a dequantization operation on the quantization weight matrix to obtain a weight matrix; the first computing unit is also used to perform inference calculations based on the input data and the weight matrix to obtain a first calculation result; the quantization unit is used to perform a quantization operation on the first calculation result to obtain a quantized first calculation result.

[0014] In this embodiment, the processing system can receive the quantized weight matrix and perform a dequantization operation on the quantized weight matrix, which can reduce the memory occupied by the data processed by the processing system. After the processing system generates a calculation result, the calculation result can be quantized, which can reduce the memory occupied by the data output by the processing system and reduce the overhead of transmitting data between computing units.

[0015] In one embodiment, the input data is a quantized activation matrix, and the dequantization unit is further used to perform a dequantization operation on the quantized activation matrix to obtain an activation matrix.

[0016] In one embodiment, the dequantization unit is specifically used to obtain the quantization weight matrix, which includes quantization coefficients, outlier values ​​and indexes of the outlier values. Based on the quantization coefficients, the quantization weight matrix is ​​dequantized to obtain a weight matrix excluding the outlier values; based on the outlier values ​​and the indexes of the outlier values, the outlier values ​​are replaced to corresponding positions in the weight matrix excluding the outlier values ​​to obtain the weight matrix.

[0017] In one embodiment, the inverse quantization unit is specifically used to divide the first calculation result into multiple block first calculation results based on the processing granularity of multiple calculation units; for each block first calculation result, extract the first value and the second value, the first value refers to one or more elements of the maximum value in the block first calculation result, and the second value refers to an element of the maximum value in the block first calculation result other than the first value; for each block first calculation result, use the second value as a quantization coefficient to perform quantization calculation on the block first calculation result excluding the first value to obtain the quantized first calculation result.

[0018] In one implementation, the at least one computing unit, the quantization unit, and the dequantization unit are all hardware units disposed in an AI processor.

[0019] In the third aspect, an embodiment of the present application provides a quantization unit, including: a block unit, used to obtain a quantization matrix, and based on the processing granularity of multiple computing units, divide the quantization matrix into multiple block quantization matrices; the quantization matrix is ​​a weight matrix or an activation matrix, the weight matrix is ​​a matrix composed of weights connecting different neurons in the AI ​​model, and the activation matrix refers to the feature matrix obtained after the AI ​​model performs a linear transformation on the input sequence; an outlier extraction unit, used to extract a first value for each block quantization matrix, and the first value refers to one or more elements with the maximum value in the block quantization matrix; a coefficient calculation unit, used to extract a second value for each block quantization matrix, and the second value refers to an element with the maximum value in the block quantization matrix except the first value; a quantization processing unit, used to perform quantization calculation on the block quantization matrix except the first value for each block quantization matrix, using the second value as the quantization coefficient, to obtain a block quantization output matrix.

[0020] In one embodiment, the quantization processing unit is specifically used to divide each element of the block quantization matrix except the first value by the second value to obtain the block quantization output matrix corresponding to the block quantization matrix.

[0021] In one embodiment, the quantization processing unit is also used to construct structured outliers corresponding to the multiple block quantization matrices, the structured outliers include a quantization information matrix and a low-precision matrix, the quantization information matrix includes quantization coefficients and high-precision outliers, the low-precision matrix is ​​the quantization data corresponding to the block quantization matrix, the quantization coefficient is the second value, and the high-precision outlier includes the first value and the index of the first value.

[0022] In one embodiment, it also includes: a statistical maximum value unit, which is specifically used to sort the elements of the block quantization matrix according to their numerical values, and let the outlier extraction unit set a set number of elements at the front of the sort as the first numerical value, and let the coefficient calculation unit set an element other than the set number at the front of the sort as the second numerical value.

[0023] In one embodiment, the quantization unit is a hardware unit disposed in an AI processor.

[0024] In a fourth aspect, an embodiment of the present application provides an inverse quantization method for artificial intelligence AI, which is performed by an inverse quantization unit, and the method includes: obtaining structured outliers, wherein the structured outliers include a quantization information matrix and a low-precision matrix, the quantization information matrix includes quantization coefficients and high-precision outliers, the low-precision matrix is ​​quantized data, and the high-precision outliers include a first value and an index of the first value; using the quantization coefficient to perform inverse quantization calculation on the quantized data to obtain a block quantization matrix with the outliers removed; based on the first value and the index of the first value, replacing the first value to the corresponding position in the block quantization matrix with the outliers removed to obtain a block quantization matrix.

[0025] In this embodiment, the dequantization unit can divide the quantization matrix into multiple block quantization matrices suitable for the processing granularity of each computing unit based on the processing granularity of each computing unit of the computing device running the AI ​​model. The dequantization unit can quantize the block quantization matrix to obtain the quantized block quantization matrix and store it in the memory, which can reduce the memory occupied by the AI ​​model. The quantization matrix can be a weight matrix or an activation matrix. During the quantization process of the dequantization unit, quantization is performed based on the processing granularity of each computing unit of the computing device running the AI ​​model, and there is no need to perform outlier statistics based on the tensor level, so that the AI ​​model can perform real-time quantization of both the weight matrix and the activation matrix.

[0026] After the next encoding block or decoding block of the AI ​​model receives the quantized new activation matrix, the dequantization unit dequantizes the quantized new activation matrix so that the next encoding block or decoding block can perform quantized inference based on the new activation matrix. The activation matrix transmitted between encoding blocks, between encoding blocks and decoding blocks, and between decoding blocks is a quantized activation matrix, which can reduce the memory occupied during data transmission.

[0027] In one embodiment, the block weight matrix is ​​a matrix that divides the quantization matrix based on the processing granularity of multiple computing units, the processing granularity refers to the elements of the maximum number of columns of the quantization matrix processed by the computing unit, and the computing unit is used to calculate the outer product of the block weight matrix and the activation matrix.

[0028] In one implementation, the first value is greater than each element in the block quantization matrix except the outlier.

[0029] In one embodiment, the quantization matrix is ​​a weight matrix or an activation matrix, the weight matrix is ​​a matrix composed of weights connecting different neurons in the AI ​​model, and the activation matrix refers to a feature matrix obtained after the AI ​​model performs a linear transformation on the input sequence.

[0030] In a fifth aspect, an embodiment of the present application provides an inverse quantization unit, comprising: an inverse quantization processing unit, used to obtain structured outliers, wherein the structured outliers include a quantization information matrix and a low-precision matrix, the quantization information matrix includes quantization coefficients and high-precision outliers, the low-precision matrix is ​​quantized data, and the high-precision outliers include a first value and an index of the first value; and using the quantization coefficient to perform inverse quantization calculation on the quantized data to obtain a block quantization matrix with outliers removed; an outlier replacement unit, used to replace the first value to a corresponding position in the block weight matrix with outliers removed based on the first value and the index of the first value to obtain a block quantization matrix.

[0031] In one embodiment, the block weight matrix is ​​a matrix that divides the quantization matrix based on the processing granularity of multiple computing units, the processing granularity refers to the elements of the maximum number of columns of the quantization matrix processed by the processing unit, and the computing unit is used to calculate the outer product of the block weight matrix and the activation matrix.

[0032] In one implementation, the first value is greater than each element in the block quantization matrix except the outlier.

[0033] In one embodiment, the quantization matrix is ​​a weight matrix or an activation matrix, the weight matrix is ​​a matrix composed of weights connecting different neurons in the AI ​​model, and the activation matrix refers to a feature matrix obtained after the AI ​​model performs a linear transformation on the input sequence.

[0034] In a sixth aspect, an embodiment of the present application provides a computing device, comprising: at least one memory, and at least one AI processor, wherein the at least one AI processor is used to execute instructions stored in the at least one memory, so that the computing device executes each possible implementation embodiment of the first aspect, or each possible implementation embodiment of the fourth aspect.

[0035] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute each possible implementation embodiment of the first aspect or each possible implementation embodiment of the fourth aspect.

[0036] In an eighth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, which, when executed by a computer, enable the computer to implement each possible implementation embodiment of the first aspect, or each possible implementation embodiment of the fourth aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0038] Figure 1 This is a schematic diagram of the architecture of a transformer model;

[0039] Figure 2 A schematic diagram of the architecture of a computing device provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of a process for quantizing a weight matrix provided in an embodiment of the present application;

[0041] Figure 4 A schematic diagram of a structured outlier formation process provided in an embodiment of the present application;

[0042] Figure 5 A schematic diagram of a process for dequantizing a structural outlier provided in an embodiment of the present application;

[0043] Figure 6 A schematic diagram of a process of performing reasoning based on a weight matrix and an activation matrix provided in an embodiment of the present application;

[0044] Figure 7A schematic diagram of the structure of a quantization unit and a dequantization unit provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0046] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0047] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0048] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0049] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0050] The AI ​​model may be a transformer model, a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a self-attention model, etc. The following takes the transformer model as an example.

[0051] It can be imagined that the technical solution protected by this application is mainly to quantize and dequantize the weight matrix and the activation matrix, which can be implemented not only in the transformer model as an example, but also in CNN models, RNN models, self-attention models and other models that require quantization and dequantization of weight matrices and activation matrices. This application again does not limit it.

[0052] Figure 1 This is a schematic diagram of the architecture of a transformer model. Figure 1 As shown in the figure, the transformer model mainly consists of an encoder and a decoder. The input embedding of the encoder can map each element in the input sequence into a low-dimensional vector representation through the embedding matrix. The positional encoding of the encoder can add a fixed-length vector representation to each position to indicate the relative position information of the position in the sequence.

[0053] The encoder consists of multiple layers of identical encoding blocks. Each encoding block consists of two sub-layers and a residual connection. The first sub-layer is a multi-head self-attention layer. The multi-head self-attention layer can model the relationship between each position in the input sequence through multiple attention heads, find the correlation between each position and other positions, and thus extract the global information of the input sequence.

[0054] The second sublayer is a position-based fully connected network layer, such as a feed-forward neural network (FFN). The fully connected network can complement the multi-head self-attention layer and perform nonlinear transformations on the features of each position, thereby enhancing the representation ability of the input sequence. The residual connection can add the input sequence to the output of the multi-head self-attention layer and the fully connected network layer, and after layer normalization, it can be used as the input sequence of the next layer. The output of the last encoding block of the encoder can be a vector of the global information of the input sequence, which serves as the input sequence of the decoder.

[0055] The target embedding of the decoder can map each element in the target sequence into a low-dimensional vector representation through the embedding matrix. The position encoding of the decoder is the same as that of the encoder, adding a fixed-length vector representation to each position to represent the relative position information of the position in the sequence.

[0056] The decoder consists of multiple layers of identical decoding blocks. Each decoding block consists of three sub-layers and a residual connection. The first sub-layer is a multi-head self-attention mechanism. The multi-head self-attention mechanism is the same as the multi-head self-attention mechanism of the encoder. It uses multiple attention heads to model the relationship between each position in the target sequence and find the correlation between each position and other positions.

[0057] The second sublayer is the multi-head encoder-decoder attention layer. Unlike the multi-head self-attention sublayer, the multi-head encoder-decoder attention layer can calculate the correlation between the current position of the decoder and each position in the encoder's output sequence, thereby incorporating the encoder's global information into the decoder.

[0058] The third sublayer is a position-based fully connected network layer. The fully connected network layer can be used as a supplement to the multi-head encoder-decoder attention layer to perform nonlinear transformations on the features of each position, thereby enhancing the representation of the input sequence. The residual connection can add the input sequence to the output of the multi-head self-attention layer, the multi-head encoder-decoder attention layer, and the fully connected network layer, and after layer normalization, it is used as the input sequence of the next layer. The output of the last decoding block of the decoder is processed by the softmax function to obtain the predicted sequence.

[0059] With the development of artificial intelligence technology, the accuracy requirements for transformer models are getting higher and higher. The accuracy of transformer models has a certain relationship with the number of parameters and memory. Generally speaking, the more parameters a transformer model has, the stronger the expressive power of the transformer model is, and the accuracy of the transformer model will also increase accordingly. However, the more parameters a transformer model has, the more memory and computing resources the transformer model consumes.

[0060] In order to reduce the number of parameters in the transformer model, the transformer model in the related technology is quantized to convert the floating-point model into a fixed-point model to reduce the memory and computing resources of the transformer model. For example, the transformer model converts the weight parameters and activation parameters that originally used 32-bit floating-point numbers into 8-bit or 16-bit integers to reduce storage space and memory bandwidth during calculation. The quantized transformer model can not only improve the inference speed, but also reduce bandwidth occupancy and energy consumption during network transmission.

[0061] The specific process of transformer model quantization is as follows: First, sample the training data, record the distribution of the data, and count the maximum and minimum values ​​of each weight parameter and activation parameter. Secondly, design a suitable quantization scheme based on the data collection and statistical results, including the number of quantization bits, the quantization range, etc. Next, convert the floating-point model to a fixed-point model, and perform training and fine-tuning. Next, evaluate the quantized model, including indicators such as model accuracy, inference speed, and storage size. Finally, deploy the quantized model to the target device and perform inference.

[0062] However, there is a certain proportion of outliers in the weight parameters and activation parameters of the transformer model. Outliers refer to abnormal values ​​that are significantly different from other values, which are often caused by errors or noise during operations such as data acquisition, transmission or processing. The transformer model requires statistics on outliers based on the tensor level, which is time-consuming, and the storage of outliers is unstructured data, which is not suitable for hardware processing. In addition, existing quantization methods can only quantize weight matrices offline, and cannot quantize activation matrices in real time.

[0063] Weight parameters (weight, W) are the weights connecting different neurons in a neural network, which are used to transmit information and calculate output. In the transformer model, weight parameters exist in the form of matrices. Weight matrices are usually network structures composed of multiple self-attention layers and fully connected layers, which are used to encode and decode input sequences. A weight matrix refers to the weights connecting different neurons in a neural network with a transformer model running a multi-head self-attention mechanism.

[0064] The activation parameter (activation, A) exists in the form of a matrix in the transformer model. The activation matrix refers to the output matrix of each layer in the neural network. When the input sequence undergoes a linear transformation at a certain layer, a new feature matrix will be obtained, which is the activation matrix. Each element in the activation matrix represents the output result of each neuron in the neural network. In the neural network, each neuron will undergo a linear transformation based on the input sequence, and a nonlinear transformation through the activation function, and output a scalar value, which is the value of each element in the activation matrix.

[0065] In order to solve the problem of the existing quantization scheme reducing the accuracy of the transformer model, the embodiment of the present application provides a quantization method for an AI model, an inverse quantization method for an AI model, and a computing device. When the quantization unit stores the quantization matrix of the multi-head self-attention mechanism running in each encoding block and decoding block of the transformer model, the quantization matrix can be divided into multiple block quantization matrices suitable for the processing granularity of each computing unit based on the processing granularity of each computing unit of the computing device running the transformer model. The quantization unit can quantize the block quantization matrix to obtain the quantized block quantization matrix and store it in the memory, which can reduce the memory occupied by the transformer model. When the quantization unit performs inference, each quantized block quantization matrix is ​​assigned to each computing unit, and the computing unit multiplies the weight matrix with the activation matrix to obtain a new activation matrix to improve the affinity between the transformer model and the hardware of the computing device. The quantization matrix can be a weight matrix or an activation matrix. During the quantization process, the quantization unit quantizes based on the processing granularity of each computing unit of the computing device running the transformer model. It does not need to perform outlier statistics based on the tensor level, so that the transformer model can perform real-time quantization on both the weight matrix and the activation matrix.

[0066] In one embodiment, when the quantization unit performs quantized inference, it can divide the weight matrix into multiple block weight matrices suitable for the processing granularity of each computing unit based on the processing granularity of each computing unit of the computing device running the transformer model, and send each block weight matrix to each computing unit, so that the computing unit multiplies the block weight matrix with the activation matrix to obtain a new activation matrix, so as to improve the affinity between the transformer model and the hardware of the computing device.

[0067] In another embodiment, after the quantization unit calculates a new activation matrix in a coding block or a decoding block, the new activation matrix can be quantized. After the next coding block or the decoding block receives the quantized new activation matrix, the dequantization unit dequantizes the quantized new activation matrix so that the next coding block or the decoding block performs quantization reasoning based on the new activation matrix. The activation matrix transmitted between coding blocks, between coding blocks and decoding blocks, and between decoding blocks is a quantized activation matrix, which can reduce the memory occupied during data transmission.

[0068] In an embodiment of the present application, the computing device running the transformer model may be a server, a computer, a desktop computer, a portable laptop computer, a smart phone or other device with computing capabilities, and may be a cluster consisting of multiple servers, a cluster consisting of multiple computers, a cluster consisting of multiple servers and multiple computers, or other clusters. A computing unit is the smallest computing unit of a computing device. For example, when a computing device may be a device consisting of multiple processors, the computing unit may be a processor. For another example, when a computing device is a device consisting of one processor, a computing unit may refer to a portion of the computing resources of a processor running a computing process. For another example, when a computing device is a computing device consisting of multiple servers, a computing unit may refer to a single server. And so on.

[0069] Figure 2 Schematic diagram of the architecture of a computing device provided in an embodiment of the present application. Figure 2 As shown, the computing device 200 may include a memory 210 and multiple AI processors 220. The multiple AI processors 220 are respectively AI processor 220-1, AI processor 220-2, ..., AI processor 220-n. n is a positive integer greater than or equal to 1. The multiple AI processors 220 are connected to the memory 210 to realize data transmission. The computing device 200 can run the processing system of the AI ​​model to realize quantization and dequantization operations on the weight matrix, quantization and dequantization operations on the activation matrix, and inference calculations on the weight matrix and the activation matrix.

[0070] The memory 210 may include a volatile memory, such as a random access memory (RAM). The memory 210 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD). In an embodiment of the present application, the memory 210 may store the weight matrices of each encoding block and each decoding block of the transformer model. The memory 210 may also store the data input by the transformer model, and the activation matrices output by each encoding block and each decoding block. Optionally, the memory 210 may include a plurality of sub-storage units, each of which is respectively configured in each AI processor 220, and may store the running program, processing data, etc. of the corresponding AI processor 220.

[0071] The AI ​​processor 220 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP). Each AI processor 220 may perform the function of a coding block or a decoder. In other embodiments, when the number of AI processors 220 of the computing device 200 is limited, Figure 2 The functions performed by the multiple AI processors 220 in the example may be performed by one or a few AI processors 220 in a time sequence.

[0072] like Figure 2 As shown, each AI processor 220 can be divided into a quantization unit 221, a dequantization unit 222, and a matrix calculation unit 223 according to the execution function. Among them, the quantization unit 221, the dequantization unit 222, and the matrix calculation unit 223 can all be implemented by software, or can be implemented by hardware, or can be implemented by a combination of software and hardware. In one embodiment, the AI ​​processor 220 can be a GPU. The GPU includes multiple cores, and each core can process and execute a function separately. Therefore, the quantization unit 221 can be one or more cores of the GPU. The dequantization unit 222 can be one or more cores of the GPU. The matrix calculation unit 223 can be one or more cores of the GPU.

[0073] The following takes the AI ​​processor 220-1 as an example to introduce the quantitative reasoning process of an encoding block or a decoding block of the transformer model.

[0074] The memory 210 stores the weight matrices of each encoding block and each decoding block of the transformer model. Among them, the weight matrix stored in the memory 210 is a quantized weight matrix, which can reduce the memory occupied by the transformer model. The weight matrix quantization process can be performed by the quantization unit 221 of the corresponding AI processor 220, or it can be performed by the main controller of the computing device 200. The following takes the quantization unit 221-1 corresponding to the weight matrix as an example to introduce the quantization of the weight matrix by the AI ​​processor 220.

[0075] Figure 3 FIG. 1 is a flow chart of quantizing a weight matrix provided in an embodiment of the present application. Figure 3 As shown, the process of quantizing the weight matrix by the quantization unit 221-1 is specifically as follows:

[0076] In step S301, the quantization unit 221-1 obtains the weight matrix W1.

[0077] In step S302, the quantization unit 221-1 divides the quantization matrix into a plurality of block weight matrices based on the processing granularities of the plurality of calculation units in the matrix calculation unit 223-1.

[0078] The computing unit refers to the computing unit that calculates part of the weight matrix in the matrix computing unit 223-1. The matrix computing unit 223-1 includes a plurality of computing units. The computing resources of each computing unit are limited, so that the number of columns of the weight matrix that the computing unit can process is also limited. In one embodiment, when the matrix computing unit 223-1 is composed of multiple cores of a GPU, each computing unit can correspond to a core of the GPU. In the embodiment of the present application, the quantization unit 221-1 can divide the weight matrix according to the processing granularity of each computing unit to obtain a plurality of block weight matrices.

[0079] Exemplarily, the matrix calculation unit 223-1 includes four calculation units. Among them, the maximum matrix that can be processed by the first calculation unit is a matrix of "a1×b1", and the processing granularity of the first calculation unit is "a1×b1". The maximum matrix that can be processed by the second calculation unit is a matrix of "a2×b2", and the processing granularity of the second calculation unit is "a2×b2". The maximum matrix that can be processed by the third calculation unit is a matrix of "a3×b3", and the processing granularity of the third calculation unit is "a3×b3". The maximum matrix that can be processed by the fourth calculation unit is a matrix of "a4×b4", and the processing granularity of the fourth calculation unit is "a4×b4". The weight matrix is ​​a matrix of "A×B".

[0080] When the quantization unit 221-1 determines that a1 is greater than or equal to A and b1 is greater than or equal to B in the processing granularity of the first computing unit, there is no need to split the weight matrix. At this time, the first computing unit can process the weight matrix independently. When the quantization unit 221-1 determines that a1 is less than A and / or b1 is less than B in the processing granularity of the first computing unit, the weight matrix is ​​split to fully cut out an "a1×b1" matrix from the "A×B" matrix. The quantization unit 221-1 can split the weight matrix of the remaining rows (or the weight matrix of the remaining columns) based on the processing granularity of the second computing unit, thereby splitting out a weight matrix that can be processed by the second computing unit. Next, the quantization unit 221-1 can split the weight matrix of the remaining columns (or the weight matrix of the remaining rows) based on the processing granularity of the third computing unit, thereby splitting out a weight matrix that can be processed by the third computing unit. Finally, the quantization unit 221 - 1 determines that the number of rows of the matrix after three-division is less than a4, and the number of columns is less than b4, and uses the matrix after three-division as the weight matrix processed by the fourth calculation unit.

[0081] In step S303, the quantization unit 221-1 extracts the first value and the second value for each block weight matrix.

[0082] The weight parameters in the weight matrix have a certain proportion of outliers. After the quantization unit 221-1 divides the weight matrix, the obtained block weight matrix contains more or less outliers. The quantization unit 221-1 can sort the weight parameters in the block weight matrix, set a set number of weight parameters to the first value, and set a weight parameter to the second value. The first value refers to one or several weight parameters with the largest value in the block weight matrix. The second value refers to the largest weight parameter in the block weight matrix except the first value. The transformer model can use the first value as an outlier of the block weight matrix and retain the original value. The quantization unit 221-1 can use the second value as a quantization coefficient for quantizing the block weight matrix.

[0083] After the quantization unit 221-1 screens out the first value of the block weight matrix, it removes the first value in the block weight matrix to prevent the outlier value of the first value from having too much influence on the calculation result after the block weight matrix is ​​quantized. After the quantization unit 221-1 removes the first value from the block weight matrix, it records the first value and the index of the first value. The index of the first value is also the row number and column number of the first value in the block weight matrix. When the quantization unit 221-1 dequantizes the block weight matrix, it can insert the first value into the block weight matrix according to the recorded first value and the index of the first value to obtain a complete block weight matrix.

[0084] In step S304, the quantization unit 221-1 performs quantization calculation on each block weight matrix except for the first value, using the second value as the quantization coefficient, to obtain a block quantization output matrix.

[0085] After obtaining the block weight matrix without the first value, the quantization unit 221-1 performs quantization on the block weight matrix. The quantization unit 221-1 selects a second value, divides each weight parameter of the block weight matrix without the first value by the second value, and obtains quantized data of the weight parameter. The value range of the quantized data is between 0 and 1.

[0086] In an embodiment of the present application, after the quantization unit 221-1 calculates each block weight matrix, the quantization data corresponding to the first value, the second value and the block weight matrix constitute a structured outlier W1', and the weight matrix is ​​stored in the memory 210 in the form of each structured outlier, which can reduce the memory occupied by the transformer model in the memory 210.

[0087] Structural outliers are abnormal values ​​that exist in a data set and are caused by inconsistencies in the data structure or pattern. Figure 4 As shown, the structured outlier generally includes a quantization information matrix and a low-precision matrix. The quantization information matrix includes a quantization coefficient and a high-precision outlier. In the embodiment of the present application, the quantization coefficient may refer to the second value. The high-precision outlier may refer to the first value and the index of the first value. The low-precision matrix refers to the quantization data corresponding to the block weight matrix.

[0088] When the AI ​​processor 220-1 performs quantization inference, it can obtain multiple structural outliers W1' of a weight matrix from the memory 210, and transmit the multiple structural outliers W1' to the dequantization unit 222-1. The dequantization unit 222-1 dequantizes the multiple structural outliers W1' to obtain multiple block weight matrices W1 after the weight matrix is ​​divided.

[0089] Figure 5 A schematic diagram of a process for dequantizing a structural outlier provided in an embodiment of the present application. Figure 5 As shown, the process of the dequantization unit 222-1 dequantizing the structural outlier W1' is specifically as follows:

[0090] Step S501: the inverse quantization unit 222-1 obtains a structured outlier W1', wherein each structured outlier is obtained by quantizing a block weight matrix divided based on the processing granularity of a computing unit.

[0091] In step S502, the dequantization unit 222-1 dequantizes the low-precision matrix based on the quantization coefficient of the structured outlier to obtain a block weight matrix with the first value removed.

[0092] The low-precision matrix is ​​the quantized data after the block weight matrix is ​​quantized without the first value. After obtaining the quantized data, the inverse quantization unit 222-1 multiplies the quantized coefficients by the quantized data to obtain each weight parameter in the block weight matrix. At this time, the first value is missing in the block weight matrix.

[0093] Step S503 , the dequantization unit 222 - 1 replaces the first value to the corresponding position in the block weight matrix without the first value based on the high-precision outlier value of the structured outlier, to obtain the block weight matrix W1 .

[0094] The high-precision outlier includes a first value and an index of the first value. The index of the first value refers to the row number and column number of the first value. The dequantization unit 222-1 obtains the high-precision outlier in the structured outlier, and determines the position of the first value in the block weight matrix based on the row number and column number of the first value. The dequantization unit 222-1 replaces the first value to the corresponding position in the block weight matrix, so that the block weight matrix is ​​a complete block weight matrix W1.

[0095] After receiving multiple block weight matrices W1 and activation matrices A1', the matrix calculation unit 223-1 allocates multiple block weight matrices to corresponding calculation units, and allocates activation matrices to each calculation unit. After each calculation unit receives the block weight matrix and activation matrix, it multiplies the block weight matrix with the activation matrix to obtain a new activation matrix.

[0096] Figure 6 Schematic diagram of a flow chart of a calculation unit performing reasoning based on a block weight matrix and an activation matrix provided in an embodiment of the present application. Figure 6 As shown in the figure, the reasoning process of each computing unit based on the block weight matrix and activation matrix is ​​as follows:

[0097] Step S601: The computing unit multiplies the block weight matrix by the activation matrix to obtain a block activation matrix.

[0098] After obtaining the block weight matrix and activation matrix, the calculation unit multiplies the weight parameters of different column numbers with the activation parameters of the corresponding rows of the activation matrix to obtain the block activation matrix. Exemplarily, the matrix calculation unit 223-1 assigns the weight parameters of the 1st to 3rd columns of the weight matrix to the first calculation unit. After the first calculation unit receives the weight parameters of the 1st to 3rd columns and the activation matrix, it multiplies the weight parameters of the 1st column with the activation parameters of the 1st row of the activation matrix to obtain the activation parameters of one column. Similarly, the first calculation unit multiplies the weight parameters of the 2nd column with the activation parameters of the 2nd row of the activation matrix to obtain the activation parameters of one column. The first calculation unit multiplies the weight parameters of the 3rd column with the activation parameters of the 3rd row of the activation matrix to obtain the activation parameters of one column.

[0099] In an embodiment of the present application, the calculation unit multiplies the block weight matrix with the full activation matrix to obtain the block activation matrix, which can reduce the amount of calculation and memory usage. This method is usually used in scenarios with limited computing resources, such as mobile terminals and embedded devices. In these scenarios, due to limited computing resources and memory capacity, if the full amount of data is calculated directly, it may lead to problems such as long calculation time or memory overflow.

[0100] Step S602: The calculation unit multiplies the block activation matrix by the weight parameters of the corresponding rows of the weight matrix to obtain an output matrix.

[0101] After the calculation unit calculates the block activation parameters, it can obtain the weight parameters of the corresponding row numbers of the weight matrix based on the number of columns of the block activation parameters. The calculation unit multiplies the activation parameters of each column by the weight parameters of each row. In one case, when the calculation unit receives the weight parameters of a row, it can multiply the activation parameters of a column by the weight parameters of a row, and superimpose the products to obtain an X×Y matrix. Among them, X represents the number of rows of the matrix, and X is equal to the number of weight parameters. Y represents the number of columns of the matrix, and Y is equal to the number of activation parameters.

[0102] In another case, when the computing unit receives k rows of weight parameters, it can multiply each activation parameter of the same column number with each weight parameter of the row, and superimpose the products to obtain k X×Y matrices. Optionally, after the computing unit obtains multiple matrices, it can use the allreduce operation to parallelize the calculation process, superimpose k X×Y matrices, and superimpose the elements of the same row and the same column of each matrix to obtain an X×Y matrix. After the computing unit superimposes multiple matrices, it outputs a matrix, which can reduce the amount of output data and thus improve computing efficiency.

[0103] In an embodiment of the present application, each eigenvector of the activation matrix is ​​multiplied by the weight matrix to obtain a new eigenvector. The new eigenvector is linearly transformed and can capture the relationship between the features of the input sequence. The activation matrix can perform weighted summation on the new eigenvector to obtain the final output. The weighted summation process is an attention mechanism that enables the transformer model to pay more attention to the features in the input sequence that are relevant to the current task. Since the product of the weight matrix and the activation matrix is ​​differentiable, the back-propagation algorithm can be used to train the transformer model so that the transformer model can automatically learn the relationship between the features of the input sequence and use it for downstream task processing.

[0104] Step S603: The computing unit multiplies the output matrix by the activation function to obtain an updated block output matrix.

[0105] The output matrix is ​​the X×Y matrix obtained by multiplying and superimposing the block activation matrix with the weight parameters of the corresponding rows. The activation function is a nonlinear transformation function in the neural network, which is used to increase the model's expressive power and nonlinear feature extraction capabilities. In the transformer model, the activation function can be a rectified linear unit (ReLU), a Gaussian error linear unit (GELU), and other functions.

[0106] In an embodiment of the present application, the computing unit multiplies the output matrix by the activation function, and the linear transformation and nonlinear change can be combined to enhance the expressive power of the transformer model. Generally, the linear transformation of the activation matrix can extract the relationship between the features of the input sequence, and the nonlinear transformation of the activation function can map the relationship between the features of the input sequence to a higher dimensional space, thereby better capturing the complex relationship between the input sequences. The combination of linear transformation and nonlinear change can effectively reduce the overfitting degree of the transformer model to improve the generalization ability of the transformer model. At the same time, the computing unit uses the activation function to limit the output of the transformer model to a certain range. For example, the sigmoid function can limit the output to between 0 and 1 to avoid excessive or small output.

[0107] After obtaining the updated block output matrices output by each computing unit, the matrix computing unit 223 can perform an allreduce operation on multiple updated block output matrices, superimpose the multiple updated block output matrices, and superimpose the elements of the same rows and columns of each updated block output matrix to obtain a complete activation matrix A1".

[0108] After the AI ​​processor 220-1 calculates the new activation matrix A1", it needs to transmit the new activation matrix to the AI ​​processor 220-2 corresponding to the next-level coding block or decoding block, so that the AI ​​processor 220-2 of the next-level coding block or decoding block can perform quantized reasoning. When data is sent between the AI ​​processor 220-1 and the AI ​​processor 220-2, a cache is required for storage. Therefore, before the AI ​​processor 220-1 sends the new activation matrix, the new activation matrix needs to be quantized, which can reduce the memory occupied by the transformer model in the cache.

[0109] After receiving the new activation matrix A1" from the matrix calculation unit 223-1, the quantization unit 221-1 performs quantization processing on the new activation matrix A1" to obtain multiple structured outliers A2 corresponding to the new activation matrix. The activation matrix quantization process is basically the same as the weight matrix quantization process. The process of quantizing the new activation matrix A1" by the quantization unit 221-1 can be referred to Figure 3 and Description Figure 3 technical solutions.

[0110] The inverse quantization unit 222-2 is generally located in the AI ​​processor 220 of the next-level coding block or decoding block. The inverse quantization unit 222-2 receives multiple structured outliers A2 corresponding to the new activation matrix sent by the AI ​​processor 220 of the previous-level coding block or decoding block, and performs inverse quantization on the multiple structured outliers to obtain multiple block activation matrices A2' divided by the new activation matrix. The activation matrix inverse quantization process is basically the same as the weight matrix inverse quantization process. The process of the quantization unit 221-2 inverse quantizing the new activation matrix can be referred to Figure 5 and Description Figure 5 technical solutions.

[0111] Figure 7 Schematic diagram of the structure of the quantization unit and the dequantization unit provided in the embodiment of the present application. Figure 7As shown, the quantization unit 221 includes a block unit 2211, a statistical maximum unit 2212, an outlier extraction unit 2213, a coefficient calculation unit 2214 and a quantization processing unit 2215. The block unit 2211 is used to obtain a quantization matrix. The quantization matrix is ​​a weight matrix or an activation matrix. The weight matrix is ​​a matrix composed of weights connecting different neurons in the AI ​​model. The activation matrix refers to the characteristic matrix obtained after the AI ​​model performs a linear transformation on the input sequence. The block unit 2211 is also used to divide the quantization matrix into multiple block quantization matrices based on the processing granularity of multiple computing units. The outlier extraction unit 2213 is used to extract a first value for each block quantization matrix. The first value refers to one or more elements of the maximum value in the block quantization matrix. The coefficient calculation unit 2214 is used to extract a second value for each block quantization matrix. The second value refers to an element of the maximum value in the block quantization matrix except the first value. The quantization processing unit 2215 is used to perform quantization calculation on each block quantization matrix except for the first value, using the second value as the quantization coefficient, to obtain a block quantization output matrix.

[0112] In one implementation, the quantization processing unit 2215 is specifically configured to divide each element of the block quantization matrix except the first value by a second value to obtain a block quantization output matrix corresponding to the block quantization matrix.

[0113] In one embodiment, the quantization processing unit 2215 is also used to construct structured outliers corresponding to multiple block quantization matrices. The structured outliers include a quantization information matrix and a low-precision matrix. The quantization information matrix includes quantization coefficients and high-precision outliers. The low-precision matrix is ​​the quantization data corresponding to the block quantization matrix. The quantization coefficient is a second value. The high-precision outlier includes a first value and an index of the first value.

[0114] In one embodiment, the statistical maximum unit 2212 is used to sort the elements of the block quantization matrix according to the numerical size, so that the outlier extraction unit 2213 sets the set number of elements at the front of the sort to the first numerical value, and the coefficient calculation unit 2214 sets an element other than the set number at the front of the sort to the second numerical value.

[0115] The inverse quantization unit 222 includes an inverse quantization processing unit 2221 and an outlier replacement unit 2222. The inverse quantization processing unit 2221 is used to obtain structured outliers. The structured outliers include a quantization information matrix and a low-precision matrix. The quantization information matrix includes quantization coefficients and high-precision outliers. The low-precision matrix is ​​quantized data. The high-precision outliers include a first value and an index of the first value. The inverse quantization processing unit 2221 is also used to perform inverse quantization calculation on the quantized data using the quantization coefficient to obtain a block quantization matrix with outliers removed. The outlier replacement unit 2222 is used to replace the first value to a corresponding position in the block quantization matrix with outliers removed based on the first value and the index of the first value to obtain a block quantization matrix.

[0116] The following takes the processing of the activation matrix by the quantization unit 221-1 and the dequantization unit 222-2 in two adjacent AI processors 220 as an example to introduce the quantization and dequantization process of the activation matrix.

[0117] In the quantization unit 221-1: after receiving the activation matrix A1", the block unit 2211 can divide the activation matrix into multiple block activation matrices based on the processing granularity of multiple computing units. The maximum value counting unit 2212 is used to count the largest values ​​of each activation parameter in each block activation matrix after receiving multiple block activation matrices, and sort the largest values. The outlier extraction unit 2213 is used to extract the first value of each block activation matrix, retain the original value as the outlier of the block activation matrix, and record the first value and the index of the first value. The first value is Refers to one or several activation parameters with the largest values ​​in the block activation matrix. The coefficient calculation unit 2214 is used to extract the second value of each block activation matrix, and use the second value as the quantization coefficient for quantizing the block activation matrix. The second value refers to the largest activation parameter in the block activation matrix except the first value. The quantization processing unit 2215 is used to remove the first value in the block activation matrix, and use the second value as the quantization coefficient to quantize the block matrix except the first value. The quantization processing unit 2214 is also used to form a structured outlier A2 with the quantization data corresponding to the first value, the second value and the block activation matrix.

[0118] In the inverse quantization unit 222-2: the inverse quantization processing unit 2221 is used to, after receiving multiple structured outliers A2, perform inverse quantization on the low-precision matrix based on the quantization coefficients of the structured outliers to obtain a block activation matrix with the first value removed. The outlier replacement unit 2222 is used to replace the first value to the corresponding position in the block activation matrix with the first value removed based on the high-precision outliers of the structured outliers to obtain the block activation matrix A2'.

[0119] In an embodiment of the present application, after the quantization unit 221 calculates a new activation matrix in a transformer model in a coding block or a decoding block, the new activation matrix can be quantized. After the next coding block or decoding block receives the quantized new activation matrix, the dequantization unit 222 dequantizes the quantized new activation matrix so that the next coding block or decoding block can perform quantized reasoning based on the new activation matrix. The activation matrix transmitted between coding blocks, between coding blocks and decoding blocks, and between decoding blocks is a quantized activation matrix, which can reduce the memory occupied during data transmission.

[0120] The present application also provides a computing device, which includes at least one memory and at least one AI processor. The AI ​​processor can execute the following Figure 1-Figure 7 And the above corresponding protected technical solutions, so that the computing device has the technical effect of the above protected technical solutions.

[0121] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer executes the above Figure 1-Figure 7 And the corresponding described AI model quantization method and AI model dequantization method.

[0122] In an embodiment of the present application, a computer program product is also provided, wherein the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements the above Figure 1-Figure 7 And the corresponding described AI model quantization method and AI model dequantization method.

[0123] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0124] In addition, various aspects or features of the embodiments of the present application can be implemented as methods, devices or products using standard programming and / or engineering techniques. The term "product" used in this application covers computer programs that can be accessed from any computer-readable device, carrier or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks or tapes, etc.), optical disks (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards and flash memory devices (e.g., erasable programmable read-only memory (EPROM), cards, sticks or key drives, etc.). In addition, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0125] In the above embodiment, Figure 7 The quantization unit 221 and the inverse quantization unit 222 in the embodiment can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, an SSD), etc.

[0126] It should be understood that in various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0127] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0128] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0129] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or an access network device, etc.) to perform all or part of the steps of the method described in each embodiment of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0131] The above is only a specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited to this. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A quantification method for an artificial intelligence (AI) model, characterized in that: The method is performed by a quantization unit, and the method includes: Obtain a quantization matrix, where the quantization matrix is ​​a weight matrix or an activation matrix, where the weight matrix is ​​a matrix composed of weights connecting different neurons in the AI ​​model, and the activation matrix refers to a feature matrix obtained after the AI ​​model performs a linear transformation on an input sequence; Based on the processing granularities of the plurality of computing units, the quantization matrix is ​​divided into a plurality of block quantization matrices; For each block quantization matrix, extract a first value and a second value, wherein the first value refers to one or more elements of the maximum value in the block quantization matrix, and the second value refers to an element of the maximum value in the block quantization matrix except the first value; For each of the block quantization matrices, the second value is used as a quantization coefficient, and a quantization calculation is performed on the block quantization matrix except for the first value, so as to obtain a block quantization output matrix.

2. The method according to claim 1, characterized in that The method further comprises: for each block quantization matrix, taking the second value as a quantization coefficient, performing quantization calculation on the block quantization matrix except for the first value, and obtaining a block quantization output matrix. Each element of the block quantization matrix except the first value is divided by the second value to obtain the block quantization output matrix corresponding to the block quantization matrix.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Construct structured outliers corresponding to the multiple block quantization matrices, wherein the structured outliers include a quantization information matrix and a low-precision matrix, the quantization information matrix includes quantization coefficients and high-precision outliers, the low-precision matrix is ​​the quantization data corresponding to the block quantization matrix, the quantization coefficient is the second value, and the high-precision outliers include the first value and an index of the first value.

4. The method according to any one of claims 1 to 3, characterized in that: The extracting the first value and the second value for each block quantization matrix specifically includes: The elements of the block quantization matrix are sorted according to numerical values, a set number of elements at the front of the sort are set to the first numerical value, and an element other than the set number at the front of the sort is set to the second numerical value.

5. The method according to any one of claims 1 to 4, characterized in that: The quantization unit is a hardware unit placed in the AI ​​processor.

6. A processing system for an artificial intelligence (AI) model, characterized in that: include: At least one computing unit, a memory, a quantization unit and a dequantization unit, wherein, The first computing unit in the at least one computing unit is used to: obtain input data of the first computing unit; The dequantization unit is used to obtain the quantization weight matrix of the first calculation unit from the memory, and perform a dequantization operation on the quantization weight matrix to obtain a weight matrix; The first calculation unit is further used to perform inference calculation according to the input data and the weight matrix to obtain a first calculation result; The quantization unit is used to perform a quantization operation on the first calculation result to obtain a quantized first calculation result.

7. The system according to claim 6, characterized in that The input data is a quantized activation matrix, The dequantization unit is further used to perform a dequantization operation on the quantized activation matrix to obtain an activation matrix.

8. The system according to claim 6 or 7, characterized in that: The dequantization unit is specifically used to obtain the quantization weight matrix, wherein the quantization weight matrix includes quantization coefficients, outlier values ​​and indexes of the outlier values, Based on the quantization coefficients, dequantization calculation is performed on the quantization weight matrix to obtain a weight matrix excluding outliers; Based on the outlier value and the index of the outlier value, the outlier value is replaced to a corresponding position in the weight matrix excluding the outlier value to obtain the weight matrix.

9. The system according to any one of claims 6 to 8, characterized in that: The dequantization unit is specifically configured to divide the first calculation result into a plurality of block first calculation results based on the processing granularities of the plurality of calculation units; For each first calculation result of each block, extract a first value and a second value, wherein the first value refers to one or more elements of the maximum value in the first calculation result of the block, and the second value refers to an element of the maximum value in the first calculation result of the block except the first value; For each of the first calculation results of each block, the second value is used as a quantization coefficient, and quantization calculation is performed on the first calculation result of the block except for the first value to obtain the quantized first calculation result.

10. The system according to any one of claims 6 to 9, characterized in that: The at least one computing unit, the quantization unit and the dequantization unit are all hardware units placed in the AI ​​processor.

11. A quantization unit, characterized in that: include: A block unit, used to obtain a quantization matrix, and divide the quantization matrix into a plurality of block quantization matrices based on the processing granularities of the plurality of computing units; The quantization matrix is ​​a weight matrix or an activation matrix. The weight matrix is ​​a matrix composed of weights connecting different neurons in the AI ​​model. The activation matrix refers to a feature matrix obtained after the AI ​​model performs a linear transformation on the input sequence. An outlier extraction unit, configured to extract a first value for each block quantization matrix, wherein the first value refers to one or more elements of the maximum value in the block quantization matrix; A coefficient calculation unit, configured to extract a second value for each of the block quantization matrices, where the second value refers to an element of a maximum value in the block quantization matrix except the first value; The quantization processing unit is used to perform quantization calculation on each block quantization matrix by using the second value as a quantization coefficient and removing the first value from the block quantization matrix to obtain a block quantization output matrix.

12. The quantization unit according to claim 11, characterized in that: The quantization processing unit is specifically used to divide each element of the block quantization matrix except the first value by the second value to obtain the block quantization output matrix corresponding to the block quantization matrix.

13. The quantization unit according to claim 11 or 12, characterized in that: The quantization processing unit is also used to construct structured outliers corresponding to the multiple block quantization matrices, the structured outliers include a quantization information matrix and a low-precision matrix, the quantization information matrix includes quantization coefficients and high-precision outliers, the low-precision matrix is ​​the quantization data corresponding to the block quantization matrix, the quantization coefficient is the second value, and the high-precision outlier includes the first value and the index of the first value.

14. The quantization unit according to any one of claims 11 to 13, characterized in that: Also includes: The statistical maximum value unit is specifically used to sort the elements of the block quantization matrix according to the numerical size, and let the outlier extraction unit set the set number of elements in front of the sorting to the first numerical value, and let the coefficient calculation unit set an element other than the set number in front of the sorting to the second numerical value.

15. The quantization unit according to any one of claims 11 to 14, characterized in that: The quantization unit is a hardware unit placed in the AI ​​processor.

16. A computing device, characterized in that: include: at least one memory, At least one AI processor, wherein the at least one AI processor is configured to execute instructions stored in the at least one memory so that the computing device performs the method as described in any one of claims 1 to 5.

17. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 5.

18. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 5.

Images