Quantitative reasoning method and reasoning system of artificial intelligence (AI) model and computing equipment

Abstract

The invention discloses a quantitative reasoning method and system of an AI model and computing equipment. When the AI model is subjected to quantitative reasoning, the reasoning system distributes the weight parameters of the column where the outliers are located in the weight matrix to one or more calculation units in a centralized mode, and distributes the weight parameters which do not contain the column where the outliers are located in the weight matrix to other calculation units, so that batch processing of the outliers is facilitated, and the processing efficiency is improved. And the expenditure caused by outlier processing when other calculation units carry out quantitative reasoning is reduced. The AI model separates the weight parameter of the column where the outlier is located from the weight parameter which does not contain the column where the outlier is located, different data formats are adopted for processing, and high-efficiency compression is achieved on the premise that precision is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a quantitative reasoning method, reasoning system and computing device for an artificial intelligence (AI) model. Background Art

[0002] Artificial intelligence (AI) models refer to the use of data and algorithms to model and solve problems in a certain field or task. Taking the transformer model as an example of an AI model, the transformer model is a neural network model based on the self-attention mechanism, which is used for natural language processing and other sequence-to-sequence tasks. The transformer model can be applied in fields such as machine translation, question-answering systems, speech recognition, and text generation. The transformer model can process long text sequences and has the ability to perform parallel computing, which allows for efficient training and prediction. With the development of artificial intelligence (AI) technology, the scale of transformer models is getting larger and larger. Large-scale transformer models have problems such as a large number of parameters and a large amount of memory occupied, which makes the computational complexity of the transformer model relatively high. Summary of the invention

[0003] In order to solve the above problems, a quantitative reasoning method for an artificial intelligence AI model is provided in an embodiment of the present application, which centrally distributes the weight parameters of the outlier columns in the weight matrix to one or more computing units, and distributes the weight parameters of the columns that do not contain outliers in the weight matrix to other computing units, which is conducive to batch processing of outliers and reduces the overhead caused by processing outliers when other computing units perform quantitative reasoning. In addition, the present application also provides an inference system and computing device for an artificial intelligence AI model corresponding to the quantitative reasoning method of the artificial intelligence AI model.

[0004] To this end, the following technical solutions are adopted in the embodiments of the present application:

[0005] In a first aspect, an embodiment of the present application provides a quantitative reasoning method for an artificial intelligence AI model, the method being executed by an inference system, the method comprising: obtaining a weight matrix, the weight matrix being a matrix composed of weights connecting different neurons in the AI ​​model; assigning weight parameters of the column where outliers are located in the weight matrix to at least one first computing unit, and assigning weight parameters that do not contain outliers to at least one second computing unit by column; performing a first type of quantitative calculation on the weight parameters in the at least one first computing unit and an activation matrix, and performing a second type of quantitative calculation on the weight parameters in the at least one second computing unit and the activation matrix; wherein the activation matrix refers to a feature matrix obtained after the AI ​​model performs a linear transformation on an input sequence; superimposing the output matrix of the at least one first computing unit and the output matrix of the at least one second computing unit to obtain an output activation matrix.

[0006] In this embodiment, when the inference system performs quantitative reasoning on the AI ​​model, the weight parameters of the column where the outliers are located in the weight matrix are centrally distributed to one or more computing units, and the weight parameters of the column where the outliers are not located in the weight matrix are distributed to other computing units, which is conducive to batch processing of outliers and reduces the overhead of processing outliers when other computing units perform quantitative reasoning.

[0007] In one embodiment, the weight parameters of the column where the outliers in the weight matrix are located are allocated to at least one first calculation unit, specifically including: extracting the weight parameters of the column where the outliers in the weight matrix are located, and setting the weight parameters of the column where the outliers are located at the head of the weight matrix; based on the set threshold of each first calculation unit, allocating the weight parameters of the column where the outliers are located to a corresponding number of columns for each first calculation unit.

[0008] In this embodiment, when the inference system assigns weight parameters to each computing unit, it generally assigns weight parameters of several consecutive columns to the same computing unit. Therefore, the inference system can centrally arrange the columns where the outliers in the weight matrix are located at the head of the weight matrix, and assign weight parameters of a corresponding number of columns to each first computing unit according to the set threshold of each first computing unit, which is beneficial to the efficiency of the AI ​​model in assigning weight parameters to each first computing unit.

[0009] In one embodiment, the weight parameters in the at least one first calculation unit are subjected to a first type of quantization calculation with the activation matrix, and the weight parameters in the at least one second calculation unit are subjected to a second type of quantization calculation with the activation matrix, specifically including: multiplying the received weight parameters with the activation matrix to obtain activation parameters of the block; the number of columns of the activation parameters of the block is the same as the number of columns of the received weight parameters; multiplying the activation parameters of the block with the weight parameters of the block to obtain the output matrix; the weight parameters of the block refer to the weight parameters of at least one row in the weight matrix received by the first calculation unit or the second calculation unit, and the row number of the weight parameters of the block is the same as the column number of the received weight parameters.

[0010] In this embodiment, the computing unit multiplies the received weight parameters by the full activation matrix to obtain the block activation, which can reduce the amount of calculation and memory usage. Each eigenvector of the activation matrix is ​​multiplied by the weight matrix to obtain a new eigenvector. The new eigenvector is linearly transformed to capture the relationship between the features of the input sequence. The activation matrix can perform weighted summation on the new eigenvector to obtain the final output. The weighted summation process is an attention mechanism that enables the AI ​​model to pay more attention to the features in the input sequence that are related to the current task. Since the product of the weight matrix and the activation matrix is ​​differentiable, the backpropagation algorithm can be used to train the AI ​​model so that the AI ​​model can automatically learn the relationship between the features of the input sequence and use it for downstream task processing.

[0011] In one embodiment, it also includes: multiplying the output matrix by an activation function, where the activation function is a nonlinear transformation function in the AI ​​model.

[0012] In this embodiment, the computing unit multiplies the output matrix by the activation function, and can combine linear transformation and nonlinear change to enhance the expressiveness of the AI ​​model. The combination of linear transformation and nonlinear change can effectively reduce the overfitting degree of the AI ​​model to improve the generalization ability of the AI ​​model.

[0013] In one embodiment, multiplying the received weight parameters with the activation matrix to obtain the block activation parameters specifically includes: multiplying the weight parameters of different columns with the activation parameters of a row in the activation matrix with the same column number as the weight parameters to obtain activation parameters of multiple columns.

[0014] In one embodiment, multiplying the activation parameters of the block with the weight parameters of the block to obtain the output matrix specifically includes: multiplying the activation parameters of multiple columns with the weight parameters of each row of the weight parameters of the block to obtain multiple output matrices.

[0015] In one embodiment, it further includes: superimposing the multiple output matrices using full reduction calculation.

[0016] In this embodiment, when the computing unit receives multiple rows of weight parameters, each activation parameter of the same column number can be multiplied by each weight parameter of the row, and the products can be superimposed to obtain multiple output matrices. The computing unit can use the allreduce operation to parallelize the computing process, superimpose multiple output matrices, and superimpose the elements of the same row and the same column of each matrix to obtain an output matrix, which can reduce the amount of output data, thereby improving computing efficiency.

[0017] In one embodiment, the output matrix of the at least one first computing unit is superimposed with the output matrix of the at least one second computing unit to obtain an output activation matrix, which specifically includes: superimposing the elements in the same row and column of the output matrix of the at least one first computing unit and the output matrix of the at least one second computing unit to obtain the output activation matrix.

[0018] In one embodiment, the first type of quantization calculation is performed on the weight parameters and the activation matrix in the at least one first calculation unit, and the second type of quantization calculation is performed on the weight parameters and the activation matrix in the at least one second calculation unit, specifically including: using the FP 16 calculation method to calculate the weight parameters and the activation matrix of the column where the outliers are located, and using the INT 8 calculation method to calculate the weight parameters and the activation matrix of the column that does not contain the outliers.

[0019] In this implementation, the AI ​​model processes the weight parameters of the column containing the outliers separately from the weight parameters of the column containing the outliers, and can use different data formats for quantization processing to achieve efficient compression while ensuring accuracy.

[0020] In a second aspect, an embodiment of the present application provides an inference system, characterized in that it includes: a first processing unit, used to obtain a weight matrix, wherein the weight matrix is ​​a matrix composed of weights connecting different neurons in the AI ​​model; the weight parameters of the column where the outliers in the weight matrix are located are allocated to at least one first calculation unit, and the weight parameters that do not contain outliers are allocated to at least one second calculation unit by column; a second processing unit, used to perform a first type of quantization calculation on the weight parameters in the at least one first calculation unit and an activation matrix, and to perform a second type of quantization calculation on the weight parameters in the at least one second calculation unit and the activation matrix; wherein the activation matrix refers to a feature matrix obtained after the AI ​​model performs a linear transformation on the input sequence; a third processing unit, used to superimpose the output matrix of the at least one first calculation unit with the output matrix of the at least one second calculation unit to obtain an output activation matrix.

[0021] In one embodiment, the first processing unit is specifically used to extract the weight parameters of the column where the outliers in the weight matrix are located, and set the weight parameters of the column where the outliers are located at the head of the weight matrix; based on the set threshold of each first calculation unit, the weight parameters of the column where the outliers are located of a corresponding number of columns are allocated to each first calculation unit.

[0022] In one embodiment, the second processing unit is specifically used to multiply the received weight parameters with the activation matrix to obtain the activation parameters of the block; the number of columns of the activation parameters of the block is the same as the number of columns of the received weight parameters; the activation parameters of the block are multiplied with the weight parameters of the block to obtain the output matrix; the weight parameters of the block refer to the weight parameters of at least one row in the weight matrix received by the first calculation unit or the second calculation unit, and the row number of the weight parameters of the block is the same as the column number of the received weight parameters.

[0023] In one embodiment, the second processing unit is further used to multiply the output matrix by an activation function, where the activation function is a nonlinear transformation function in the AI ​​model.

[0024] In one embodiment, the second processing unit is specifically used to multiply the weight parameters of different columns by the activation parameters of a row in the activation matrix having the same column number as the weight parameters to obtain activation parameters of multiple columns.

[0025] In one embodiment, the second processing unit is specifically configured to multiply activation parameters of multiple columns by weight parameters of each row of the weight parameters of the block, respectively, to obtain multiple output matrices.

[0026] In one implementation, the third processing unit is further configured to superimpose the multiple output matrices using full reduction calculation.

[0027] In one embodiment, the third processing unit is specifically used to superimpose the elements in the same row and column of the output matrix of the at least one first computing unit and the output matrix of the at least one second computing unit to obtain the output activation matrix.

[0028] In one embodiment, the second processing unit is specifically used to calculate the weight parameters of the column where the outlier is located and the activation matrix using the FP 16 calculation method, and to calculate the weight parameters of the column that does not contain the outlier and the activation matrix using the INT 8 calculation method.

[0029] In a third aspect, an embodiment of the present application provides a computing device, comprising: at least one memory, and at least one processor, wherein the at least one processor is used to execute instructions stored in the at least one memory, so that the computing device can be implemented as each of the possible embodiments of the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed in a computer, the computer is caused to execute various possible implementation embodiments of the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements each possible implementation embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0033] Figure 1 This is a schematic diagram of the architecture of a transformer model;

[0034] Figure 2 Schematic diagram of the process of quantizing reasoning for the multi-head self-attention mechanism of the transformer model;

[0035] Figure 3 A schematic diagram of the multi-head self-attention mechanism of the transformer model provided in an embodiment of the present application performing an outer product operation on a weight matrix and an activation matrix;

[0036] Figure 4 A schematic diagram of rearranging the weight matrices of different encoding blocks or decoding blocks of a transformer model by the inference system provided in an embodiment of the present application;

[0037] Figure 5 A schematic diagram of a flow chart of quantitative reasoning performed by a reasoning system provided in an embodiment of the present application;

[0038] Figure 6 A schematic diagram of an architecture for performing quantitative reasoning in an inference system provided in an embodiment of the present application;

[0039] Figure 7 A schematic diagram of a flow chart of parallel reasoning of outliers by an inference system provided in an embodiment of the present application;

[0040] Figure 8 A schematic diagram of the structure of an inference system provided in an embodiment of the present application;

[0041] Fig. 9 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0042] Fig.10 A schematic diagram of the architecture of a computing device cluster provided in an embodiment of the present application;

[0043] Fig.11 A schematic diagram of the architecture of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0045] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0046] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0047] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0048] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0049] AI models can include transformer models, convolutional neural network (CNN) models, recurrent neural network (RNN) models, self-attention models, etc. The following takes the transformer model as an example.

[0050] It can be imagined that the technical solution protected by this application is mainly to optimize the outer product calculation, which can be implemented not only in the transformer model as an example, but also in models with outer product calculations such as CNN model, RNN model, self-attention model, etc. This application again does not limit it.

[0051] Figure 1 This is a schematic diagram of the architecture of a transformer model. Figure 1 As shown in the figure, the transformer model mainly consists of an encoder and a decoder. The input embedding of the encoder can map each element in the input sequence into a low-dimensional vector representation through the embedding matrix. The positional encoding of the encoder can add a fixed-length vector representation to each position to indicate the relative position information of the position in the sequence.

[0052] The encoder consists of multiple layers of identical encoding blocks. Each encoding block consists of two sub-layers and a residual connection. The first sub-layer is a multi-head self-attention layer. The multi-head self-attention layer can model the relationship between each position in the input sequence through multiple attention heads, find the correlation between each position and other positions, and thus extract the global information of the input sequence.

[0053] The second sublayer is a position-based fully connected network layer, such as a feed-forward neural network (FFN). The fully connected network can be used as a supplement to the multi-head self-attention layer to perform nonlinear transformations on the features of each position, thereby enhancing the representation ability of the input sequence.

[0054] The residual connection can add the input sequence to the output of the multi-head self-attention layer and the fully connected network layer, and after layer normalization, it can be used as the input sequence of the next layer. The output of the last encoding block of the encoder can be a vector of the global information of the input sequence, which can be used as the input sequence of the decoder.

[0055] The target embedding of the decoder can map each element in the target sequence into a low-dimensional vector representation through the embedding matrix. The position encoding of the decoder is the same as that of the encoder, adding a fixed-length vector representation to each position to represent the relative position information of the position in the sequence.

[0056] The decoder consists of multiple layers of identical decoding blocks. Each decoding block consists of three sub-layers and a residual connection. The first sub-layer is a multi-head self-attention mechanism. The multi-head self-attention mechanism is the same as the multi-head self-attention mechanism of the encoder. It uses multiple attention heads to model the relationship between each position in the target sequence and find the correlation between each position and other positions.

[0057] The second sublayer is the multi-head encoder-decoder attention layer. Unlike the multi-head self-attention sublayer, the multi-head encoder-decoder attention layer can calculate the correlation between the current position of the decoder and each position in the encoder's output sequence, thereby incorporating the encoder's global information into the decoder.

[0058] The third sublayer is a position-based fully connected network layer. The fully connected network layer can complement the multi-head encoder-decoder attention layer to perform nonlinear transformations on the features of each position, thereby enhancing the representation ability of the input sequence.

[0059] The residual connection can add the input sequence to the output of the multi-head self-attention layer, the multi-head encoder-decoder attention layer, and the fully connected network layer, and after layer normalization, it is used as the input sequence of the next layer. The output of the last decoding block of the decoder is processed by the softmax function to obtain the predicted sequence.

[0060] With the development of artificial intelligence technology, the accuracy requirements for transformer models are getting higher and higher. The accuracy of transformer models has a certain relationship with the number of parameters and memory. Generally speaking, the more parameters a transformer model has, the stronger the expressive power of the transformer model is, and the accuracy of the transformer model will also increase accordingly. However, the more parameters a transformer model has, the more memory and computing resources the transformer model consumes.

[0061] In order to reduce the calculation complexity of the parameters of the transformer model, the transformer model in the related technology performs quantized reasoning, converting the floating-point model into a fixed-point model to reduce the memory and computing resources of the transformer model. For example, the transformer model converts the weight parameters and activation parameters originally using 32-bit floating-point numbers into 8-bit or 16-bit integers to reduce storage space and memory bandwidth during calculation. Quantized reasoning of the transformer model can not only improve the reasoning speed, but also reduce bandwidth occupancy and energy consumption during network transmission.

[0062] The specific process of transformer model quantization inference is as follows: First, sample the training data, record the distribution of the data, and count the maximum and minimum values ​​of each weight parameter and activation parameter. Secondly, design a suitable quantization scheme based on the data collection and statistical results, including the number of quantization bits, the quantization range, etc. Next, convert the floating-point model to a fixed-point model, and perform training and fine-tuning. Next, evaluate the quantized model, including indicators such as model accuracy, inference speed, and storage size. Finally, deploy the quantized model to the target device and perform inference.

[0063] However, there is a certain proportion of outliers in the weight parameters and activation parameters of the transformer model. Outliers refer to abnormal values ​​that are significantly different from other values, which are often caused by errors or noise during operations such as data collection, transmission or processing. Although the number of outliers is small, the values ​​of the outliers may be relatively large, which will affect the accuracy of the transformer model. When the transformer model performs quantitative inference, the impact of outliers on the accuracy of the transformer model is amplified, resulting in a serious decrease in the accuracy of the transformer model. Moreover, the larger the scale of the transformer model, the greater the impact of outliers on the accuracy of the transformer model.

[0064] In order to solve the problem of reduced accuracy of transformer models caused by existing quantization schemes, an embodiment of the present application provides a quantization reasoning method, reasoning system, computing device and computing device cluster for an AI model.

[0065] Weight parameters are the weights connecting different neurons in a neural network, which are used to transmit information and calculate output. In the transformer model, weight parameters exist in the form of a matrix. The weight matrix is ​​usually a network structure composed of multiple self-attention layers and fully connected layers, which is used to encode and decode input sequences. A weight matrix refers to the weights connecting different neurons in a neural network running a multi-head self-attention mechanism in the transformer model.

[0066] The activation parameters exist in the form of a matrix in the transformer model. The activation matrix refers to the output matrix of each layer in the neural network. When the input sequence undergoes a linear transformation at a certain layer, a new feature matrix will be obtained, which is the activation matrix. Each element in the activation matrix represents the output result of each neuron in the neural network. In the neural network, each neuron will undergo a linear transformation based on the input sequence, and a nonlinear transformation through the activation function, and output a scalar value, which is the value of each element in the activation matrix.

[0067] Transformer models are typically run by an inference system. An inference system is a combination of software programs and hardware that runs the transformer model.

[0068] The transformer model includes a large number of weight parameters and a large number of activation parameters. The transformer model can multiply the weight matrix with the activation matrix to obtain a new activation matrix to achieve quantitative reasoning of the weight matrix and the activation matrix, as well as compression of the transformer model. Since both the weight parameters and the activation parameters may have outliers, the reasoning system can reorder the weight matrix. For example, the reasoning system arranges the column where the outliers are located in the weight matrix at the head of the weight matrix. When the reasoning system performs an outer product operation on the transformer model, the weight parameters of the column where the outliers are located in the weight matrix are centrally distributed to one or more computing units, and the weight parameters of the column where the outliers are located in the weight matrix are distributed to other computing units, which is conducive to batch processing of outliers and reduces the overhead of processing outliers when other computing units perform quantitative reasoning. The reasoning system processes the weight parameters of the column where the outliers are located separately from the weight parameters of the column where the outliers are located, and can use different data formats for processing, so as to achieve efficient compression while ensuring accuracy. The inference system can use the allreduce algorithm to superimpose the calculation results of different computing units into a matrix. This process does not require the use of INT 8 to sum the elements of each column of the weight matrix with the elements of each row of the activation matrix. It can reduce the data transfer between the cube and the vector to reduce the latency of the transformer model.

[0069] The transformer model usually uses a multi-head self-attention mechanism to implement the input and output of the encoder and decoder. In the multi-head self-attention mechanism, the transformer model uses operations such as inner product, outer product, and allreduce for quantitative reasoning.

[0070] The inner product of a matrix refers to multiplying each element of the i-th row of the first matrix with each element of the j-th column of the second matrix, and superimposing each product to obtain an element of the i-th row and j-th column. Both i and j are positive integers greater than or equal to 1. In an embodiment of the present application, the transformer model obtains the query (query, Q), key (key, K) and value (value, V) of the multi-head self-attention mechanism, and performs an inner product operation on the query Q, key K and value V to calculate the weight parameter of attention. The inner product operation can effectively calculate the similarity between the query Q and the key K, thereby determining the attention weight of each query Q for the key K.

[0071] In one embodiment, Figure 2 As shown, the first matrix A is The second matrix B is In the process of calculating the inner product of the matrix by the transformer model, each element of the first row of matrix A can be multiplied with each element of the first column of matrix B, and the products are superimposed to obtain an element of the first row and first column of the new matrix C, which is 1×1+2×3+3×5=22.

[0072] The outer product of a matrix refers to calculating the outer product of each element of each column of the first matrix with each element of a row of the second matrix with the same column number as the first matrix, and superimposing the products of the same row and column to obtain a new matrix. In an embodiment of the present application, the inference system performs an outer product operation on the attention weight and the value V to calculate a weighted sum. The outer product operation can perform a weighted summation of the attention weight and the value V to obtain the final output.

[0073] In one embodiment, Figure 2 As shown in the figure, when the transformer model calculates the outer product of the matrix, it multiplies the first column of the first matrix by the first row of the second matrix to obtain the first partial matrix C1. The elements of the partial matrix C1 are: 1×1=1 for the first row and first column, 1×2=2 for the first row and second column, 4×1=4 for the second row and first column, and 4×2=8 for the second row and second column.

[0074] The transformer model multiplies the second column of the first matrix by the second row of the second matrix to obtain the second partial matrix C2. The elements of the partial matrix C2 are: 2×3=6 in the first row and first column, 2×4=8 in the first row and second column, 5×3=15 in the second row and first column, and 5×4=20 in the second row and second column.

[0075] The transformer model multiplies the third column of the first matrix by the third row of the second matrix to obtain the third partial matrix C3. The elements of the partial matrix C3 are: 3×5=15 in the first row and first column, 3×6=18 in the first row and second column, 6×5=30 in the second row and first column, and 6×6=36 in the second row and second column.

[0076] Allreduce means merging the target arrays of all processes into a single array and returning the result array to the operators of all processes. In an embodiment of the present application, the inference system uses the allreduce operation to merge the outputs of multiple heads of the transformer model. The allreduce operation can aggregate and average the results of different nodes. It does not need to use INT 8 to multiply the elements of each column of the weight matrix by the elements of each row of the activation matrix, which can reduce the data transfer between cube roots and vectors, thereby improving computational efficiency and model accuracy.

[0077] In one embodiment, Figure 2 As shown in the figure, after the transformer model obtains partial matrix C1, partial matrix C2 and partial matrix C3, it uses the allreduce algorithm to superimpose the elements of the same row and the same column of each partial matrix to obtain the complete matrix C. The elements of matrix C are: 1+6+15=22 in the first row and first column, 2+8+18=28 in the first row and second column, 4+15+30=49 in the second row and first column, and 8+20+36=64 in the second row and second column.

[0078] The weighted sum calculated by the inference system using the outer product operation is related to the weight parameters and activation parameters. Based on the calculation results of the attention weight, the inference system linearly transforms the value V to obtain a new query Q and key K. The inference system performs a dot product operation on the linearly transformed query Q and key K to obtain the weight of the key K corresponding to each query Q, and then multiplies the weight by the value V to obtain the weighted sum corresponding to each query. The inference system merges the weighted sum results of multiple heads to obtain the final output. In the merging process, the weight parameters and activation parameters are mainly used for the calculation of linear transformation. Specifically, the inference system uses the weight matrix to linearly transform the value V, and uses the activation matrix to nonlinearly transform the result to obtain a new query Q and a new key K.

[0079] The first matrix refers to the weight matrix that connects different neurons in the neural network inside the transformer model. The second matrix refers to the activation matrix input to the transformer model. Both weight parameters and activation parameters may have outliers. If outliers are randomly distributed in different rows and columns of the weight matrix, when the inference system directly multiplies the weight matrix by the activation matrix, the error in calculating the new activation matrix is ​​relatively large, resulting in poor accuracy in quantization of the transformer model.

[0080] In the embodiment of the present application, the inference system can reorder the weight matrix so that the weight parameter of the outlier is at the head of all the weight parameters. The inference system can assign the weight parameter of the column where the outlier is located to one or a few computing units. Each computing unit can run an outer product calculation engine to calculate the outer product of each element of a column of the first matrix with each element of a row of the second matrix.

[0081] A computing unit refers to the smallest computing unit of the hardware corresponding to the inference system running the transformer model. For example, when the inference system can be a device composed of multiple processors, the computing unit can be a processor. For another example, when the inference system is a device composed of one processor, the computing unit can refer to part of the computing resources of the processor running a computing process. For another example, when the inference system is composed of multiple devices connected to a network, the computing unit can refer to a single device.

[0082] The number of columns of the weight matrix that can be run by each computing unit is limited. The reasoning system can use some of the multiple computing units as computing units for calculating the weight parameters of the columns where the outliers are located, and refer to them as "first computing units". The reasoning system can use another part of the multiple computing units as computing units for calculating the weight parameters of the columns that do not contain the outliers, and refer to them as "second computing units". The reasoning system can assign a corresponding number of column weight parameters to each computing unit according to the set threshold of each computing unit.

[0083] Figure 3 This is a schematic diagram of the multi-head self-attention mechanism of the transformer model provided in the embodiment of the present application performing an outer product operation on the weight matrix and the activation matrix. Figure 3 As shown, the weight matrix A is divided into multiple blocks according to the columns, namely, block R0, block R1, ..., block Rn. Correspondingly, the activation matrix B is divided into multiple blocks according to the rows, namely, block R0', block R1', ..., block Rn'. When the computing unit uses the outer product computing engine to perform the outer product operation, the weight parameters of each column of the weight matrix A are multiplied by the activation parameters of each row of the activation matrix B to obtain the activation parameters of multiple columns to form a new activation matrix. Among them, when the computing unit receives the weight parameters of the column where the outlier is located, the computing unit can use the FP 16 calculation method to perform the outer product operation on the weight matrix and the activation matrix. When the computing unit receives the weight parameters of the column that does not contain the outlier, the computing unit can use the INT 8 calculation method to perform the outer product operation on the weight matrix and the activation matrix.

[0084] When the transformer model assigns weight parameters, the weight parameters of the column where the outliers are located can be assigned to the same computing unit. The computing unit calculates the outer product of each element of a column of the weight parameters of the column where the outliers are located with each element of a row of the activation parameters of the outliers to obtain a partial matrix of the outliers. The transformer model can centrally assign the weight parameters of the column where the outliers are located to one computing unit, and assign the weight parameters of the column where the outliers are not included in the weight matrix to other computing units, which is conducive to batch processing of outliers and reduces the overhead of processing outliers when other computing units perform quantitative reasoning. The transformer model can assign the weight parameters of the column where the outliers are located and the weight parameters of the column where the outliers are not included to different computing units, allowing the computing units to use different data formats for processing and achieve efficient compression while ensuring accuracy.

[0085] In an embodiment of the present application, the transformer model includes multiple decoding blocks and multiple encoding blocks, each decoding block runs a multi-head self-attention mechanism, and each encoding block runs a multi-head self-attention mechanism. When each multi-head self-attention mechanism performs quantitative reasoning on weight parameters and activation parameters, the weight parameters of the outliers can be set to different column numbers of the weight matrix based on the serial numbers of the multiple computing units running. Each multi-head self-attention mechanism assigns the weight parameter columns of the outliers to different computing units. Different computing units switch with each other in different multi-head self-attention mechanisms, and can receive the weight parameters of the columns where the outliers are located, avoiding a single computing unit from constantly performing quantitative reasoning on the weight parameters of the outliers to achieve load balancing.

[0086] like Figure 4 As shown in the figure, the number of computing units of the multi-head self-attention mechanism running the transformer model is m. The number of columns of the weight matrix is ​​n. Both m and n are positive integers greater than or equal to 1. The multi-head self-attention mechanism can split the weight parameters of the 1st to n1th columns of the weight matrix into a block R1. Block R1 is assigned to the computing unit numbered "1". The multi-head self-attention mechanism can split the weight parameters of the n2th to n3th columns of the weight matrix into a block R2. Block R2 is assigned to the computing unit numbered "2". Similarly, the multi-head self-attention mechanism can split the weight parameters of the nkth to mth columns of the weight matrix into a block Rn. Block Rn is assigned to the computing unit numbered "n".

[0087] For the first encoding block of the transformer model, the multi-head self-attention mechanism rearranges the weight parameters of the outlier columns into the 1st to n1th columns of the weight matrix as block R1. For the second encoding block of the transformer model, the multi-head self-attention mechanism rearranges the weight parameters of the outlier columns into the n2th to n3th columns of the weight matrix, and rearranges the n2th to n3th columns of the weight matrix to the head of the weight matrix as block R2. And so on.

[0088] Similarly, for the first decoding block of the transformer model, the multi-head self-attention mechanism rearranges the weight parameters of the outlier columns to the 1st to n1th columns of the weight matrix as block R1. For the second decoding block of the transformer model, the multi-head self-attention mechanism rearranges the outlier weight parameters to the n2th to n3th columns of the weight matrix, and rearranges the n2th to n3th columns of the weight matrix to the head of the weight matrix as block R2. And so on.

[0089] Figure 5 FIG. 1 is a flow chart of quantitative reasoning performed by a reasoning system provided in an embodiment of the present application. Figure 5 As shown in the figure, when the transformer model performs an outer product operation, the process of quantitative reasoning by the inference system is as follows:

[0090] Step S501: the inference system obtains a weight matrix.

[0091] In step S502, the inference system distributes the weight parameters of the column where the outliers in the weight matrix are located to at least one first computing unit, and distributes the weight parameters that do not contain outliers to at least one second computing unit by column.

[0092] In the transformer model, the weight parameters calculated by the multi-head self-attention mechanism of a decoding block and an encoding block are in the form of a matrix. The multi-head self-attention mechanism can assign the elements of one or several columns of the weight matrix to different computing units to achieve quantitative reasoning in parallel for multiple computing units.

[0093] In the embodiment of the present application, there may be outliers in the weight parameters corresponding to each element of the weight matrix. The multi-head self-attention mechanism can extract the weight parameters of the column where the outliers are located from the weight matrix, and arrange the weight parameters of the column where the outliers are located at the head of the weight matrix to rearrange the weight matrix. When allocating weight parameters, the multi-head self-attention mechanism can allocate the weight parameters of the column where the outliers are located to the same computing unit, or to several computing units, and allocate the weight parameters of the column where the outliers are located in the weight matrix to other computing units, which is conducive to batch processing of outliers and reduces the overhead of processing outliers when other computing units perform quantitative reasoning. The multi-head self-attention mechanism processes the weight parameters of the column where the outliers are located separately from the weight parameters of the column where the outliers are located, and can use different data formats for processing, achieving efficient compression while ensuring accuracy. Figure 6 As shown, the inference system includes n computing units. The inference system divides the weight matrix into n blocks, and assigns block R1 to computing unit 1, assigns block R2 to computing unit 2, and so on.

[0094] The reasoning system can be divided according to the computing resources of the running hardware to obtain multiple computing units. Each computing unit can run an outer product operation. In one embodiment, when the reasoning system is a device composed of multiple processors, the computing unit can be a single processor. In another embodiment, when the reasoning system is a device composed of one processor, the computing unit can refer to a portion of the computing resources of the processor running a computing process. In other embodiments, when the reasoning system is composed of multiple devices connected to a network, the computing unit can refer to a single device.

[0095] The inference system includes a plurality of computing units. The inference system can be divided into a first computing unit and a second computing unit according to the type of weight parameter received. The first computing unit is used to receive the weight parameter of the column of the outlier, and perform outer product calculation on the weight parameter of the column where the outlier is located and the activation matrix. The second computing unit is used to receive the weight parameter of the column that does not contain the outlier, and perform outer product calculation on the weight parameter of the column that does not contain the outlier and the activation matrix.

[0096] In one possible embodiment, the first computing unit and the second computing unit switch with each other in different layers of the multi-head self-attention mechanism of the neural network, so that all computing units can receive the weight parameters of the column where the outliers are located, avoiding a single computing unit from constantly performing quantitative inference on the weight parameters of the outliers, so as to achieve load balancing.

[0097] In step S503, the inference system performs a first type of quantization calculation on the weight parameters in at least one first calculation unit and the activation matrix, and performs a second type of quantization calculation on the weight parameters in at least one second calculation unit and the activation matrix.

[0098] The first type of quantization calculation can be FP 16 calculation. The second type of quantization calculation can be INT 8. INT 8 and FP16 are common quantization data types used in neural network calculations. Compared with traditional single-precision floating-point numbers (FP32), INT 8 and FP 16 have advantages such as high computing efficiency and small model size. FP 16 refers to a half-precision floating-point number type, which is used to represent weights and activation values ​​in neural networks. The value range of FP 16 is wider than that of integer types of the same length, but its precision is lower than that of single-precision floating-point numbers. FP 16 calculations have higher precision than INT 8 calculations, but require more storage space and computing resources.

[0099] In an embodiment of the present application, the computing unit calculates the weight parameters and activation matrix of the column where the outliers are located using the FP 16 calculation method. Exemplarily, in the process of using FP 16 calculation, the computing unit needs to first quantize the weight parameters and activation matrix of the column where the outliers are located, and convert the weight parameters and activation matrix of the column where the outliers are located to FP 16. The computing unit uses direct operation functions of frameworks such as TensorFlow and PyTorch, such as the tf.float16 data type in TensorFlow. The computing unit uses FP 16 for matrix multiplication. After the computing unit completes the calculation, it is necessary to convert the matrix result of the FP 16 type back to the FP32 type. For example, the tf.cast function provided by TensorFlow can perform type conversion.

[0100] INT 8 refers to an 8-bit integer type, which is used to represent model parameters such as weights, activation values, and gradients. The value range of INT 8 is -128 to +127. In the INT 8 calculation process, the weights and activation values ​​are first quantized, truncated, and scaled, and then the quantized integers are used for matrix multiplication, convolution operations, etc., and assembly code and hardware instructions can be used to implement the calculation. Therefore, the speed of INT 8 calculation is very fast and the storage space occupied is relatively small. However, the accuracy of INT 8 calculation is low, which may cause a certain loss of accuracy in model training and reasoning.

[0101] In the embodiment of the present application, the calculation unit uses the INT 8 calculation method to calculate the weight parameters and activation matrix of the column where the outlier is not included. Exemplarily, when using the INT 8 calculation process, the calculation unit needs to first quantize the weight parameters and activation matrix of the column where the outlier is not included, so that the weight parameters and activation matrix of the column where the outlier is not included can be calculated using 8-bit integers. The specific quantization process of the calculation unit is as follows:

[0102] First, the calculation unit truncates the weight matrix and the activation matrix. For 8-bit numbers, the integer value range is -128 to +127, so the calculation unit needs to scale the weight parameters and activation parameters to within this range. Then, the calculation unit needs to scale the truncated weight parameters and activation parameters. Next, the calculation unit rounds the scaled values ​​to the nearest integer. This process requires mapping decimals to integer values. Finally, after the calculation unit completes the quantization conversion, the quantized matrix can be used for matrix multiplication. When calculating the weight matrix and the activation matrix, the calculation unit uses the quantized matrix for matrix multiplication, and the result is also a quantized 8-bit integer matrix.

[0103] After the multi-head self-attention mechanism assigns the quantized weight parameters of different columns of the weight matrix to different computing units, it is necessary to assign the quantized activation matrix to different computing units respectively. After obtaining the weight parameters and the activation matrix, the computing unit multiplies the weight parameters of different column numbers with the activation parameters of the corresponding rows of the activation matrix to obtain the activation parameters of the blocks. Exemplarily, the multi-head self-attention mechanism assigns the weight parameters of columns 1-3 of the weight matrix to the first computing unit. After the first computing unit receives the weight parameters and the activation matrix of columns 1-3, it multiplies the weight parameters of the first column with the activation parameters of the first row of the activation matrix to obtain the activation parameters of one column. Similarly, the first computing unit multiplies the weight parameters of the second column with the activation parameters of the second row of the activation matrix to obtain the activation parameters of one column. The first computing unit multiplies the weight parameters of the third column with the activation parameters of the third row of the activation matrix to obtain the activation parameters of one column. As Figure 6 As shown, the number of blocks of activation parameters calculated by the inference system is the same as the number of blocks of the weight matrix. Among them, the activation parameter R1' of the block is obtained by multiplying the block R1 with the activation matrix. The activation parameter R2' of the block is obtained by multiplying the block R2 with the activation matrix. And so on.

[0104] In an embodiment of the present application, in an embodiment of the present application, the computing unit uses the FP 16 computing method to perform quantized calculations on the weight parameters of the column where the outliers are located, and uses the INT 8 computing method to perform quantized calculations on the weight parameters of the column where the outliers are not located, so as to separate the weight parameters of the column where the outliers are located from the weight parameters of the column where the outliers are located, and different data formats can be used for quantized processing to achieve efficient compression while ensuring accuracy.

[0105] The weight parameters received by the calculation unit are multiplied by the full activation matrix to obtain the block activation, which can reduce the amount of calculation and memory usage. This method is usually used in scenarios with limited computing resources, such as mobile terminals and embedded devices. In these scenarios, due to limited computing resources and memory capacity, if the full amount of data is directly calculated, it may lead to problems such as long calculation time or memory overflow.

[0106] The weight parameters of the block refer to the weight matrix of some rows assigned to each calculation unit. The activation parameters of the block are the activation matrices of some columns calculated by the calculation unit. After the calculation unit calculates the activation parameters of the block, it can obtain the weight parameters of the corresponding row numbers of the weight matrix based on the number of columns of the activation parameters of the block. The calculation unit multiplies the activation parameters of each column by the weight parameters of each row. Figure 6 As shown, the number of activation parameters of the blocks is n. The inference system can divide the weight matrix into n blocks of weight parameters in the order of rows, namely S1, S2, ..., Sn. Among them, the row number of the weight parameters of the blocks assigned to the same computing unit by the inference system is the same as the column number of the activation parameters of the blocks.

[0107] In one case, when the computing unit receives a row of weight parameters, it can multiply each activation parameter of a column with each weight parameter of a row, and superimpose the products to obtain an X×Y matrix. Among them, X represents the number of rows of the matrix, and X is equal to the number of weight parameters. Y represents the number of columns of the matrix, and Y is equal to the number of activation parameters.

[0108] In another case, when the computing unit receives k rows of weight parameters, it can multiply each activation parameter of the same column number with each weight parameter of the row, and superimpose the products to obtain k X×Y matrices. Optionally, after the computing unit obtains multiple matrices, it can use the allreduce operation to parallelize the calculation process, superimpose k X×Y matrices, and superimpose the elements of the same row and the same column of each matrix to obtain an X×Y matrix. After the computing unit superimposes multiple matrices, it outputs a matrix, which can reduce the amount of output data and thus improve computing efficiency.

[0109] In an embodiment of the present application, each eigenvector of the activation matrix is ​​multiplied by the weight matrix to obtain a new eigenvector. The new eigenvector is linearly transformed and can capture the relationship between the features of the input sequence. The activation matrix can perform weighted summation on the new eigenvector to obtain the final output. The weighted summation process is an attention mechanism that enables the transformer model to pay more attention to the features in the input sequence that are relevant to the current task. Since the product of the weight matrix and the activation matrix is ​​differentiable, the back-propagation algorithm can be used to train the transformer model so that the transformer model can automatically learn the relationship between the features of the input sequence and use it for downstream task processing.

[0110] The output matrix Cn refers to the X×Y matrix obtained by multiplying and superimposing the weight parameters of the blocks and the activation parameters of the blocks. The activation function is a nonlinear transformation function in the neural network, which is used to increase the expression ability and nonlinear feature extraction ability of the model. In the transformer model, the activation function can be a rectified linear unit (ReLU), a Gaussian error linear unit (GELU), and other functions.

[0111] In an embodiment of the present application, the computing unit multiplies the output matrix Cn by the activation function, and the linear transformation and nonlinear change can be combined to enhance the expressive power of the transformer model. Generally, the linear transformation of the activation matrix can extract the relationship between the features of the input sequence, and the nonlinear transformation of the activation function can map the relationship between the features of the input sequence to a higher dimensional space, thereby better capturing the complex relationship between the input sequences. The combination of linear transformation and nonlinear change can effectively reduce the overfitting degree of the transformer model to improve the generalization ability of the transformer model. At the same time, the computing unit uses the activation function to limit the output of the transformer model to a certain range. For example, the sigmoid function can limit the output to between 0 and 1 to avoid excessive or small output.

[0112] Step S504: the inference system superimposes the output matrix of at least one first computing unit and the output matrix of at least one second computing unit to obtain an output activation matrix.

[0113] The reasoning system can use the allreduce operation to superimpose the output matrices of the first computing unit and the second computing unit. In the parallel computing process, the reasoning system can superimpose the updated output matrices calculated by multiple computing units, so that the elements of the same rows and columns of each partial matrix are superimposed to obtain a complete activation matrix C. The complete activation matrix C is also the output activation matrix.

[0114] In an embodiment of the present application, when the inference system performs an outer product operation on the transformer model, it distributes, compresses, aggregates and communicates weight parameters and activation parameters, thereby reducing the memory occupied by the transformer model and the bandwidth for accessing the memory while maintaining the high precision of the transformer model.

[0115] Figure 7 FIG. 1 is a flow chart of a parallel reasoning process of an inference system for outliers provided in an embodiment of the present application. Figure 7 As shown in the figure, when the transformer model performs an inner product operation, the parallel reasoning process of the reasoning system is as follows:

[0116] Step S701: The inference system obtains inference data, including weight parameters of the neural network, structural parameters of the transformer model, data sets, etc.

[0117] The inference system can use the save and load functions provided by PyTorch to save and load the weight parameters of the transformer model. The inference system can use the print function or summary function provided by PyTorch to obtain the structural parameters of the model. For example, the number of encoding blocks of the encoder, the multi-head self-attention layer and FFN layer of the encoding block, the number of decoding blocks of the decoder, the multi-head self-attention layer of the decoding block, the multi-head encoder-decoder attention layer and FFN layer, etc.

[0118] The reasoning system can use the Dataset and DataLoader classes provided by PyTorch to obtain datasets. For example, when the transformer model is applied to the workshop on machine translation (WMT), the dataset refers to the translation task dataset from English to other languages, including English to German, French, Chinese and other languages. For another example, when the transformer model is applied to the general language understanding evaluation (GLUE), the dataset refers to a wide range of datasets used to evaluate natural language understanding models, including 9 different tasks such as sentiment analysis, natural language reasoning, etc.

[0119] In step S702, the inference system records the index information of the outliers in the activation parameters in the channel, and reorders the weight parameters based on the index information.

[0120] The inference system records the index information of the channel where the outlier in the activation parameter is located. The corresponding code needs to be added to the model to implement this function, specifically:

[0121] First, in the Forward Pass, the inference system obtains the output tensor of the activation function. Assume that the shape of the output tensor is Batch_size, Seq_length, and Hidden_size. Secondly, for each Batch, the inference system uses the argmax function to find the K values ​​in the output tensor that are greater than the set threshold. Assume that the subscripts of the K values ​​are idx_1, idx_2,..., idx_K. Then, the inference system records the subscripts of the K values. For example, the inference system can use a dictionary to save the subscripts of the K values ​​and correspond to the index information of the Batch. Finally, the inference system can use the subscripts of the K values ​​to obtain the index information of the channel where the outlier is located.

[0122] Step S703: the inference system distributes weight parameters to different computing units.

[0123] In the transformer model, the weight parameters calculated by the multi-head self-attention mechanism of a decoding block and an encoding block are in the form of a matrix. The multi-head self-attention mechanism can assign the elements of one or several columns of the weight matrix to different computing units to achieve quantitative reasoning in parallel for multiple computing units.

[0124] In the embodiment of the present application, there may be outliers in the weight parameters corresponding to each element of the weight matrix. The multi-head self-attention mechanism can extract the weight parameters of the column where the outliers are located from the weight matrix, and arrange the weight parameters of the column where the outliers are located at the head of the weight matrix to rearrange the weight matrix. When allocating weight parameters, the multi-head self-attention mechanism can allocate the weight parameters of the column where the outliers are located to the same calculation unit, or to several calculation units, and allocate the weight parameters of the column where the outliers are located in the weight matrix to other calculation units, which is conducive to batch processing of outliers and reduces the overhead caused by processing outliers when other calculation units perform quantitative reasoning. The multi-head self-attention mechanism processes the weight parameters of the column where the outliers are located separately from the weight parameters of the column where the outliers are located, and can use different data formats for processing, and achieve efficient compression while ensuring accuracy.

[0125] In step S704, when the inference system performs an inner product operation on the transformer model, the weight parameters of the column where the outlier is located are calculated using FP16, and the weight parameters of the column that does not contain the outlier are calculated using INT8.

[0126] In the transformer model, the inner product calculation part is the core part of the multi-head self-attention mechanism, and it is also one of the parts with the largest amount of model calculation. In order to improve the computational efficiency of the transformer model and reduce the computational cost, the reasoning system can use mixed precision computing technology to optimize the calculation of the model. In the embodiment of the present application, when the computing unit is assigned with the weight parameter of the column where the outlier is located, the computing unit uses FP 16 (half precision) to calculate the inner product calculation part of the weight parameter of the column where the outlier is located. When the computing unit is assigned with the weight parameter of the column where the outlier is located, the computing unit uses INT 8 (8-bit integer) to calculate the inner product calculation part of the weight parameter of the column where the outlier is located.

[0127] FP16 calculations can reduce memory bandwidth during storage and transmission, and speed up matrix operations and activation parameter calculations. INT8 can use smaller memory and fewer computing resources in transformer model calculations while maintaining high computing accuracy. Therefore, both FP16 calculations and INT8 calculations are faster than FP32 (single precision) calculations, but FP16 calculations are less accurate than FP32 calculations, and INT8 calculations are less accurate than FP32 calculations. By adopting FP16 and INT8 mixed precision computing technology, the inference system can significantly improve the computing speed and efficiency of the transformer model while maintaining the inference accuracy of the transformer model.

[0128] In the embodiment of the present application, when the inference system performs an inner product operation on the transformer model, the weight parameters of the column where the outlier is located are calculated using FP 16, and the weight parameters of the column where the outlier is not located are calculated using INT 8. The inference system can improve the calculation speed and efficiency of the transformer model while maintaining high inference accuracy by adopting FP 16 and INT 8 mixed precision calculation technology.

[0129] Figure 8 FIG. 1 is a schematic diagram of a structure of an inference system provided in an embodiment of the present application. Figure 8 As shown, the reasoning system 800 can be divided into a first processing unit 810, a second processing unit 820 and a third processing unit 830 according to the execution function. The specific implementation process of the reasoning system 800 is as follows:

[0130] The first processing unit 810 is used to obtain a weight matrix. The weight matrix is ​​a matrix composed of weights connecting different neurons in the AI ​​model. The first processing unit 810 is also used to assign the weight parameters of the column where the outliers in the weight matrix are located to at least one first calculation unit, and to assign the weight parameters that do not contain outliers to at least one second calculation unit by column. The second processing unit 820 is used to perform a first type of quantization calculation on the weight parameters in at least one first calculation unit and the activation matrix, and to perform a second type of quantization calculation on the weight parameters in at least one second calculation unit and the activation matrix. Among them, the activation matrix refers to the characteristic matrix obtained after the AI ​​model performs a linear transformation on the input sequence. The third processing unit 830 is used to superimpose the output matrix of at least one first calculation unit with the output matrix of at least one second calculation unit to obtain an output activation matrix.

[0131] In one possible embodiment, the first processing unit 810 is specifically used to extract the weight parameters of the columns where the outliers in the weight matrix are located, and set the weight parameters of the columns where the outliers are located at the head of the weight matrix; and based on the set threshold of each first calculation unit, allocate the weight parameters of the columns where the outliers of a corresponding number of columns to each first calculation unit.

[0132] In a possible implementation, the second processing unit 820 is specifically used to multiply the received weight parameters with the activation matrix to obtain the activation parameters of the block. The number of columns of the activation parameters of the block is the same as the number of columns of the received weight parameters. The second processing unit 820 is specifically used to multiply the activation parameters of the block with the weight parameters of the block to obtain an output matrix. The weight parameters of the block refer to the weight parameters of at least one row in the weight matrix received by the first calculation unit or the second calculation unit. The row number of the weight parameters of the block is the same as the column number of the received weight parameters.

[0133] In a possible implementation, the second processing unit 820 is further configured to multiply the output matrix by an activation function. The activation function is a nonlinear transformation function in the AI ​​model.

[0134] In a possible implementation embodiment, the second processing unit 820 is specifically configured to multiply the weight parameters of different columns by the activation parameters of a row in the activation matrix having the same column number as the weight parameters to obtain activation parameters of multiple columns.

[0135] In a possible implementation embodiment, the second processing unit 820 is specifically configured to multiply activation parameters of multiple columns by weight parameters of each row of the weight parameters of the block, respectively, to obtain multiple output matrices.

[0136] In a possible implementation embodiment, the third processing unit 830 is used to superimpose the multiple output matrices using allreduce calculation.

[0137] In one possible implementation, the third processing unit 830 is specifically configured to superimpose the elements in the same row and column of an output matrix of at least one first computing unit and an output matrix of at least one second computing unit to obtain an output activation matrix.

[0138] In one possible embodiment, the second processing unit 820 is specifically used to calculate the weight parameters and activation matrix of the column where the outliers are located using the FP 16 calculation method, and to calculate the weight parameters and activation matrix of the column that does not contain the outliers using the INT 8 calculation method.

[0139] Fig. 9 Schematic diagram of a computing device provided in an embodiment of the present application. Fig. 9 As shown, the computing device 900 includes a bus 910, a processor 920, a memory 930, and a communication interface 940. The processor 920, the memory 930, and the communication interface 940 communicate with each other through the bus 910. The computing device 900 can be a server, a computer, a portable notebook, a cabinet, etc. It should be understood that the present application does not limit the number of processors and memories in the computing device 900.

[0140] The bus 910 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9 The bus 910 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 910 may include a path for transmitting information between various components of the computing device 900 (for example, the processor 920, the memory 930, and the communication interface 940).

[0141] The processor 920 may be the above-mentioned computing unit, and may be any one or more of processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0142] The memory 930 may include a volatile memory, such as a random access memory (RAM). The memory 930 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0143] The memory 930 stores executable program codes, and the processor 920 executes the executable program codes to respectively implement the functions of the aforementioned multiple computing units (e.g., computing unit 930-1, computing unit 930-2, ..., computing unit 930-n), thereby implementing the quantitative reasoning method. That is, the memory 930 stores instructions for executing the quantitative reasoning method.

[0144] Alternatively, the memory 930 stores executable codes, and the processor 920 executes the executable codes to respectively implement the functions of the aforementioned computing units, thereby implementing the quantitative reasoning method. That is, the memory 930 stores instructions for executing the quantitative reasoning method.

[0145] The communication interface 940 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 900 and other devices or a communication network.

[0146] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0147] like Fig.10 As shown, the computing device cluster includes at least one computing device 900. The memory 930 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the quantized reasoning method.

[0148] In some possible implementations, the memory 930 of one or more computing devices 900 in the computing device cluster may also store some instructions for executing the quantitative reasoning method. In other words, the combination of one or more computing devices 100 can jointly execute the instructions for executing the quantitative reasoning method.

[0149] It should be noted that the memory 930 in different computing devices 900 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the computing units. That is, the instructions stored in the memory 930 in different computing devices 900 can implement the functions of one or more modules in multiple computing units.

[0150] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.11 A possible implementation is shown. Fig.11 As shown, two computing devices are computing device 900A and computing device 900B respectively, which are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 930 in the computing device 900A stores instructions for executing the functions of some computing units in the multiple computing units. At the same time, the memory 930 in the computing device 900B stores instructions for executing the functions of another part of the computing units in the multiple computing units.

[0151] Fig.11 The connection method between the computing device clusters shown may be based on the consideration that the quantitative reasoning method provided in the present application requires a large amount of data storage, and therefore the functions implemented by another part of the multiple computing units may be handed over to the computing device 900B for execution.

[0152] It should be understood that Fig.11 The functions of the computing device 900A shown in FIG. 9A may also be completed by multiple computing devices 900. Similarly, the functions of the computing device 900B may also be completed by multiple computing devices 900.

[0153] The present application embodiment also provides another computing device cluster. The connection relationship between the computing devices in the computing device cluster can be similar to that of Fig. 9 and Fig.10 The connection mode of the computing device cluster is different in that the memory 930 in one or more computing devices 900 in the computing device cluster may store the same instructions for executing the quantitative reasoning method.

[0154] In some possible implementations, the memory 930 of one or more computing devices 900 in the computing device cluster may also store some instructions for executing the quantitative reasoning method. In other words, the combination of one or more computing devices 900 can jointly execute the instructions for executing the quantitative reasoning method.

[0155] It should be noted that the memory 930 in different computing devices 900 in the computing device cluster may store different instructions for executing partial functions of the computing device 900. That is, the instructions stored in the memory 930 in different computing devices 900 may implement the functions of one or more devices in the computing unit.

[0156] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device performs a quantitative reasoning method.

[0157] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to perform a quantitative reasoning method.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A quantitative reasoning method for an artificial intelligence (AI) model, characterized in that: The method is performed by an inference system, and the method includes: Obtain a weight matrix, where the weight matrix is ​​a matrix composed of weights connecting different neurons in the AI ​​model; Distribute the weight parameters of the column where the outliers in the weight matrix are located to at least one first computing unit, and distribute the weight parameters that do not contain outliers to at least one second computing unit by column; Performing a first type of quantization calculation on the weight parameters in the at least one first calculation unit and the activation matrix, and performing a second type of quantization calculation on the weight parameters in the at least one second calculation unit and the activation matrix; wherein the activation matrix refers to a feature matrix obtained after the AI ​​model performs a linear transformation on the input sequence; The output matrix of the at least one first computing unit is superimposed with the output matrix of the at least one second computing unit to obtain an output activation matrix.

2. The method according to claim 1, characterized in that The step of allocating the weight parameters of the column where the outliers in the weight matrix are located to at least one first calculation unit specifically includes: Extracting the weight parameter of the column where the outlier value is located in the weight matrix, and setting the weight parameter of the column where the outlier value is located at the head of the weight matrix; Based on the set threshold of each first computing unit, each first computing unit is allocated weight parameters of the columns where the outliers of a corresponding number of columns are located.

3. The method according to claim 1 or 2, characterized in that: The performing a first type of quantization calculation on the weight parameters in the at least one first calculation unit and the activation matrix, and performing a second type of quantization calculation on the weight parameters in the at least one second calculation unit and the activation matrix, specifically includes: Multiplying the received weight parameters by the activation matrix to obtain activation parameters of the blocks; the number of columns of the activation parameters of the blocks is the same as the number of columns of the received weight parameters; The activation parameter of the block is multiplied by the weight parameter of the block to obtain the output matrix; the weight parameter of the block refers to the weight parameter of at least one row in the weight matrix received by the first calculation unit or the second calculation unit, and the row number of the weight parameter of the block is the same as the column number of the received weight parameter.

4. The method according to claim 3, characterized in that Also includes: The output matrix is ​​multiplied by an activation function, which is a nonlinear transformation function in the AI ​​model.

5. The method according to claim 3 or 4, characterized in that: The multiplication of the received weight parameter with the activation matrix to obtain the activation parameter of the block specifically includes: The weight parameters of different columns are multiplied by the activation parameters of a row in the activation matrix having the same column number as the weight parameters to obtain activation parameters of multiple columns.

6. The method according to any one of claims 3 to 5, characterized in that: The multiplication of the activation parameter of the block with the weight parameter of the block to obtain the output matrix specifically includes: The activation parameters of multiple columns are multiplied by the weight parameters of each row of the weight parameters of the block to obtain multiple output matrices.

7. The method according to claim 6, characterized in that Also includes: The multiple output matrices are superimposed using allreduce calculation.

8. The method according to any one of claims 1 to 7, characterized in that: The output matrix of the at least one first computing unit is superimposed with the output matrix of the at least one second computing unit to obtain an output activation matrix, specifically comprising: The output activation matrix is ​​obtained by superimposing the elements in the same row and column of the output matrix of the at least one first computing unit and the output matrix of the at least one second computing unit.

9. The method according to any one of claims 1 to 8, characterized in that: The performing a first type of quantization calculation on the weight parameters in the at least one first calculation unit and the activation matrix, and performing a second type of quantization calculation on the weight parameters in the at least one second calculation unit and the activation matrix, specifically includes: The weight parameters of the column where the outlier is located and the activation matrix are calculated using the FP 16 calculation method, and the weight parameters of the column where the outlier is not located and the activation matrix are calculated using the INT 8 calculation method.

10. An inference system for an artificial intelligence (AI) model, characterized in that: include: A first processing unit, used to obtain a weight matrix, where the weight matrix is ​​a matrix composed of weights connecting different neurons in the AI ​​model; Distribute the weight parameters of the column where the outliers in the weight matrix are located to at least one first computing unit, and distribute the weight parameters that do not contain outliers to at least one second computing unit by column; a second processing unit, configured to perform a first type of quantization calculation on the weight parameters in the at least one first calculation unit and the activation matrix, and to perform a second type of quantization calculation on the weight parameters in the at least one second calculation unit and the activation matrix; wherein the activation matrix refers to a feature matrix obtained after the AI ​​model performs a linear transformation on the input sequence; The third processing unit is used to superimpose the output matrix of the at least one first computing unit and the output matrix of the at least one second computing unit to obtain an output activation matrix.

11. The device according to claim 10, characterized in that The first processing unit is specifically used to extract the weight parameter of the column where the outlier value in the weight matrix is ​​located, and set the weight parameter of the column where the outlier value is located at the head of the weight matrix; Based on the set threshold of each first computing unit, each first computing unit is allocated weight parameters of the columns where the outliers of a corresponding number of columns are located.

12. The device according to claim 10 or 11, characterized in that The second processing unit is specifically configured to multiply the received weight parameter by the activation matrix to obtain the activation parameter of the block; the number of columns of the activation parameter of the block is the same as the number of columns of the received weight parameter; The activation parameter of the block is multiplied by the weight parameter of the block to obtain the output matrix; the weight parameter of the block refers to the weight parameter of at least one row in the weight matrix received by the first calculation unit or the second calculation unit, and the row number of the weight parameter of the block is the same as the column number of the received weight parameter.

13. The device according to claim 12, characterized in that The second processing unit is also used to multiply the output matrix by an activation function, where the activation function is a nonlinear transformation function in the AI ​​model.

14. The device according to claim 12 or 13, characterized in that The second processing unit is specifically used to multiply the weight parameters of different columns by the activation parameters of a row in the activation matrix having the same column number as the weight parameters to obtain activation parameters of multiple columns.

15. The device according to any one of claims 12 to 14, characterized in that: The second processing unit is specifically used to multiply the activation parameters of multiple columns by the weight parameters of each row of the weight parameters of the block to obtain multiple output matrices.

16. The device according to claim 15, characterized in that The third processing unit is used to superimpose the multiple output matrices by using full reduction calculation.

17. The device according to any one of claims 10 to 16, characterized in that: The third processing unit is specifically used to superimpose the elements in the same row and column of the output matrix of the at least one first computing unit and the output matrix of the at least one second computing unit to obtain the output activation matrix.

18. The device according to any one of claims 10 to 17, characterized in that: The second processing unit is specifically used to calculate the weight parameters of the column where the outlier is located and the activation matrix using the FP 16 calculation method, and to calculate the weight parameters of the column that does not contain the outlier and the activation matrix using the INT 8 calculation method.

19. A computing device, characterized in that include: at least one memory, At least one processor, the at least one processor is used to execute instructions stored in the at least one memory, so that the computing device performs the method according to any one of claims 1-9.

20. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 9.

21. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, enable the computer to implement the method according to any one of claims 1 to 9.

Images