Artificial intelligence model training method, chip system, and related device

Abstract

The present application provides an AI model training method, a chip, and a related device. The method comprises: a model training system acquiring a first matrix in the forward propagation stage of a model, and determining a scaling coefficient corresponding to each sub-matrix among a plurality of m * m sub-matrices comprised in the first matrix, wherein one scaling coefficient is used for converting data in the corresponding sub-matrix from a high-precision first data type to a low-precision second data type; then converting the first matrix into a second matrix on the basis of the scaling coefficient of each sub-matrix, and performing forward propagation of the model on the basis of the second matrix and the plurality of scaling coefficients; and in the backward propagation stage, performing backward propagation of the model on the basis of a transposed matrix of the second matrix and the plurality of scaling coefficients. When a transposed matrix of a low-precision matrix needs to be used to participate in computation, the low-precision matrix is directly transposed to obtain the transposed matrix to participate in computation, thereby reducing errors caused by conversion between high-precision data and low-precision data, and improving the accuracy of model training.

Description

A training method of an artificial intelligence model, a chip system and related devices

[0001] The present application claims priority to the Chinese patent application No. 202411095488.0, filed on August 9, 2024, and entitled "A training method of an artificial intelligence model, a chip system and related devices", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a training method of an artificial intelligence model, a chip system and related devices. BACKGROUND

[0003] In the scene of training an artificial intelligence (AI) model, the activation value and weight of the model are usually represented by single-precision floating point (FP32) data. However, the FP32 data occupies a large storage space, and requires a large amount of computing resources, storage resources and bandwidth resources in the training process. Therefore, in the training process, high-precision data (such as FP32) is usually converted into low-precision data (such as FP16) through scaling operation, so as to reduce the consumption of resources in the training process and improve the efficiency of model training.

[0004] When converting high-precision data into low-precision data through scaling operation, the scaling coefficients are usually calculated according to the rows or columns of the model parameters. Taking the calculation of scaling coefficients according to the columns as an example, each column of model parameters is divided into one or more data blocks, and a scaling coefficient is calculated for each data block. Then, the high-precision matrix is converted into a low-precision matrix based on the scaling coefficients of the data blocks. However, the method of calculating scaling coefficients according to the rows or columns of the model parameters will cause the model parameters to be converted between high-precision and low-precision multiple times in the training process. For example, in the process of forward propagation, a weight matrix is converted from high-precision to low-precision. In the process of backward propagation, the transposed matrix of the weight matrix is needed, and at this time, the low-precision weight matrix needs to be converted into a high-precision weight matrix, and then the high-precision weight matrix is transposed, and the transposed high-precision weight matrix is converted into a low-precision matrix. Each conversion operation will cause errors in the model parameters, such as rounding errors introduced in the conversion process. Therefore, how to reduce the errors caused by scaling operation in the model training process is a problem to be solved. SUMMARY

[0005] The application provides a training method of an AI model, a chip system and related equipment. By converting the data type of a matrix in the model training process from high-precision data to low-precision data, the calculation amount of model training can be reduced, and the efficiency of model training is provided.

[0006] In a first aspect, the application provides a training method of an AI model. The method is applied to a model training system and includes: obtaining a first matrix and determining a scaling coefficient corresponding to each sub-matrix included in the first matrix in a forward propagation training phase of the AI model, to obtain k scaling coefficients, wherein one scaling coefficient is used to convert data in the corresponding sub-matrix from a first data type to a second data type, the first matrix is activation value data or weight data in the AI model; k and m are positive integers greater than 1, and the precision of the first data type is higher than that of the second data type; then converting the first matrix into a second matrix according to the scaling coefficient corresponding to each sub-matrix, and performing forward propagation training of the AI model according to the second matrix and the k scaling coefficients; in a backward propagation training phase of the AI model, performing backward propagation training of the AI model according to a transposed matrix of the second matrix and the k scaling coefficients. The data type of the data in the second matrix is the second data type.

[0007] In the application, by dividing a matrix into multiple m*m sub-matrices, a scaling coefficient is calculated for each sub-matrix, and a high-precision matrix is converted into a low-precision matrix according to the scaling coefficient of each sub-matrix, and then a low-precision matrix is used for calculation in the training process of the AI model, which can reduce the occupation of computing resources, storage resources and bandwidth resources. When a transposed matrix of a low-precision matrix is needed to participate in calculation, the transposed matrix is obtained by directly performing a transposition operation on the low-precision matrix, and the transposed matrix is also divided into multiple sub-matrices according to m*m. Since the data amount included in the original matrix (the matrix before the transposition of the transposed matrix) and the transposed matrix is the same, both are sub-matrices divided according to m*m. Each sub-matrix in the transposed matrix has a corresponding mapping matrix in the original matrix, and each sub-matrix in the original matrix has a corresponding scaling coefficient, so the scaling coefficient of each sub-matrix in the transposed matrix can use the scaling coefficient of the corresponding mapping matrix in the original matrix, which can reduce the number of mutual conversions between high-precision matrices and low-precision matrices in the calculation process, reduce the error caused by the conversion between high-precision data and low-precision data, and improve the accuracy of model training. The mapping matrix of the transposed matrix in the original matrix refers to a sub-matrix in the original matrix that has the same data as the transposed matrix.

[0008] In a possible implementation, the second matrix includes a converted sub-matrix corresponding to each sub-matrix in the first matrix, and a scaling coefficient of a first converted sub-matrix in the second matrix is a first scaling coefficient corresponding to a first sub-matrix in the first matrix, and the first converted sub-matrix is converted from the first sub-matrix.

[0009] The second matrix includes a converted sub-matrix corresponding to each sub-matrix in the first matrix, and a scaling coefficient of a converted sub-matrix in the second matrix is a scaling coefficient of a sub-matrix corresponding to the converted sub-matrix in the first matrix. For example, a first converted sub-matrix is converted from a first sub-matrix, and a scaling coefficient of the first converted sub-matrix is a first scaling coefficient corresponding to the first sub-matrix. Each sub-matrix in the transposed matrix of the second matrix has a mapping matrix in the second matrix, and a scaling coefficient of a sub-matrix in the transposed matrix of the second matrix can use a scaling coefficient of a mapping matrix of the sub-matrix in the second matrix.

[0010] In a possible implementation, the model training system includes a first computing unit, and the back propagation training of the AI model according to the transposed matrix of the second matrix and the k scaling coefficients includes: performing, by the first computing unit, a matrix multiplication operation according to the transposed matrix of the second matrix, the k scaling coefficients, and a third matrix to obtain an intermediate result of the back propagation, and performing the back propagation according to the intermediate result. The data in the third matrix is of a second data type, and the third matrix is gradient data in the AI model.

[0011] In a possible implementation, the first computing unit includes a vector computing unit, a first adder, and a first multiplier, and the first computing unit performs the matrix multiplication operation according to the transposed matrix of the second matrix, the k scaling coefficients, and the third matrix to obtain the intermediate result of the back propagation, including: the vector computing unit obtains a first vector and a second vector, and performs a vector inner product operation to obtain an output result, the output result including a plurality of products obtained by multiplying corresponding elements in the first vector and the second vector, the first vector being a row of data in a sub-matrix in the third matrix, and the second vector being a column of data in a sub-matrix in the transposed matrix of the second matrix; the first adder performs an addition operation on the output result of the vector computing unit to obtain an operation result of the vector inner product operation; and the first multiplier obtains a value in the intermediate result according to the operation result of the inner product operation and a second scaling coefficient, and obtains the intermediate result according to the value in the intermediate result, the second scaling coefficient being a scaling coefficient corresponding to a sub-matrix to which the second vector belongs.

[0012] In a possible implementation, the model training system further includes a second computing unit; and the determining the scaling factor corresponding to each of the k m*m sub-matrices included in the first matrix includes: the second computing unit acquires a first sub-matrix, and determines a maximum value in the first sub-matrix; the first sub-matrix is any one of the k sub-matrices; and the second computing unit determines a first scaling factor corresponding to the first sub-matrix based on the maximum value in the first sub-matrix and a maximum value that can be represented by the second data type.

[0013] In a possible implementation, the converting the first matrix into the second matrix according to the scaling factor corresponding to each of the sub-matrices includes: the second computing unit multiplies the first sub-matrix by the first scaling factor to obtain a first converted sub-matrix corresponding to the first sub-matrix, and obtains the second matrix based on the first converted sub-matrix.

[0014] In a possible implementation, before the determining the scaling factor corresponding to each of the k m*m sub-matrices included in the first matrix, the method further includes: converting the fourth matrix into an n*n matrix to obtain the first matrix; n is an integer multiple of m.

[0015] In a second aspect, the present application provides a chip system, which includes at least one first computing unit and at least one second computing unit, wherein the second computing unit is configured to acquire a first matrix, and determine a scaling factor corresponding to each of k m*m sub-matrices included in the first matrix to obtain k scaling factors, wherein one scaling factor is used to convert data in a corresponding sub-matrix from a first data type to a second data type, the first matrix is activation data or weight data in an AI model; k and m are positive integers greater than 1, and the precision of the first data type is higher than that of the second data type; the second computing unit is further configured to convert the first matrix into a second matrix according to the scaling factor corresponding to each of the sub-matrices; the first computing unit is configured to perform forward propagation training of the AI model based on the second matrix and the k scaling factors; and the first computing unit is further configured to perform back propagation training of the AI model based on a transposed matrix of the second matrix and the k scaling factors in a back propagation training stage of the AI model.

[0016] In a possible implementation, the second matrix includes a converted sub-matrix corresponding to each of the sub-matrices in the first matrix, the first converted sub-matrix in the second matrix is a converted sub-matrix corresponding to a first sub-matrix in the first matrix, and the scaling factor corresponding to the first converted sub-matrix is a first scaling factor corresponding to the first sub-matrix; the scaling factor corresponding to a second sub-matrix in the transposed matrix of the second matrix is a scaling factor corresponding to the first converted sub-matrix, and the second sub-matrix and the first converted sub-matrix include the same data.

[0017] In a possible implementation, the first calculation unit is specifically configured to: perform a matrix multiplication operation according to a transpose matrix of the second matrix, the k scaling coefficients, and a third matrix to obtain an intermediate result of back propagation, and perform the back propagation training according to the intermediate result; wherein data in the third matrix is of the second data type, and the third matrix is gradient data in the AI model.

[0018] In a possible implementation, the first calculation unit includes a vector calculation unit, an adder, and a first multiplier. The vector calculation unit is configured to obtain a first vector and a second vector, and perform a vector inner product operation to obtain an output result, the output result including a plurality of products obtained by multiplying corresponding elements in the first vector and the second vector, the first vector being a row of data in a submatrix in the third matrix, and the second vector being a column of data in a submatrix in the transpose matrix of the second matrix. The adder is configured to perform an addition operation on the output result of the vector calculation unit to obtain an operation result of the vector inner product operation. The first multiplier is configured to obtain one value in the intermediate result according to the operation result of the inner product operation and a second scaling coefficient, and obtain the intermediate result according to the one value in the intermediate result. The second scaling coefficient is a scaling coefficient corresponding to the submatrix to which the second vector belongs.

[0019] In a possible implementation, the second calculation unit is specifically configured to: obtain a first submatrix, and determine a maximum value in the first submatrix; wherein the first submatrix is any one of the k submatrices; and determine a first scaling coefficient corresponding to the first submatrix according to the maximum value in the first submatrix and a maximum value that can be represented by the second data type.

[0020] In a possible implementation, the second calculation unit includes a maximum value calculation unit and a second multiplier. The maximum value calculation unit is configured to obtain a first submatrix, and determine a maximum value in the first submatrix. The second multiplier is configured to determine a first scaling coefficient corresponding to the first submatrix according to the maximum value in the first submatrix and a maximum value that can be represented by the second data type. The second multiplier is further configured to multiply the first submatrix by the first scaling coefficient to obtain a first converted submatrix corresponding to the first submatrix, and obtain the second matrix according to the first converted submatrix.

[0021] In a third aspect, the present application provides a computing device including the chip system and the memory as described in the second aspect or any possible implementation of the second aspect, and the memory is configured to store data in the AI model, and the chip system is capable of implementing the method as described in the first aspect or any possible implementation of the first aspect.

[0022] In a fourth aspect, a computer readable storage medium is provided. The computer readable storage medium includes computer program instructions. When the computer program instructions are executed by a computing device, the computing device is caused to perform the method according to the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0023] FIG. 1 is a schematic diagram of a self-attention structure according to the present disclosure;

[0024] FIG. 2 is a schematic diagram of data conversion according to the present disclosure;

[0025] FIG. 3 is a schematic diagram of an AI model training method according to the present disclosure;

[0026] FIG. 4 is a schematic diagram of dividing a matrix block according to the present disclosure;

[0027] FIG. 5 is a schematic diagram of a first calculation unit according to the present disclosure;

[0028] FIG. 6 is a schematic diagram of a second calculation unit according to the present disclosure;

[0029] FIG. 7 is a schematic diagram of a third calculation unit according to the present disclosure;

[0030] FIG. 8 is a schematic diagram of implementing a matrix multiplication operation according to the present disclosure;

[0031] FIG. 9 is a schematic diagram of a computing device according to the present disclosure. DETAILED DESCRIPTION

[0032] The data processing method according to the present disclosure will be described below with reference to the accompanying drawings.

[0033] The data processing method according to the present disclosure is related to converting high-precision data into low-precision data. It should be understood that high-precision data and low-precision data are relative concepts. For example, for double-precision floating point (FP64) data and FP32 data, the FP64 type data is high-precision data, and the FP32 type data is low-precision data. For FP32 type data and FP16 type data, the FP32 type data is high-precision data, and the FP16 type data is low-precision data. Generally, high-precision data requires more storage space than low-precision data. For example, FP32 type data requires 32 bits to represent a floating point number, and FP16 type data only requires 16 bits to represent a floating point number.

[0034] In the process of training an artificial intelligence (AI) model, single-precision floating point (FP32) data is usually used to represent the activation values, weights and gradients of the model parameters of the model. The range of data that can be represented by FP32 is large, making the model training more accurate. However, the FP32 type of data consumes more computing resources and storage resources during the training process, and in distributed training, more bandwidth resources are also required. Therefore, in the process of AI model training, high-precision data (such as FP32) is usually converted into low-precision data (such as FP16) through a scaling operation, so as to reduce the consumption of resources during the training process and improve the efficiency of model training.

[0035] In the forward propagation training phase of the AI model, matrix multiplication between two matrices is usually involved, such as matrix multiplication between activation value data and weight data in convolutional neural networks, recurrent neural networks or feedforward neural networks. For example, in a transformer model, multiple self-attention structures are included, each of which is shown in FIG. 1 and includes a first matrix multiplication structure, a softmax structure and a second matrix multiplication structure. For an input matrix X, the input matrix X is multiplied by three weight matrices W Q , W K and W V , respectively, to convert the input matrix X into a query matrix Q, a key matrix K and a value matrix V required for calculating attention values. As can be seen from FIG. 1, each self-attention structure includes at least two matrix multiplications, which are matrix multiplication between the query matrix Q and the key matrix K, and matrix multiplication between the output matrix of the softmax and the value matrix V, etc.

[0036] In the backpropagation training phase of the AI model, the gradient data corresponding to the weights of the AI model is calculated according to the loss value obtained by the forward propagation, and then the weight data of the AI model is updated according to the gradient data. In the process of backpropagation training, when calculating the gradient corresponding to the weights of the L-1 layer of the AI model, the gradient data corresponding to the weights of the L layer and the transpose matrix of the weight matrix of the L layer need to be multiplied.

[0037] In the training process of AI model forward propagation and back propagation, a large number of matrix multiplication operations can convert the data in the two matrices involved in the matrix multiplication operation from high-precision data to low-precision data, and then perform matrix multiplication operation, thereby reducing the calculation amount in the training process, reducing the consumption of resources in the training process, and improving the training efficiency. Since the numerical range that can be represented by the high-precision data type is greater than the numerical range that can be represented by the low-precision data type, when converting high-precision data to low-precision data, a scaling factor needs to be calculated first, and then the numerical range of the high-precision data is mapped to the numerical range that can be represented by the low-precision data type according to the scaling factor, and then the data format is converted to the data format of the low-precision data, and finally the low-precision data is output.

[0038] Currently, the scaling factor is usually calculated according to the row or column of the matrix, and then the high-precision data is converted to low-precision data according to the scaling factor. Taking the calculation of the scaling factor according to the column as an example, the data in each column of a matrix is divided into one or more data blocks (blocks), a scaling factor is calculated for each data block, and then the high-precision data included in the data block is converted to low-precision data according to the scaling factor corresponding to the data block, and finally the low-precision matrix and the scaling factor corresponding to each block are output.

[0039] For example, referring to FIG. 2, FIG. 2 is a schematic diagram of data conversion provided by the present application. In the process of AI model training, for a p*q weight matrix A, if the matrix A is the right matrix of a matrix multiplication operation in the forward propagation process, when the scaling operation is performed on the matrix A, the scaling factor is first calculated according to the column of the high-precision matrix A, for example, the data in each column of the matrix A is divided into 3 data blocks, and each data block includes p / 3 values in 1 column of data. A scaling factor is calculated for each data block, so that each column of data corresponds to 3 scaling factors S i1 ~S i3 , for example, the three scaling factors of the first column are S 11 , S 12 and S 13 ; the three scaling factors of the second column are S 21 , S 22 and S 23 ; and the three scaling factors of the qth column are S q1 , S q2 and S q3; each row of data corresponds to q scaling coefficients, that is, each data in a row of data corresponds to a scaling coefficient. Wherein, the scaling coefficients corresponding to a data block are used to convert the data in the data block to the numerical range that the low-precision data type can represent. The above p and q are positive integers, and i is a positive integer less than or equal to q. Then, according to the scaling coefficients of each matrix block, a scaling operation is performed on the matrix A to convert the high-precision matrix A into a low-precision matrix B, which is used for matrix multiplication that requires the participation of the matrix B. Through the above method, the high-precision data in the matrix is converted into low-precision data for matrix multiplication, which reduces the calculation amount in the training process, reduces the consumption of resources in the training process, and improves the training efficiency.

[0040] However, in the process of AI model back propagation training, the transpose matrix of the weight matrix is needed. At this time, if the transpose operation is directly performed on the matrix B to obtain the transpose matrix B T , a row of data in the matrix B is converted into a column of data in the transpose matrix B T . Each data in a column of data in the transpose matrix B T corresponds to a scaling coefficient. However, usually after the matrix multiplication is implemented using low-precision data, the result of the matrix multiplication needs to be represented by high-precision data. Specifically, when implementing matrix multiplication, a row of data of the left matrix and a column of data of the right matrix are used to perform vector inner product operation to obtain the partial result of matrix multiplication. After obtaining the calculation result by performing a vector inner product operation once, the calculation result and the scaling coefficients corresponding to each data participating in the vector inner product operation are processed to obtain high-precision data. Since each data participating in the vector inner product operation corresponds to a separate scaling coefficient, the current hardware cannot implement the above processing, or it is quite complex to implement the above processing through hardware, which has a high cost and a low benefit compared with directly using high-precision data for matrix multiplication.

[0041] As shown in FIG. 2, when the transpose matrix of the weight matrix is needed, the above low-precision matrix B is first converted into a high-precision matrix C according to the above scaling coefficients, and then the transpose operation is performed on the matrix C to obtain the transpose matrix C T . The scaling coefficients are calculated according to the columns of the transpose matrix C T , for example, each column of data in the transpose matrix C T is divided into 3 data blocks, each data block includes q / 3 values in a column of data, and then a scaling coefficient is calculated for each data block. According to the scaling coefficients, the high-precision transpose matrix C TThe scaling operation yields a low-precision matrix D, which in turn provides the low-precision weight matrix used for backpropagation. It should be noted that converting the high-precision matrix A to the low-precision matrix B involves quantization and / or rounding of the high-precision data. Therefore, after converting the low-precision matrix B to the high-precision matrix C, the data in matrix C and matrix A are not entirely identical.

[0042] For a weight matrix in the model, during forward propagation, the high-precision matrix A needs to be transformed into a low-precision matrix B. During backpropagation, the low-precision matrix B needs to be transformed into a high-precision matrix C first, and then the high-precision matrix C... T Converting to a low-precision matrix D introduces errors in model parameters with each transformation operation, such as quantization and rounding during the conversion from high-precision to low-precision data. Therefore, reducing the errors caused by scaling operations during model training is a pressing issue.

[0043] This application provides a data processing method. When it is necessary to convert a matrix from high-precision data to low-precision data, for example, to convert the aforementioned high-precision weight matrix A into a low-precision matrix E, matrix A is divided into m*m matrix blocks. In this application, a matrix block is called a submatrix. Then, for any submatrix, a scaling factor is calculated based on the data in the submatrix. The scaling factor corresponding to a submatrix is ​​used to convert the high-precision data in the submatrix into low-precision data. Thus, for any m*m submatrix in the matrix, each submatrix corresponds to a scaling factor. Then, the matrix is ​​scaled based on the scaling factor of each submatrix to obtain the low-precision matrix E for calculation in forward propagation.

[0044] During backpropagation, the low-precision matrix E is transposed to obtain the transpose matrix E. T Similarly, the transpose matrix E T Divide the matrix into m*m blocks to obtain the transpose matrix E. T Includes each submatrix. Due to matrix E and its transpose E T The submatrix of E has both m rows and m columns, therefore for the transpose matrix E T Each submatrix in matrix E has a corresponding mapping matrix and a transpose matrix E. T The mapping matrix of a submatrix in matrix E is the submatrix in matrix E that has the same data as the original submatrix. Therefore, the transpose matrix E TEach sub-matrix in the matrix E can use the scaling coefficient of the corresponding mapping matrix in the matrix E, and each sub-matrix in the transposed matrix corresponds to a scaling coefficient, instead of each data in a sub-matrix corresponding to a scaling coefficient. Therefore, when back propagation, the transposed matrix of the matrix E and the scaling coefficient corresponding to each sub-matrix in the matrix E can be directly used to participate in the calculation, the number of conversions between high-precision data and low-precision data in the training process can be reduced, and the error caused by the conversion between high-precision data and low-precision data can be reduced.

[0045] The AI model training method provided in the present application will be described below in conjunction with the accompanying drawings. Referring to FIG. 3, FIG. 3 is a schematic diagram of an AI model training method provided in the present application. The method includes the following S201-S204.

[0046] S201. Obtain a first matrix for matrix multiplication, and divide the first matrix into multiple sub-matrices.

[0047] In the forward propagation training phase of the AI model, when matrix multiplication is needed, the model training system obtains a first matrix in the matrix multiplication, and then divides the first matrix into k sub-matrices, and the number of rows and the number of columns of any one sub-matrix are the same. For example, the sub-matrices are all m*m matrices, wherein the data in any two sub-matrices does not include the data in the same position in the matrix. The data type of the data in the first matrix is a first data type, k and m are positive integers greater than 1,

[0048] The above-mentioned first matrix can be an activation value matrix or a weight matrix in the AI model training process. If it is the first layer of the AI model, the first matrix can also be an input matrix corresponding to the input data of the model. For example, the query matrix, the key matrix or the value matrix in the above-mentioned transformer model. The data type in the first matrix is the first data type. The above-mentioned m can be 8, 16, etc. The size of m is not limited in the present application.

[0049] It should be noted that if the number of rows or the number of columns of the first matrix is not an integer multiple of m, the first matrix needs to be converted into an n*n matrix. Wherein n is an integer multiple of m, for example, the first matrix is converted into an n*n matrix by padding.

[0050] S202. Determine the scaling coefficient corresponding to each sub-matrix according to the data included in each sub-matrix, and convert the first matrix into a second matrix according to the scaling coefficient corresponding to each sub-matrix.

[0051] After dividing each matrix into k sub-matrices, for each sub-matrix, a scaling coefficient corresponding to each sub-matrix is determined according to the data included in each sub-matrix, and k scaling coefficients are obtained. In this application, when determining the scaling coefficient corresponding to each sub-matrix, for any sub-matrix in any matrix, for example, the first sub-matrix in the first matrix, first determine the maximum value T max and the maximum value V max that can be represented by the second data type; wherein the precision of the first data type is higher than the precision of the second data type, or the storage space occupied by the data of one first data type is greater than the storage space occupied by the data of one second data type. After determining T max and V max , the first scaling coefficient S1 corresponding to the first sub-matrix is determined according to T max and V max ; for example, S1 = V max / T max or S1 = (V max / T max ) / 2 margina . The computing device can determine the scaling coefficient corresponding to each sub-matrix in the first matrix according to the above method. After determining the scaling coefficient corresponding to each sub-matrix, the data in each sub-matrix is converted from high-precision data to low-precision data according to the scaling coefficient corresponding to each sub-matrix, and a low-precision conversion matrix corresponding to each sub-matrix is obtained, that is, a conversion matrix corresponding to each matrix is obtained.

[0052] Taking the first sub-matrix as an example, after determining the first scaling coefficient S1, the first sub-matrix is multiplied by the first scaling coefficient S1, so that the value of each data in the first sub-matrix is mapped to the numerical range that can be represented by the second data type; since the data in the first sub-matrix is still represented in the format of the first data type, after multiplying the first sub-matrix by the first scaling coefficient S1, the data represented in the first data type format is converted to the format represented by the second data type. For example, the first data type is FP32, the second data type is FP16, the data in the first sub-matrix is represented by 32 bits, and FP16 is represented by 16 bits. After multiplying the first sub-matrix by the scaling coefficient S1, although each data in the first sub-matrix belongs to the numerical range that can be represented by FP16, it is still represented by 32 bits. Therefore, after multiplying the first sub-matrix by the scaling coefficient, the data represented in the first data type format needs to be converted to the data represented in the second data type format, and finally the first conversion sub-matrix represented in the format of the second data type and the scaling coefficient corresponding to the first conversion sub-matrix, that is, the low-precision first conversion matrix corresponding to the high-precision first sub-matrix, are obtained. The scaling coefficient corresponding to the first conversion sub-matrix is also the first scaling coefficient S1 corresponding to the first sub-matrix.

[0053] The above describes a method of converting a high-precision first sub-matrix into a low-precision first conversion sub-matrix. For each sub-matrix in the first matrix, a high-precision sub-matrix can be converted into a low-precision conversion sub-matrix by the above method, thereby obtaining a low-precision second matrix corresponding to the first matrix. That is, the second matrix includes a conversion sub-matrix corresponding to each sub-matrix in the first matrix. The scaling coefficients of a conversion sub-matrix in the second matrix are the scaling coefficients of the sub-matrix corresponding to the conversion sub-matrix in the first matrix. For example, the first conversion sub-matrix is converted from the first sub-matrix, and the scaling coefficients of the first conversion sub-matrix are the first scaling coefficients S1 corresponding to the first sub-matrix.

[0054] S203. Perform forward propagation training of the AI model according to the second matrix and the scaling coefficients corresponding to each sub-matrix.

[0055] In the forward propagation training phase, for two matrices that need to perform matrix multiplication operations, the low-precision matrix corresponding to each matrix can be obtained according to the method in S201 and S202, and then the forward propagation training is performed according to the low-precision matrix corresponding to each matrix. For example, after determining the low-precision second matrix corresponding to the first matrix, for a matrix that performs matrix multiplication with the first matrix, the matrix is also converted into a low-precision fifth matrix according to the method in S201 and S202; then the forward propagation training of the AI model is performed according to the second matrix, the scaling coefficients corresponding to each sub-matrix in the first matrix, and the fifth matrix, to obtain the result of the forward propagation of the AI model.

[0056] For example, if the first matrix A is divided into four sub-matrices, the four conversion sub-matrices included in the second matrix E are A1, A2, A3, and A4, and the six conversion sub-matrices included in the fifth matrix W are W1, W2, W3, W4, W5, and W6. Then the operation result O of the matrix multiplication of the second matrix A and the fifth matrix W can be represented as the following formula 1:

[0057] Wherein, since each conversion sub-matrix is m*m, the above-mentioned matrix Oij is also an m*m matrix, Oij = Ai x Wj, i is a positive integer less than or equal to 4, and j is a positive integer less than or equal to 6. It should be noted that, since the matrix A1-A4 and the matrix W1-W6 are low-precision data matrices, but in the AI model training, the result of the matrix multiplication operation usually needs to be converted into high-precision data to participate in subsequent calculations, for example, in the above-mentioned FIG. 1, the input of the softmax needs to be a high-precision matrix, so the result of the matrix multiplication of the query matrix and the key matrix needs to be converted into high-precision. Therefore, in the process of calculating the matrix multiplication of the sub-matrix, the hardware unit has converted the operation result of the matrix multiplication of the two sub-matrices into high-precision data according to the scaling coefficients corresponding to the two sub-matrices. The process of the hardware unit implementing the matrix multiplication is described in the following FIG. 5, and will not be described in detail here.

[0058] S204. In the back propagation training phase of the AI model, the back propagation training of the AI model is performed according to the transposed matrix of the second matrix and the scaling coefficient corresponding to each sub-matrix.

[0059] In the back propagation training phase of the AI model, since in the back propagation process, the gradient of the loss function with respect to the weight needs to be calculated, and then the weight is updated, and according to the chain rule, when calculating the gradient of the loss function with respect to the weight, the transposed matrix of the weight matrix needs to be used.

[0060] In this application, if the above-mentioned first matrix is a weight matrix, the transposed matrix of the weight matrix is needed for related calculations in the back propagation training phase, and the transposed matrix of the low-precision weight matrix is also needed in the matrix multiplication and other calculations. In this application, the transposed matrix of the low-precision second matrix corresponding to the first matrix is used in the back propagation phase of the model training. For ease of description, the transposed matrix of the second matrix is referred to as the sixth matrix. If the sixth matrix is divided into sub-matrices according to m*m, the sixth matrix can also be divided into k sub-matrices, and each sub-matrix in the sixth matrix has a mapping matrix in the second matrix. Wherein, the mapping matrix of the transposed matrix in the original matrix refers to the sub-matrix in the original matrix which has the same data as the transposed matrix. The scaling coefficient of a sub-matrix in the sixth matrix can use the scaling coefficient of the mapping matrix of the sub-matrix in the second matrix. For example, the second sub-matrix in the sixth matrix corresponds to the first conversion sub-matrix in the second matrix, and the scaling coefficient of the first conversion sub-matrix is the first scaling coefficient S1. Therefore, the scaling coefficient of the second sub-matrix in the sixth matrix is also the first scaling coefficient S1. Therefore, the scaling coefficient corresponding to the sixth matrix is the scaling coefficient corresponding to the second matrix, that is, the scaling coefficient corresponding to each sub-matrix in the first matrix determined in S202.

[0061] After determining the transpose of the second matrix and its corresponding scaling factor, backpropagation training is performed based on the sixth matrix, the scaling factors corresponding to each of the above sub-matrices, and the third matrix. The third matrix can be gradient data; for example, during the backpropagation training phase, calculating the gradient of the (L-1)th layer requires the gradient data of the Lth layer and the transpose of the Lth layer's weight matrix. The data in the third matrix is ​​of the second data type. The method for matrix multiplication between the sixth and third matrices can be found in Equation 1 and the relevant explanations in Figure 5 below, and will not be detailed here.

[0062] In this application, a matrix is ​​divided into multiple m*m submatrices, a scaling factor is calculated for each submatrix, and the high-precision matrix is ​​converted into a low-precision matrix based on the scaling factor of each submatrix. Then, the calculation is performed using the low-precision matrix during the training process of the AI ​​model, which can reduce the occupation of computing resources, storage resources and bandwidth resources.

[0063] When a low-precision matrix's transpose is needed for computation, the low-precision matrix is ​​directly transposed to obtain the transpose matrix. The transpose matrix is ​​then divided into multiple submatrices in an m*m format. Since the original matrix (the matrix before transpose) and the transpose matrix contain the same amount of data, both being submatrices divided in an m*m format, the transpose matrix has a corresponding mapping matrix in the original matrix.

[0064] For example, as shown in Figure 4, which is a schematic diagram of dividing a matrix into blocks using the original matrix and its transpose, according to this application, a low-precision matrix E is obtained from the high-precision matrix A in Figure 2 using the methods described in S201 and S202. Each data point in matrix E is represented as a. ij The matrix E, as shown in Figure 4, consists of four submatrices: A1, A2, A3, and A4. Transposing matrix E yields the transpose matrix E. T If the value of m above is 4, then matrix E and its transpose E T Each matrix is ​​divided into multiple submatrices using a 4x4 matrix, as shown in Figure 4. The transpose matrix E T Each submatrix in matrix E has a corresponding mapping matrix in matrix E. For example, a submatrix A1 in matrix E contains data (a 11 a 12 a 13 a 14 ;a 21 a 22 a 23 a 24 ;a 31 a 32 a 33 a 34 ;a 41a 42 a 43 a 44 ) ; a T sub-matrix of the matrix E 11 a 21 a 31 a 41 a 12 a 22 a 32 a 42 a 13 a 23 a 33 a 43 a 14 a 24 a 34 a 44 ), which is the mapping matrix of the matrix A1, denoted as A1 T ; for example, a sub-matrix A3 of the matrix E includes data (a 15 a 16 a 17 a 18 a 25 a 26 a 27 a 28 a 35 a 36 a 37 a 38 a 45 a 46 a 47 a 48 ), and a sub-matrix of the transposed matrix E T includes data (a 15 a 25 a 35 a 45 a 16 a 26 a 36 a 46 a 17 a 27 a 37 a 47 a 18 a 28 a 38 a 48 ), which is the mapping matrix of the matrix A3, denoted as A3 T .

[0065] Each sub-matrix in the transposed matrix has a corresponding mapping matrix in the original matrix, and each sub-matrix in the original matrix has a corresponding scaling coefficient, so the scaling coefficient of each sub-matrix in the transposed matrix can use the scaling coefficient of the corresponding mapping matrix in the original matrix, for example, the scaling coefficient of the sub-matrix A1 T The scaling coefficient of the sub-matrix A1 is used, the scaling coefficient of the sub-matrix A3 T The scaling coefficient of the sub-matrix A3 is used; in this way, when the transposed matrix of the matrix E needs to be used, the above low-precision conversion matrix E can be directly transposed to be used for calculation, without the need to obtain the matrix D according to the corresponding method in FIG. 2, the number of mutual conversions between the high-precision matrix and the low-precision matrix can be reduced, the error caused by the conversion between the high-precision data and the low-precision data can be reduced, the accuracy of model training can be improved, and the process of obtaining the matrix C T The process of recalculating the scaling coefficient by column is reduced, the calculation amount is reduced, and the efficiency of model training is improved.

[0066] For the above method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action sequence described, and those skilled in the art should also know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily required by the present application. Other reasonable step combinations that those skilled in the art can think of based on the above description also belong to the protection scope of the present application.

[0067] Based on the above method, the present application also provides a chip system for implementing the above scaling operation and matrix multiplication. The chip system provided by the present application includes a first calculation unit and a second calculation unit. The first calculation unit is used to implement matrix multiplication of two matrices; the second calculation unit is used to determine the above scaling coefficient and perform scaling operation on the matrix based on the scaling coefficient to obtain the conversion matrix corresponding to each matrix. The process of implementing calculation by the first calculation unit and the second calculation unit will be introduced respectively.

[0068] Referring to FIG. 5, FIG. 5 is a schematic diagram of a first calculation unit provided by the present application. The first calculation unit includes a vector calculation unit, a first adder and a first multiplier. The vector calculation unit is used to perform vector inner product operation. It should be understood that the operation result of matrix multiplication is obtained by multiple vector inner product operations, for example, the matrix multiplication between the matrix A1 and the matrix W1. Since the matrix A1 and the matrix W1 are both m rows and m columns, the matrix A1 is equivalent to m row vectors, and the matrix W1 is equivalent to m column vectors. The matrix multiplication between the matrix A1 and the matrix W1 needs m*m times of vector inner product operation.

[0069] The following takes the first computing unit to complete the matrix multiplication operation of matrix A1 and matrix W1 as an example to introduce the calculation process of the first computing unit. As shown in FIG. 5, the vector computing unit includes a multiplication array including k multipliers, the vector computing unit reads continuous k values of a row of data in matrix A1 each time, takes the k values as a vector x, and reads continuous k values of a column of data in matrix W1, takes the k values as a vector y. Among them, the vector x can be a row of data in matrix A1, or part of the data in a row of matrix A1. Then the inner product operation of the vector x and the vector y is performed, and the result z1 of the inner product operation of the vector x and the vector y is calculated through the vector computing unit and the first adder. Wherein, z1 is a numerical value. Specifically, each multiplier in the vector computing unit obtains a numerical value in the vector x and the vector y, and performs multiplication operation to obtain k products, and then inputs the k products into the first adder to perform addition calculation to obtain the result z1 of the inner product operation of the vector x and the vector y.

[0070] Since the data in matrix A1 and matrix W1 are all low-precision data, z1 is also low-precision data, that is, z1 is data of the second data type. However, in AI model training, the result of matrix multiplication operation usually needs to be converted to high-precision data to participate in subsequent calculation. Therefore, the first computing unit further includes a first multiplier, which is used to convert the numerical value represented by the low-precision data into a numerical value within the range that can be represented by the high-precision data. Specifically, the second multiplier is used to multiply the numerical value output by the adder by the inverse of the scaling coefficient corresponding to the vector x, and then multiply by the inverse of the scaling coefficient corresponding to the vector y. Thus, the data represented by the second data type is converted to a numerical value within the range that can be represented by the first data type, so as to convert the result of the matrix multiplication operation to high-precision data. Wherein, the scaling coefficient corresponding to the vector x is the scaling coefficient corresponding to the sub-matrix to which the vector x belongs, and the scaling coefficient corresponding to the vector y is the scaling coefficient corresponding to the sub-matrix to which the vector y belongs.

[0071] It should be understood that the vector computing unit may not be able to complete the vector inner product operation of a row of data in matrix A1 and a column of data in matrix W1 at one time, for example, m is large, the vector computing unit can only read part of the data in a row of matrix A1 and part of the data in a column of matrix W1 at one time to complete partial vector inner product operation, and the vector inner product operation of a row of data in matrix A1 and a column of data in matrix W1 can be completed through multiple vector inner product operations.

[0072] For example, m equals to 3k, the vector calculation unit can only read 1 / 3 of the data in a row and 1 / 3 of the data in a column of the sub-matrix for vector inner product operation at one time, then for one row of data in matrix Al and one column of data in matrix Wl, the vector calculation unit needs to perform 3 times of vector inner product calculation to complete the vector inner product operation of one row of data in matrix Al and one column of data in matrix Wl. The vector calculation unit reads the first k values of the row of data in matrix Al and the first k values of the column of data in matrix Wl at the first time, and then performs vector inner product calculation to obtain a value zl, and then the value zl is multiplied by 1 / S A1 and 1 / S W1 to obtain C1, and the first calculation unit records C1. Wherein, S A1 is the scaling coefficient corresponding to the vector x, that is, the scaling coefficient corresponding to matrix Al, S W1 is the scaling coefficient corresponding to the vector y, that is, the scaling coefficient corresponding to matrix Wl.

[0073] Then the vector calculation unit reads the k values in matrix Al and the k values in matrix Wl again to perform vector inner product operation to obtain a value z2, and then the value z2 is multiplied by 1 / S A1 and 1 / S W1 to obtain C2, and the first calculation unit further comprises a second adder, and the first calculation unit adds C2 and C1 through the second adder to obtain C3, and records C3. The vector calculation unit reads the last k values in matrix Al and the last k values in matrix Wl again, and then performs vector inner product operation to obtain a value z3, and then the value z3 is multiplied by 1 / S A1 and 1 / S W1 to obtain C4, and the first calculation unit adds C3 and C4 through the second adder to obtain C5, and C5 is the vector inner product of one row of data in matrix Al and one column of data in matrix Wl, that is, one data in matrix O11 in the above formula 1. For example, C5 is the vector inner product of the second row of data in matrix Al and the third column of data in matrix Wl, then C5 is the data in the second row and the third column in matrix O11. Since matrix Al and matrix Wl each include m rows and m columns of data, the vector calculation unit needs to perform three times of calculation to calculate the inner product of one row in Al and one column in Wl, and the vector calculation unit needs to perform 3m*m times of vector inner product operation to complete the matrix multiplication of matrix Al and matrix Wl, and the above formula 1 is matrix O11.

[0074] For example, if k is 4, in one vector inner product operation, the vector x=(a 15 , a 16 , a 17 , a 18 ), the vector y=(w 51 , w61 , w 71 , w 81 ), as shown in FIG. 5, the four multipliers of the vector calculation unit calculate a 15 x w 51 , a 16 x w 61 , a 17 x w 71 and a 18 x w 81 , respectively. The first adder adds the four products output by the vector calculation unit to obtain a value z2, and multiplies the value z2 by 1 / S A1 and 1 / S W1 to obtain C2 through the first multiplier, the second adder adds C2 and C1 to obtain C3, and records C3.

[0075] In a possible implementation, for one of the two matrices that need to be multiplied, for example, the first matrix described above, the scaling coefficients corresponding to the matrix can be determined according to the methods in S201 to S203, and the conversion matrix corresponding to each sub-matrix in the matrix can be determined based on the scaling coefficients. For the high-precision matrix corresponding to the fifth matrix W, only one scaling coefficient can be obtained for the high-precision matrix, and in the calculation of the scaling coefficient corresponding to the high-precision matrix, first, the maximum value T max in the high-precision matrix and the maximum value V max that can be represented by the second data type are determined; then the scaling coefficient S4 corresponding to the high-precision matrix is determined according to T max and V max ; for example, S4 = V max / T max or S4 = (V max / T max ) / 2 margina .

[0076] If the fifth matrix W corresponds to only one scaling coefficient, in the process of implementing the matrix multiplication operation of the second matrix and the fifth matrix through the first calculation unit shown in FIG. 5, the first multiplier only needs to multiply the result output by the first adder by the scaling coefficient corresponding to the vector x each time, and after the matrix multiplication operation of the second matrix and the fifth matrix is completed to obtain an operation result, the operation result is multiplied by the reciprocal of the scaling coefficient corresponding to the fifth matrix again, so that the data in the operation result of the matrix multiplication operation is converted to the value range that can be represented by the first data type.

[0077] Referring to FIG. 6, FIG. 6 is a schematic diagram of a second calculation unit provided by the present application. The second calculation unit includes a maximum value calculation unit and a second multiplier. The input of the second calculation unit is a matrix and the maximum value Vmax The output is the scaling factor and the transformation matrix corresponding to the matrix. After the matrix is ​​input into the second calculation unit, the maximum value calculation unit first determines the maximum value in the matrix based on the data included in the matrix, denoted as T. max The second calculation unit obtains the maximum value V that the second data type can represent. max T is multiplied by the second multiplier max The reciprocal and maximum value V max The first multiplier performs a multiplication operation to obtain the scaling factor S corresponding to the matrix. The second multiplier then multiplies the matrix with the scaling factor S to obtain the transformation matrix corresponding to the matrix.

[0078] Optionally, as shown in Figure 6, the second calculation unit further includes a reciprocal calculation unit, which is used to calculate T. max Taking the reciprocal, we get 1 / T max .

[0079] Optionally, the second multiplier described above can be implemented using two multipliers, one of which is used to implement T. max The reciprocal and maximum value V max One multiplier performs the multiplication operation, while another multiplier is used to perform the multiplication operation between the matrix and the scaling factor S.

[0080] It should be understood that this application can also be implemented through a combination of hardware and software, using software instructions to determine the scaling factors and perform scaling operations on the matrix based on the scaling factors to obtain the transformation matrix corresponding to each matrix.

[0081] It should be noted that if matrix B in Figure 2 is directly transposed to obtain the transpose matrix B... T Then use the transpose matrix B T When performing matrix multiplication, since each data point in a row of matrix B corresponds to a scaling factor, a row of data in matrix B is transformed into the transpose of matrix B. T If a column of data is given, then the transpose matrix B is obtained. T Each data point in a column of data corresponds to a scaling factor, just as each value in the vector y corresponds to a scaling factor. To convert the result of the vector dot product into high-precision data, the product output by each multiplier in the vector calculation unit must first be multiplied by the reciprocal of the corresponding scaling factor before being input into the adder for addition. The calculation unit that performs matrix multiplication of two matrices is shown in Figure 7, which is a schematic diagram of a third calculation unit provided in this application. The third calculation unit includes a vector calculation unit, a first multiplication array, a first adder, and a second adder. The first multiplication array includes multiple multipliers, and one multiplier in the first multiplication array is used to multiply the value output by one multiplier in the vector calculation unit by the reciprocal of the corresponding scaling factor.

[0082] For example, if in a once vector inner product operation, vector x = (a 15 , a 16 , a 17 , a 18 ), vector y = (w 51 , w 61 , w 71 , w 81 ), the scaling factor corresponding to the four values in vector x is S A1 , and the scaling factor corresponding to the four values in vector y is S 11 , S 21 , S 31 and S 41 respectively. As shown in FIG. 7, after the vector calculation unit calculates the product a 15 x w 51 , the product needs to be input into the first multiplication array and multiplied by 1 / S 11 x (1 / S A1 ) to obtain the result, and then input into the first adder, where the functions of the first adder and the second adder are the same as in FIG. 5 and will not be described again. Through FIGS. 5 and 7 and the related description, FIG. 5 only needs one multiplier in addition to the multiplication array in the vector calculation unit, while FIG. 7 needs one multiplication array in addition to the multiplication array in the vector calculation unit, and the calculation unit in FIG. 6 is more complex, and through the implementation of the matrix multiplication in FIG. 7, the calculation amount of the first multiplication array in FIG. 7 is k times that of the first multiplier in FIG. 5, and relative to directly using two high-precision matrices to implement matrix multiplication, the corresponding scheme in FIG. 7 has lower benefits.

[0083] It should be understood that the above embodiments take the weight matrix as an example to introduce the method when a low-precision transpose matrix of a matrix needs to be used for matrix multiplication, and other transpose matrices of other matrices may also be used for matrix multiplication and other operations in the forward propagation training and the backward propagation training of the AI model. If a matrix F is a corresponding low-precision matrix G obtained according to the methods in S201 and S202, when the low-precision transpose matrix corresponding to the matrix F needs to be used, the transpose matrix corresponding to the matrix G can be directly used.

[0084] The application also provides a chip system, which comprises at least one first computing unit and at least one second computing unit, and the at least one second computing unit is capable of simultaneously computing scaling coefficients and conversion matrices corresponding to a plurality of sub-matrices. The matrix multiplication between two sub-matrices can also be completed by a plurality of first computing units. For example, the matrix multiplication between the above-mentioned matrix A1 and the matrix W1 is implemented by two first computing units, each of which completes the computation of m / 2 rows in the matrix A1 and m columns in the matrix W1, and then the results of the two first computing units are combined to obtain the computation result of the matrix multiplication between the matrix A1 and the matrix W1.

[0085] The application can complete the matrix multiplication between two matrices by one or more chips. For example, in the above-mentioned formula 1, the matrix multiplication between the second matrix A and the fifth matrix W needs to perform 12 sets of matrix multiplication between sub-matrices, and the matrix multiplication between the second matrix A and the fifth matrix W can be completed by 6 chips, as shown in FIG. 8, which is a schematic diagram for implementing the matrix multiplication provided by the application. When the matrix multiplication between the second matrix A and the fifth matrix W is completed by 6 chips, each chip completes the matrix multiplication between two sub-matrices in the second matrix A and two sub-matrices in the fifth matrix W. As shown in FIG. 8, for the second matrix A and the fifth matrix W, A1×W1+A2×W2 is allocated to a chip to complete, and the chip completes the matrix multiplication between the two sets of sub-matrices by a plurality of second computing units included therein. After the 12 sets of sub-matrices in the formula 1 are respectively allocated to the 6 chips, each chip completes the matrix multiplication between the two sets of sub-matrices allocated thereto, for example, chip 1 completes the operation of A1×W1+A2×W2 to obtain the matrix O1, and finally obtains the result of the matrix multiplication between the first matrix and the second matrix, i.e., the matrix O.

[0086] It should be understood that the structure of the first computing unit and the second computing unit in the above-mentioned chip system is only an example and cannot be construed as a specific limitation, and other reasonable hardware structures that can be conceived by those skilled in the art based on the above description also belong to the protection scope of the application.

[0087] Referring to FIG. 9, which is a schematic diagram of a computing device provided by the application, as shown in FIG. 9, the computing device 900 comprises a bus 902, a processor 904, a chip system 906, a communication interface 908 and a memory 910. The processor 904, the chip system 906, the communication interface 908 and the memory 910 communicate with each other through the bus 902. It should be understood that the number of processors, chip systems and memories in the computing device 900 is not limited by the application. The computing device can be a server, for example, a server in a cloud data center, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a desktop computer, a notebook computer or other terminal device.

[0088] Bus 902 can be a peripheral component interconnect express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), or the like. The bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one line is shown in FIG. 9, but it does not mean that there is only one bus or only one type of bus. Bus 902 can include a path for transmitting information between various components (e.g., processor 904, chip system 906, communication interface 908, and memory 910) of computing device 900. The unified bus can also be referred to as a coherent bus.

[0089] Processor 904 can be a central processing unit (CPU), and can also include a CPU and other hardware chips. The hardware chips can be of various types, such as any of programmable logic device (PLD), complex programmable logic device (CPLD), field programmable gate array (FPGA), digital signal processor (DSP), or the like. Computing device 900 can include one or more hardware chips of any of the above types, or can include multiple hardware chips of various types, and the embodiments of the present application are not limited in this regard.

[0090] The chip system 906 can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other special AI processing chip, etc., and the present application does not make specific limitations. The chip system includes at least one first computing unit shown in FIG. 5 and at least one second computing unit shown in FIG. 6. The chip system 906 is used to perform the operations performed by the model training system in the method embodiment corresponding to FIG. 3, and realizes the AI model training method provided by the present application, which will not be repeated here.

[0091] The memory 910 can be a non-volatile memory, for example, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The memory 910 can also be a volatile memory, which can be a random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). In addition, the memory 910 can also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0092] It should be noted that the same type of storage medium can be configured to implement the function of the memory 910 in the same computing device, or two or more types of storage media can be configured to implement the function of the memory 910, and the present application does not make limitations.

[0093] The executable program code stored in the memory 910 is executed by the processor 904 to issue a computing task to the chip system 906, so that the chip system 906 schedules the first computing unit and the second computing unit to implement the operations performed in the method embodiment shown in FIG. 3.

[0094] The communication interface 908 uses a transceiving module such as, but not limited to, a network interface card and a transceiver to implement communication between the computing device 900 and other devices or communication networks.

[0095] The present application also provides a computer program product containing instructions, which can be a software or program product containing instructions capable of running on a computing device or stored in any available medium. When the computer program product runs on the computing device, it causes the computing device to perform the model training method in the method embodiment corresponding to FIG. 3 described above.

[0096] The present application also provides a computer readable storage medium, which can be any available medium capable of being stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium includes instructions instructing the computing device to perform the model training method in the method embodiment corresponding to FIG. 3 described above.

[0097] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training an artificial intelligence (AI) model, the method comprising: The method is applied to a model training system, and comprises the following steps: In a forward propagation training phase of an AI model, a first matrix is obtained, the first matrix being activation value data or weight data in the AI model; A scaling coefficient corresponding to each of k m*m sub-matrices included in the first matrix is determined, to obtain k scaling coefficients; wherein k and m are positive integers greater than 1, and one scaling coefficient is used to convert data in a corresponding sub-matrix from a first data type to a second data type, the first data type having higher precision than the second data type; The first matrix is converted into a second matrix according to the scaling coefficient corresponding to each of the sub-matrices, and forward propagation training of the AI model is performed according to the second matrix and the k scaling coefficients; In a backward propagation training phase of the AI model, backward propagation training of the AI model is performed according to a transposed matrix of the second matrix and the k scaling coefficients.

2. The method of claim 1, wherein, The second matrix includes a converted sub-matrix corresponding to each of the sub-matrices in the first matrix, a first converted sub-matrix in the second matrix being a converted sub-matrix corresponding to a first sub-matrix in the first matrix, and a scaling coefficient corresponding to the first converted sub-matrix being a first scaling coefficient corresponding to the first sub-matrix; A scaling coefficient corresponding to a second sub-matrix in the transposed matrix of the second matrix is the scaling coefficient corresponding to the first converted sub-matrix, and the second sub-matrix and the first converted sub-matrix include the same data.

3. The method of claim 2, wherein, The model training system comprises a first computing unit; The backward propagation training of the AI model according to the transposed matrix of the second matrix and the k scaling coefficients comprises the following steps: The first computing unit performs matrix multiplication according to the transposed matrix of the second matrix, the k scaling coefficients and a third matrix, to obtain an intermediate result of the backward propagation, and performs the backward propagation according to the intermediate result; wherein data in the third matrix is of the second data type, and the third matrix is gradient data in the AI model.

4. The method of claim 3, wherein, The first computing unit comprises a vector computing unit, a first adder and a first multiplier; The first computing unit performs matrix multiplication according to the transposed matrix of the second matrix, the k scaling coefficients and a third matrix, to obtain an intermediate result of the backward propagation, and comprises the following steps: The vector computing unit obtains a first vector and a second vector, and performs vector inner product operation to obtain an output result, the output result comprising a plurality of products obtained by multiplying corresponding elements in the first vector and the second vector, the first vector being one row of data in one sub-matrix in the third matrix, and the second vector being one column of data in one sub-matrix in the transposed matrix of the second matrix; The first adder performs addition calculation on the output result of the vector computing unit, to obtain an operation result of the vector inner product operation; The first multiplier obtains one numerical value in the intermediate result according to the inner product operation result and a second scaling coefficient, and obtains the intermediate result according to the one numerical value in the intermediate result; the second scaling coefficient being a scaling coefficient corresponding to a sub-matrix to which the second vector belongs.

5. The method according to claim 3 or 4, characterized in that, The model training system further comprises a second computing unit; The determining of the scaling coefficient corresponding to each of the k m*m sub-matrices included in the first matrix comprises: The second computing unit acquires the first sub-matrix and determines the maximum value in the first sub-matrix; the first sub-matrix is any one of the k sub-matrices; The second computing unit determines the first scaling coefficient corresponding to the first sub-matrix according to the maximum value in the first sub-matrix and the maximum value that can be represented by the second data type.

6. The method of claim 5, wherein, The conversion of the first matrix into a second matrix according to the scaling coefficient corresponding to each sub-matrix comprises: The second computing unit multiplies the first sub-matrix by the first scaling coefficient to obtain a first converted sub-matrix corresponding to the first sub-matrix, and obtains the second matrix according to the first converted sub-matrix.

7. The method according to any one of claims 1 to 6, characterized in that, Before the determining of the scaling coefficient corresponding to each of the k m*m sub-matrices included in the first matrix, the method further comprises: Converting a fourth matrix into an n*n matrix to obtain the first matrix; wherein n is an integer multiple of m.

8. A chip system, characterized by The chip system comprises at least one first computing unit and at least one second computing unit, wherein, The second computing unit is configured to, in a forward propagation training phase of an AI model, acquire a first matrix, determine a scaling coefficient corresponding to each of k m*m sub-matrices included in the first matrix, and obtain k scaling coefficients; wherein k and m are positive integers greater than 1, one scaling coefficient is used to convert data in a corresponding sub-matrix from a first data type to a second data type, the precision of the first data type is higher than that of the second data type, and the first matrix is activation value data or weight data in the AI model; The second computing unit is further configured to convert the first matrix into a second matrix according to the scaling coefficient corresponding to each sub-matrix; The first computing unit is configured to perform forward propagation training of the AI model according to the second matrix and the k scaling coefficients; The first computing unit is further configured to, in a backward propagation training phase of the AI model, perform backward propagation training of the AI model according to a transposed matrix of the second matrix and the k scaling coefficients.

9. The chip system according to claim 8, characterized by The second matrix includes a converted sub-matrix corresponding to each sub-matrix in the first matrix, a first converted sub-matrix in the second matrix is a converted sub-matrix corresponding to a first sub-matrix in the first matrix, and a scaling coefficient corresponding to the first converted sub-matrix is a first scaling coefficient corresponding to the first sub-matrix; A scaling coefficient corresponding to a second sub-matrix in the transposed matrix of the second matrix is a scaling coefficient corresponding to the first converted sub-matrix, and the second sub-matrix and the first converted sub-matrix include the same data.

10. The chip system according to claim 9, characterized by The first computing unit is specifically configured to: The matrix multiplication operation is performed according to a transpose matrix of the second matrix, the k scaling coefficients and a third matrix, to obtain an intermediate result of the back propagation, and the back propagation is performed according to the intermediate result; wherein data in the third matrix is of the second data type, and the third matrix is gradient data in the AI model.

11. The chip system according to claim 10, characterized by The first calculation unit comprises a vector calculation unit, an adder and a first multiplier; wherein The vector calculation unit is configured to obtain a first vector and a second vector, and perform a vector inner product operation to obtain an output result, the output result comprising a plurality of products obtained by multiplying corresponding elements in the first vector and the second vector, the first vector being a row of data in a submatrix in the third matrix, and the second vector being a column of data in a submatrix in the transpose matrix of the second matrix; The adder is configured to perform addition calculation on the output result of the vector calculation unit to obtain an operation result of the vector inner product operation; The first multiplier is configured to obtain one numerical value in the intermediate result according to the inner product operation result and a second scaling coefficient, and obtain the intermediate result according to the one numerical value in the intermediate result; the second scaling coefficient being a scaling coefficient corresponding to a submatrix to which the second vector belongs.

12. The chip system according to any one of claims 9 to 11, characterized in that The second calculation unit is specifically configured to: obtain the first submatrix and determine a maximum value in the first submatrix; the first submatrix being any one of the k submatrices; determine a first scaling coefficient corresponding to the first submatrix according to the maximum value in the first submatrix and a maximum value representable by the second data type.

13. The chip system according to claim 12, characterized by The second calculation unit comprises a maximum value calculation unit and a second multiplier; wherein The maximum value calculation unit is configured to obtain the first submatrix and determine a maximum value in the first submatrix; The second multiplier is configured to determine a first scaling coefficient corresponding to the first submatrix according to the maximum value in the first submatrix and a maximum value representable by the second data type; The second multiplier is further configured to multiply the first submatrix by the first scaling coefficient to obtain a first conversion submatrix corresponding to the first submatrix, and obtain the second matrix according to the first conversion submatrix.

14. A computing device, comprising: The chip system and the memory as claimed in any one of claims 8-13, wherein the memory is configured to store data in the AI model.

15. A computer readable storage medium, characterized in that, The computer program instructions, when executed by a computing device, cause the computing device to perform the method as claimed in any one of claims 1-7.

Images