Matrix multiplier optimization method and device, storage medium and program product

Abstract

The embodiment of the invention provides an optimization method and device for a matrix multiplier, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: determining target layout modes of a plurality of input matrixes of the matrix multiplier in a memory and a target multiplication mode of the matrix multiplier before the matrix multiplier is executed, therefore, the target layout mode and the target multiplication mode support that a plurality of input matrixes are read from a memory in a continuous memory access mode in the execution process of the matrix multiplier. In this way, in the execution process of the matrix multiplier, the artificial intelligence chip can read a plurality of input matrixes in a continuous memory access mode (namely, read data continuously stored in a memory) to carry out matrix multiplication calculation; compared with the reading of non-continuous storage data, the reading performance of the artificial intelligence chip on the continuous storage data is higher, so that the data reading efficiency in the execution process of the matrix multiplier is effectively improved, and the calculation efficiency of the matrix multiplier is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an optimization method, device, storage medium, and program product for a matrix multiplication operator. Background Art

[0002] Artificial intelligence models usually refer to neural network models that are trained for inference and prediction, such as image inference models, speech inference models, etc. The operations of artificial intelligence models can be implemented by operators in the computation graph. Operators refer to various operations performed on tensors at each layer in the artificial intelligence model. Matrix multiplication operators are common and time-consuming operators in artificial intelligence models. Accelerating the calculation of matrix multiplication operators can accelerate the training efficiency and inference efficiency of artificial intelligence models.

[0003] Under the relevant technology, when an AI chip executes a matrix multiplication operator, if the requested data is discretely distributed in the memory, the AI ​​chip needs to read the data that is discretely distributed (i.e., non-continuously stored) in the memory. Since the AI ​​chip is less efficient when reading non-continuously stored data from the memory, the performance of the matrix multiplication operator will be reduced. Summary of the invention

[0004] The embodiments of the present application provide a matrix multiplication operator optimization method, device, storage medium and program product, which are used to increase the speed of reading data from the memory during the calculation process of the matrix multiplication operator, thereby improving the performance of the matrix multiplication operator.

[0005] On the one hand, an embodiment of the present application provides a method for optimizing a matrix multiplication operator, including:

[0006] Before executing a matrix multiplication operator, determining a target layout mode of each of a plurality of input matrices of the matrix multiplication operator in a memory and a target multiplication mode of the matrix multiplication operator;

[0007] The target layout mode and the target multiplication mode support reading the multiple input matrices from the memory in a continuous memory access mode during the execution of the matrix multiplication operator.

[0008] On the one hand, an embodiment of the present application provides an optimization device for a matrix multiplication operator, including:

[0009] A configuration module, used for determining, before executing a matrix multiplication operator, a target layout mode of each of a plurality of input matrices of the matrix multiplication operator in a memory and a target multiplication mode of the matrix multiplication operator;

[0010] The target layout mode and the target multiplication mode support reading the multiple input matrices from the memory in a continuous memory access mode during the execution of the matrix multiplication operator.

[0011] Optionally, the multiple input matrices include: a left matrix and a right matrix; the target layout mode of each of the multiple input matrices and the target multiplication mode of the matrix multiplication operator are any one of the following multiple forms:

[0012] The target layout mode of the left matrix is ​​row-major order, the target layout mode of the right matrix is ​​column-major order, and the target multiplication mode is inner product mode; or,

[0013] The target layout mode of the left matrix is ​​a column-major order mode, the target layout mode of the right matrix is ​​a row-major order mode, and the target multiplication mode is an outer product mode.

[0014] Optionally, the configuration module is specifically used to:

[0015] Obtaining a target layout mode of each of a plurality of input matrices of the matrix multiplication operator in the memory;

[0016] A target multiplication method matching the target layout methods of the plurality of input matrices is selected from the plurality of candidate multiplication methods.

[0017] Optionally, the multiple candidate multiplication modes include: an inner product mode and an outer product mode; the multiple input matrices include a left matrix and a right matrix;

[0018] The configuration module is specifically used for:

[0019] When the target layout mode of the left matrix is ​​a row-major order mode, and the target layout mode of the right matrix is ​​a column-major order mode, selecting the inner product mode as the target multiplication mode;

[0020] When the target layout mode of the left matrix is ​​a column-major order mode, and the target layout mode of the right matrix is ​​a row-major order mode, the outer product mode is selected as the target multiplication mode.

[0021] Optionally, the configuration module is specifically used to:

[0022] Based on the parameter configuration information of the matrix multiplication operator, obtaining a target multiplication mode of the matrix multiplication operator from a plurality of candidate multiplication modes;

[0023] A target layout mode of each of the plurality of input matrices is selected from a plurality of candidate layout modes based on the target multiplication mode.

[0024] Optionally, the parameter configuration information of the matrix multiplication operator includes: shape information of each of the multiple input matrices;

[0025] The configuration module is specifically used for:

[0026] For the multiple candidate multiplication methods, the following operations are respectively performed: based on the shape information of each of the multiple input matrices, an estimated processing performance corresponding to when a candidate multiplication method is used to perform matrix multiplication calculations on the multiple input matrices;

[0027] Based on the estimated processing performance corresponding to each of the multiple candidate multiplication modes, a target multiplication mode of the matrix multiplication operator is obtained from the multiple candidate multiplication modes.

[0028] Optionally, the configuration module is specifically used to:

[0029] When performing matrix multiplication calculation on the multiple input matrices using the one candidate multiplication mode, obtaining a first number of clock cycles required to load the multiple input matrices based on shape information of each of the multiple input matrices; and obtaining a second number of clock cycles required to calculate the multiple input matrices based on shape information of each of the multiple input matrices;

[0030] The maximum number of clock cycles between the first number of clock cycles and the second number of clock cycles is used as the estimated processing performance corresponding to the one candidate multiplication method.

[0031] Optionally, the multiple candidate layout modes include: a row-major order mode and a column-major order mode; the multiple input matrices include a left matrix and a right matrix;

[0032] The configuration module is specifically used for:

[0033] When the target multiplication mode is an inner product mode, the row-major order mode is used as the target layout mode of the left matrix, and the column-major order mode is used as the target layout mode of the right matrix;

[0034] When the target multiplication mode is the outer product mode, the column-major order mode is used as the target layout mode of the left matrix, and the row-major order mode is used as the target layout mode of the right matrix.

[0035] On the one hand, an embodiment of the present application provides a computer device, including a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, wherein the artificial intelligence chip implements the steps of the above-mentioned matrix multiplication operator optimization method when executing the computer program.

[0036] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned matrix multiplication operator optimization method.

[0037] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned matrix multiplication operator optimization method.

[0038] In an embodiment of the present application, before executing the matrix multiplication operator, the target layout mode of each of the multiple input matrices of the matrix multiplication operator in the memory and the target multiplication mode of the matrix multiplication operator are determined, so that the target layout mode and the target multiplication mode support the use of continuous memory access to read multiple input matrices from the memory during the execution of the matrix multiplication operator. In this way, during the execution of the matrix multiplication operator, the artificial intelligence chip can use a continuous memory access method to read multiple input matrices (i.e., read data stored continuously in the memory) to perform matrix multiplication calculations; compared to reading non-continuously stored data, the artificial intelligence chip has a higher reading performance for continuously stored data, which effectively improves the data reading efficiency during the execution of the matrix multiplication operator, thereby improving the calculation efficiency of the matrix multiplication operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of an input matrix storage method provided in an embodiment of the present application Figure 1 ;

[0042] Figure 3 A schematic diagram of an input matrix storage method provided in an embodiment of the present application Figure 2 ;

[0043] Figure 4 A schematic diagram of a process of inner product method provided in an embodiment of the present application Figure 1 ;

[0044] Figure 5A schematic diagram of a process of an outer product method provided in an embodiment of the present application Figure 1 ;

[0045] Figure 6 A schematic diagram of a process of inner product method provided in an embodiment of the present application Figure 2 ;

[0046] Figure 7 A schematic diagram of a process of inner product method provided in an embodiment of the present application Figure 2 ;

[0047] Figure 8 A schematic diagram of a process for optimizing a matrix multiplication operator provided in an embodiment of the present application Figure 2 ;

[0048] Fig. 9 A schematic diagram of a process for optimizing a matrix multiplication operator provided in an embodiment of the present application Figure 3 ;

[0049] Fig.10 A schematic diagram of the structure of an optimization device for a matrix multiplication operator provided in an embodiment of the present application;

[0050] Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] refer to Figure 1 , which is a structural diagram of an artificial intelligence chip applicable to an embodiment of the present application. The artificial intelligence chip 100 includes at least: a video memory 101, an on-chip cache 102 and multiple computing units 103.

[0053] The video memory 101 may be a high bandwidth memory (HBM) or other types of memory. The on-chip cache 102 is a temporary memory, which has a smaller capacity than the video memory 101 but has a faster data reading and writing speed than the video memory 101.

[0054] The computing unit 103 may be used to execute various types of operators, including but not limited to: a convolution operator, an attention mechanism operator, and a matrix multiplication operator.

[0055] The computing unit 103 can read and write data to the video memory 101, and can also read and write data to the on-chip cache 102. For example, the computing unit 103 can read data from the video memory 101 for calculation; it can also read data from the video memory 101 and then write the read data to the on-chip cache 102. If the subsequent calculation process needs to use the data multiple times, the data can be read from the on-chip cache 102 for calculation multiple times. Compared with reading the data from the video memory 101 multiple times, reading the data from the on-chip cache 102 multiple times can greatly improve the efficiency of data reading.

[0056] In addition to the above-mentioned structure, the artificial intelligence chip 100 in the present application may also include other structures, which are not specifically limited in the present application.

[0057] The artificial intelligence chip 100 may be: a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a domain specific architecture (DSA), etc.

[0058] In practical applications, the matrix multiplication operator is a common and time-consuming operator in artificial intelligence models. Accelerating the calculation of the matrix multiplication operator can accelerate the training efficiency and reasoning efficiency of the artificial intelligence model.

[0059] Under the relevant technology, when an AI chip executes a matrix multiplication operator, if the requested data is discretely distributed in the memory, the AI ​​chip needs to read the data that is discretely distributed (i.e., non-continuously stored) in the memory. Since the AI ​​chip is less efficient when reading non-continuously stored data from the memory, the performance of the matrix multiplication operator will be reduced.

[0060] In view of this, the present application provides an optimization method for a matrix multiplication operator, which can be applied to various scenarios. For example, image processing scenarios, speech processing scenarios, text processing scenarios, etc. In different application scenarios, the physical meanings of the input matrix and output matrix of the matrix multiplication operator may be different.

[0061] For example, in a speech processing scenario, the input matrix and the output matrix can be speech data used in tasks such as speech enhancement, speech recognition, and speech synthesis. For example, in an image processing scenario, the input matrix and the output matrix can be image data used in tasks such as image preprocessing, image segmentation, and object detection. For example, in a text processing scenario, the input matrix and the output matrix can be text data used in tasks such as text generation and text recognition.

[0062] The following specifically introduces the process of the optimization method of the matrix multiplication operator. The process of the method is executed by a computer device, and the computer device includes Figure 1 The artificial intelligence chip shown includes the following steps:

[0063] Before executing a matrix multiplication operator, determine the target layout of each of the multiple input matrices of the matrix multiplication operator in memory and the target multiplication method of the matrix multiplication operator.

[0064] Specifically, the target layout of the input matrix in the memory refers to: the storage method of the input matrix in the memory, where the memory can be Figure 1 The video memory 101 or on-chip cache 102 shown may, of course, be other memories.

[0065] A two-dimensional matrix is ​​usually stored in a linear layout in memory. Specifically, the target layout can be a row-major order or a column-major order.

[0066] The row-major order means that the input matrix is ​​stored row by row in the memory; the column-major order means that the input matrix is ​​stored column by column in the memory.

[0067] For example, the input matrix is ​​a 2×3 matrix, and the six elements included in the input matrix are: 1, 2, 3, 4, 5, and 6 respectively.

[0068] See also Figure 2 , when the input matrix is ​​stored in memory in row-major order, the elements stored consecutively in the memory are: 1, 2, 3, 4, 5, 6.

[0069] See also Figure 3 , when the input matrix is ​​stored in memory in column-major order, the elements stored consecutively in the memory are: 1, 4, 2, 5, 3, 6.

[0070] In some embodiments, the plurality of input matrices include: a left matrix and a right matrix, wherein the left matrix refers to a matrix located on the left side of the multiplication operator, and the right matrix refers to a matrix located on the right side of the multiplication operator.

[0071] For example, suppose A is an m×k input matrix and B is a k×n input matrix. The product of input matrix A and input matrix B is an m×n output matrix C, where input matrix A is the left matrix and input matrix B is the right matrix, and m, n, and k are all positive integers.

[0072] The target multiplication method of the matrix multiplication operator is the inner product method or the outer product method.

[0073] When the inner product method is used to perform matrix multiplication calculation on the input matrix A and the input matrix B, the input matrix A can be first divided into m row vectors and the input matrix B can be divided into n column vectors. For each row vector of the m row vectors, the dot product of the row vector and each column vector of the n column vectors is calculated. In a specific implementation, the dot product calculation of a row vector and a column vector is specifically shown in the following formula (1):

[0074] a·b=a1b1+a2b2+…+a k b k ..........(1)

[0075] Where a represents a 1×k row vector, and the k components contained in the row vector a are specifically (a1, a2, …, a k ), a k is the kth component of the row vector a; b represents a k×1 column vector, and the k components contained in the column vector b are specifically (b1, b2, …, b k ), b k is the kth component of column vector b.

[0076] When the inner product method is used to perform matrix multiplication on input matrix A and input matrix B, it is necessary to perform m×n dot products of row vectors and column vectors on input matrix A and input matrix B to obtain output matrix C.

[0077] For example, see Figure 4 , set the input matrix A to be a 2×3 matrix, and the 6 elements included are: 1, 2, 3, 4, 5, 6. The input matrix B is a 3×2 matrix, and the 6 elements included are: 1, 2, 3, 4, 5, 6.

[0078] When the inner product method is used to perform matrix multiplication on input matrix A and input matrix B, the row vector corresponding to the first row of input matrix A (containing elements: 1, 2, 3) is first dot-producted with the column vector corresponding to the first column of input matrix B (containing elements: 1, 3, 5). The calculation result is 22, which is the element of the first row and first column of output matrix C.

[0079] The dot product of the row vector corresponding to the first row of the input matrix A (containing elements: 1, 2, 3) and the column vector corresponding to the second column of the input matrix B (containing elements: 2, 4, 6) is calculated, and the result is 28, which is the element of the first row and second column of the output matrix C.

[0080] By analogy, the element in the second row and first column of the output matrix C is 49, and the element in the second row and second column of the output matrix C is 64.

[0081] When the outer product method is used to perform matrix multiplication calculation on input matrix A and input matrix B, the input matrix A can be first divided into k column vectors, and the k column vectors are specifically: column vector 1, column vector 2, ..., column vector k. The input matrix B can be divided into k row vectors, and the k row vectors are specifically: row vector 1, row vector 2, ..., row vector k.

[0082] Multiply column vector 1 by row vector 1 to obtain an intermediate matrix; multiply column vector 2 by row vector 2 to obtain an intermediate matrix; and so on, until multiply column vector k by row vector k to obtain an intermediate matrix. In a specific implementation, the process of multiplying a column vector by a row vector to generate an intermediate matrix is ​​specifically shown in the following formula (2):

[0083]

[0084] Where c represents an m×1 column vector, and the m components contained in the column vector c are specifically (c1, c2, …, c m ), c m is the mth component of the column vector c; d represents a 1×n row vector, and the n components contained in the row vector d are specifically (d1, d2, …, d n ), d n is the nth component of the row vector d.

[0085] Since the input matrix A includes k column vectors and the input matrix B includes k row vectors, a total of k intermediate matrices can be obtained. Then, the obtained k intermediate matrices are added together to obtain the output matrix C.

[0086] For example, see Figure 5 , set the input matrix A to be a 2×3 matrix, and the 6 elements included are: 1, 2, 3, 4, 5, 6. The input matrix B is a 3×2 matrix, and the 6 elements included are: 1, 2, 3, 4, 5, 6.

[0087] When the outer product method is used to perform matrix multiplication on input matrix A and input matrix B, the column vector corresponding to the first column of input matrix A (containing elements: 1, 4 respectively) is first multiplied with the row vector corresponding to the first row of input matrix B (containing elements: 1, 2 respectively) to obtain a 2×2 intermediate matrix 1. The intermediate matrix 1 includes 4 elements: 1, 2, 4, 8 respectively.

[0088] Multiply the column vector corresponding to the second column of the input matrix A (containing elements: 2, 5) with the row vector corresponding to the second row of the input matrix B (containing elements: 3, 4) to obtain a 2×2 intermediate matrix 2. The intermediate matrix 2 includes 4 elements: 6, 8, 15, and 20.

[0089] Multiply the column vector corresponding to the third column of the input matrix A (containing elements: 3, 6 respectively) with the row vector corresponding to the third row of the input matrix B (containing elements: 5, 6 respectively) to obtain a 2×2 intermediate matrix 3. The intermediate matrix 3 includes 4 elements: 15, 18, 30, and 36 respectively.

[0090] Add the elements in the first row and first column of intermediate matrix 1, the elements in the first row and first column of intermediate matrix 2, and the elements in the first row and first column of intermediate matrix 3 to obtain an output matrix C whose first row and first column element is 22.

[0091] Using the same method, the element in the first row and second column of the output matrix C is 28, the element in the second row and first column of the output matrix C is 48, and the element in the second row and second column of the output matrix C is 64.

[0092] Before executing a matrix multiplication operator, a target layout mode of each of the multiple input matrices of the matrix multiplication operator in the memory and a target multiplication mode of the matrix multiplication operator are determined for the purpose of realizing a continuous memory access mode for reading multiple input matrices from the memory during the execution of the matrix multiplication operator.

[0093] During the execution of the matrix multiplication operator, the target layout mode and the target multiplication mode determine the way to read multiple input matrices from the memory. After the target layout mode and the target multiplication mode are determined, the matrix multiplication operator is executed based on the target layout mode and the target multiplication mode (i.e., matrix multiplication calculation is performed on multiple input matrices) to obtain an output matrix.

[0094] It should be noted that in the present application, the above process of obtaining the target layout mode and the target multiplication mode is performed by the central processing unit (CPU) in the computer device. After determining the target layout mode and the target multiplication mode, the artificial intelligence chip in the computer device executes the matrix multiplication operator (i.e., performs matrix multiplication calculation on multiple input matrices) based on the target layout mode and the target multiplication mode to obtain the output matrix.

[0095] In some embodiments, when the target layout mode of each of the multiple input matrices and the target multiplication mode of the matrix multiplication operator are any one of the following forms, the artificial intelligence chip can use a continuous memory access method to read the multiple input matrices from the memory when performing matrix multiplication.

[0096] Form 1: The target layout mode of the left matrix is ​​row-major order, the target layout mode of the right matrix is ​​column-major order, and the target multiplication mode is inner product mode.

[0097] Specifically, since the target multiplication mode is the inner product mode, when executing the matrix multiplication operator, it is necessary to read the elements of the left matrix from the memory row by row, and read the elements of the right matrix from the memory column by column. The target layout mode of the left matrix is ​​the row-major order mode, that is, the left matrix is ​​stored row by row in the memory; therefore, when executing the matrix multiplication operator, the left matrix can be read from the memory in a continuous memory access mode. Similarly, the target layout mode of the right matrix is ​​the column-major order mode, that is, the right matrix is ​​stored column by column in the memory. Therefore, when executing the matrix multiplication operator, the right matrix can be read from the memory in a continuous memory access mode.

[0098] For example, see Figure 6 , set the input matrix A to be a 2×3 matrix, and the 6 elements included are: 1, 2, 3, 4, 5, 6. The input matrix B is a 3×2 matrix, and the 6 elements included are: 1, 2, 3, 4, 5, 6.

[0099] The input matrix A is stored in memory in row-major order, that is, the elements stored consecutively in the memory are: 1, 2, 3, 4, 5, 6.

[0100] The input matrix B is stored in the memory in column-major order, that is, the elements stored consecutively in the memory are: 1, 3, 5, 2, 4, 6.

[0101] When the inner product method is used to perform matrix multiplication of input matrix A and input matrix B, the row vector 1 (containing elements 1, 2, and 3) corresponding to the first row of input matrix A is first read from the memory. Since the three elements contained in row vector 1 are stored continuously in the memory, the row vector 1 is read from the memory using the continuous memory access method.

[0102] Read the column vector 1 (containing elements 1, 3, and 5) corresponding to the first column of the input matrix B from the memory. Since the three elements contained in the column vector 1 are also stored continuously in the memory, the column vector 1 is read from the memory using the continuous memory access method.

[0103] Then, dot product calculation is performed on row vector 1 and column vector 1, and the result is 22, which is the element of the first row and first column of the output matrix C.

[0104] The other elements of the output matrix C can be obtained in the same manner as above, which will not be described here.

[0105] Form 2: The target layout mode of the left matrix is ​​column-major order, the target layout mode of the right matrix is ​​row-major order, and the target multiplication mode is outer product.

[0106] Specifically, since the target multiplication mode is the outer product mode, when executing the matrix multiplication operator, it is necessary to read the elements of the left matrix from the memory column by column, and read the elements of the right matrix from the memory row by row. The target layout mode of the left matrix is ​​the column-major order mode, that is, the left matrix is ​​stored column by column in the memory; therefore, when executing the matrix multiplication operator, the left matrix can be read from the memory in a continuous memory access mode. Similarly, the target layout mode of the right matrix is ​​the row-major order mode, that is, the right matrix is ​​stored row by row in the memory. Therefore, when executing the matrix multiplication operator, the right matrix can be read from the memory in a continuous memory access mode.

[0107] For example, see Figure 7 , set the input matrix A to be a 2×3 matrix, and the 6 elements included are: 1, 2, 3, 4, 5, 6. The input matrix B is a 3×2 matrix, and the 6 elements included are: 1, 2, 3, 4, 5, 6.

[0108] The input matrix A is stored in the memory in column-major order, that is, the elements of the input matrix A stored consecutively in the memory are: 1, 4, 2, 5, 3, 6.

[0109] The input matrix B is stored in the memory in row-major order, that is, the elements of the input matrix B stored consecutively in the memory are: 1, 2, 3, 4, 5, 6.

[0110] When the outer product method is used to perform matrix multiplication of input matrix A and input matrix B, the column vector 1 corresponding to the first column of input matrix A is first read from the memory (the elements contained are: 1 and 4). Since the two elements contained in column vector 1 are stored continuously in the memory, the column vector 1 is read from the memory using the continuous memory access method.

[0111] Read row vector 1 (containing elements 1 and 2) corresponding to the first row of input matrix B from the memory. Since the two elements contained in row vector 1 are also stored continuously in the memory, row vector 1 is read from the memory using continuous memory access.

[0112] Then multiply column vector 1 by row vector 1 to obtain intermediate matrix 1.

[0113] The intermediate matrix 2 and the intermediate matrix 3 can be obtained by using the same method as above, which will not be described here. Then, the intermediate matrix 1, the intermediate matrix 2 and the intermediate matrix 3 are added together to obtain the output matrix C.

[0114] In an embodiment of the present application, before executing the matrix multiplication operator, the target layout mode of each of the multiple input matrices of the matrix multiplication operator in the memory and the target multiplication mode of the matrix multiplication operator are determined, so that the target layout mode and the target multiplication mode support the use of continuous memory access to read multiple input matrices from the memory during the execution of the matrix multiplication operator. In this way, during the execution of the matrix multiplication operator, the artificial intelligence chip can use a continuous memory access method to read multiple input matrices (i.e., read data stored continuously in the memory) to perform matrix multiplication calculations; compared to reading non-continuously stored data, the artificial intelligence chip has a higher reading performance for continuously stored data, which effectively improves the data reading efficiency during the execution of the matrix multiplication operator, thereby improving the calculation efficiency of the matrix multiplication operator.

[0115] In some embodiments, in an eager mode, a target layout of each of the multiple input matrices of a matrix multiplication operator in memory is obtained, and then a target multiplication method matching the target layout of each of the multiple input matrices is selected from multiple candidate multiplication methods.

[0116] Specifically, the dynamic graph mode is an imperative programming mode. In this mode, the computational graph used to represent each operator to be executed is not defined. Instead, when building the model framework, for each operator required by the model framework, the input tensor is first defined, and then the operator that calculates the input tensor is defined (that is, the specific operation performed on the input tensor), and then the operator is compiled separately. During the execution process, the input tensor is first clarified, and then the corresponding operator is called according to the program logic to calculate the input tensor.

[0117] Then, in the dynamic graph mode, for the matrix multiplication operator, we first define the multiple input matrices that need to perform matrix multiplication calculations and the target layout of the multiple input matrices in memory, then define the matrix multiplication operator that performs matrix multiplication calculations on the multiple input matrices, and then compile the matrix multiplication operator separately.

[0118] Since the target layout mode and the target multiplication mode jointly determine the method of reading multiple input matrices from the memory, when defining a matrix multiplication operator that performs matrix multiplication calculations on multiple input matrices, a matching target multiplication mode can be selected based on the target layout modes of the multiple input matrices, so that the target layout mode and the target multiplication mode support reading multiple input matrices from the memory using a continuous memory access method during the execution of the matrix multiplication operator.

[0119] Since each operator is compiled separately, the shape information of the input matrices of each operator in the entire calculation graph, the dependencies between operators, etc. are not clear during the compilation stage. Therefore, if the input, output, calculation method, etc. of an operator are adjusted during the compilation stage, it may affect the calculation of other operators in the calculation graph. Based on this, when the matrix multiplication operator is compiled separately, the target layout method of each of the defined multiple input matrices and the matching target multiplication method cannot be adjusted.

[0120] In some embodiments, the multiple candidate multiplication methods include: an inner product method and an outer product method; the multiple input matrices include a left matrix and a right matrix. On this basis, the present application at least adopts the following methods to obtain the target multiplication method, see Figure 8 , including the following steps:

[0121] Step 801, obtaining the target layout mode of the left matrix.

[0122] Step 802, obtaining the target layout mode of the right matrix.

[0123] Step 803, determine whether the left matrix is ​​in row-major order and the right matrix is ​​in column-major order. If so, execute step 804, otherwise execute step 805.

[0124] Step 804, select the inner product method as the target multiplication method, and then execute step 807.

[0125] Specifically, when the target layout mode of the left matrix is ​​row-major order (i.e., the left matrix is ​​stored row by row in the memory) and the target layout mode of the right matrix is ​​column-major order (i.e., the right matrix is ​​stored column by column in the memory), the selected target multiplication mode needs to satisfy the requirement of reading the elements of the left matrix row by row from the memory and the elements of the right matrix column by column from the memory, so that the left matrix and the right matrix can be read in a continuous memory access mode. The multiplication mode that satisfies the requirement of reading the elements of the left matrix row by row from the memory and the elements of the right matrix column by column from the memory is the inner product mode, so the inner product mode is selected as the target multiplication mode.

[0126] Step 805, determine whether the left matrix is ​​in column-major order and the right matrix is ​​in row-major order. If so, execute step 806, otherwise end.

[0127] Step 806, select the outer product method as the target multiplication method, and then execute step 807.

[0128] Specifically, when the target layout mode of the left matrix is ​​column-major order (i.e., the left matrix is ​​stored column by column in the memory) and the target layout mode of the right matrix is ​​row-major order (i.e., the right matrix is ​​stored row by row in the memory), the selected target multiplication mode needs to satisfy the requirement of reading the elements of the left matrix column by column from the memory and the elements of the right matrix row by row from the memory, so that the left matrix and the right matrix can be read in a continuous memory access mode. The multiplication mode that satisfies the requirement of reading the elements of the left matrix column by column from the memory and the elements of the right matrix row by row from the memory is the outer product mode, so the outer product mode is selected as the target multiplication mode.

[0129] Step 807, output the target multiplication method.

[0130] It should be noted that when the target layout modes of the left matrix and the right matrix are both row-major, a layout adjustment operator can be inserted before the matrix multiplication operator. The layout adjustment operator is used to first adjust the target layout mode of the right matrix from row-major to column-major, and then the inner product mode is selected as the target multiplication mode. Of course, the layout adjustment operator can also be used to first adjust the target layout mode of the left matrix from row-major to column-major, and then the outer product mode is selected as the target multiplication mode.

[0131] In addition, when the target layout modes of the left matrix and the right matrix are both column-major, the layout adjustment operator can also be used to first adjust the target layout mode of the left matrix or the right matrix, and then the target multiplication mode that matches the target layout mode is selected. This application does not make any specific limitations on this.

[0132] In an embodiment of the present application, the target layout mode of each of the multiple input matrices of the matrix multiplication operator in the memory is first obtained, and then a target multiplication mode that matches the target layout mode is selected to achieve that during the execution of the matrix multiplication operator, the artificial intelligence chip can use a continuous memory access method to read multiple input matrices for matrix multiplication calculations. This can improve the data reading efficiency during the execution of the matrix multiplication operator, thereby improving the calculation efficiency of the matrix multiplication operator.

[0133] In some embodiments, in static graph mode, the entire computational graph needs to be compiled, that is, the shape information of the input matrix of each operator in the entire computational graph, the dependency relationship between each operator, etc. can be obtained during the compilation stage. Therefore, the input, output, calculation method, etc. of each operator in the entire computational graph can be adjusted during the compilation stage.

[0134] For the matrix multiplication operator in the computation graph, the layout of the multiple input matrices of the matrix multiplication operator can be adjusted during the compilation phase, where the multiple input matrices of the matrix multiplication operator can be the output matrices of the previous operator; of course, the matrix multiplication operator can also be the first operator executed in the computation graph (that is, there is no operator that needs to be executed before the matrix multiplication operator). In this case, the multiple input matrices of the matrix multiplication operator are also the input matrices of the entire computation graph. In addition, the multiplication method of the matrix multiplication operator can also be adjusted during the compilation phase.

[0135] Since the layout of multiple input matrices and the multiplication method of the matrix multiplication operator are both adjustable, the multiplication method of the matrix multiplication operator not only determines the data reading method (that is, affects the data loading performance) together with the layout of the input matrices, but also affects the data calculation performance of the matrix multiplication operator together with the parameter configuration information of the matrix multiplication operator.

[0136] In view of this, the present application first determines the target multiplication method of the matrix multiplication operator to ensure data calculation performance, and then matches the target layout method based on the target multiplication method to ensure data loading performance.

[0137] Specifically, based on the parameter configuration information of the matrix multiplication operator, a target multiplication mode of the matrix multiplication operator is obtained from multiple candidate multiplication modes. Then, based on the target multiplication mode, target layout modes of multiple input matrices are selected from multiple candidate layout modes; finally, multiple input matrices are respectively stored in memory according to their respective target layout modes.

[0138] In some embodiments, the parameter configuration information of the matrix multiplication operator includes: shape information of each of the multiple input matrices. The embodiment of the present application adopts at least the following method to obtain the target multiplication method of the matrix multiplication operator:

[0139] For multiple candidate multiplication methods, the following operations are performed respectively: based on the shape information of each of the multiple input matrices, an estimated processing performance corresponding to the matrix multiplication calculation of the multiple input matrices using one candidate multiplication method is obtained. Based on the estimated processing performance corresponding to each of the multiple candidate multiplication methods, a target multiplication method of the matrix multiplication operator is obtained from the multiple candidate multiplication methods.

[0140] Specifically, the expected processing performance refers to: the theoretical performance corresponding to the matrix multiplication calculation of multiple input matrices using a candidate multiplication method, and the expected processing performance can be characterized by the number of clock cycles. After obtaining the expected processing performance corresponding to each of the multiple candidate multiplication methods, the candidate multiplication method with the best expected processing performance can be selected from the multiple candidate multiplication methods as the target multiplication method of the matrix multiplication operator; that is, the candidate multiplication method corresponding to the minimum number of clock cycles is selected as the target multiplication method of the matrix multiplication operator.

[0141] In an embodiment of the present application, based on the shape information of each of the multiple input matrices, the expected processing performance corresponding to each candidate multiplication method when performing matrix multiplication calculations on multiple input matrices is pre-calculated. Then, the target multiplication method with the best performance is selected from the multiple candidate multiplication methods. Then, the target layout method of each of the multiple input matrices is matched in combination with the target multiplication method. In this way, during the execution of the matrix multiplication operator, the artificial intelligence chip not only uses a continuous memory access method to read multiple input matrices from the memory, but also uses the target multiplication method with the best performance to perform matrix multiplication calculations on multiple input matrices, thereby greatly improving the data reading efficiency and data calculation efficiency during the execution of the matrix multiplication operator.

[0142] In some embodiments, the embodiments of the present application use at least the following methods to obtain the estimated processing performance corresponding to each candidate multiplication method:

[0143] When a candidate multiplication method is used to perform matrix multiplication calculation on multiple input matrices, a first number of clock cycles required to load the multiple input matrices is obtained based on the shape information of each of the multiple input matrices; and a second number of clock cycles required to calculate the multiple input matrices is obtained based on the shape information of each of the multiple input matrices. The maximum number of clock cycles between the first number of clock cycles and the second number of clock cycles is used as the estimated processing performance corresponding to the candidate multiplication method.

[0144] Specifically, when a candidate multiplication method is used to perform matrix multiplication calculation on multiple input matrices, it includes a data loading process and a data calculation process. Based on the shape information of each of the multiple input matrices, the first number of clock cycles required for the data loading process (i.e., loading multiple input matrices) and the second number of clock cycles required for the data calculation process (calculating multiple input matrices) are calculated respectively. The maximum number of clock cycles between the first number of clock cycles and the second number of clock cycles is used as the expected processing performance corresponding to the candidate multiplication method.

[0145] In some embodiments, the plurality of candidate layout modes include: a row-major order mode and a column-major order mode; the plurality of input matrices include a left matrix and a right matrix. On this basis, the present application at least adopts the following methods to obtain the target layout modes of the plurality of input matrices, see Fig. 9 , including the following steps:

[0146] Step 901, based on the shape information of the left matrix and the right matrix, a target multiplication method of the matrix multiplication operator is obtained from a plurality of candidate multiplication methods.

[0147] Step 902, determine whether the target multiplication method is the inner product method, if so, execute step 903, otherwise execute step 905.

[0148] Step 903: Use the row-major order as the target layout of the left matrix.

[0149] Step 904 , using the column-major order as the target layout of the right matrix, and then executing step 907 .

[0150] Specifically, when the target multiplication mode is the inner product mode, it is necessary to read the elements of the left matrix from the memory row by row, and to read the elements of the right matrix from the memory column by column during the execution of the matrix multiplication operator; therefore, the row-major order is used as the target layout mode of the left matrix (that is, the left matrix is ​​stored row by row in the memory), and the column-major order is used as the target layout mode of the right matrix (that is, the right matrix is ​​stored column by column in the memory). In this way, the left matrix and the right matrix can be read in a continuous memory access manner during the execution of the matrix multiplication operator.

[0151] In addition, when the original layout mode of the left matrix is ​​column-major, the original layout mode of the left matrix (ie, column-major) can be adjusted to a target layout mode (ie, row-major) through a layout adjustment operator.

[0152] Similarly, when the original layout of the right matrix is ​​row-major, the layout adjustment operator can be used to adjust the original layout of the right matrix (i.e., row-major) to the target layout (i.e., column-major).

[0153] It should be noted that since the left matrix and the right matrix of the matrix multiplication operator are the output matrices of the predecessor operator of the matrix multiplication operator, when the present application adjusts the layout of the left matrix and the right matrix of the matrix multiplication operator, it actually adjusts the layout of the output matrix of the predecessor operator of the matrix multiplication operator based on the calculation graph (i.e., the static graph).

[0154] Step 905: Use the column-major order as the target layout of the left matrix.

[0155] Step 906 , taking the row-major order as the target layout of the right matrix, and then executing step 907 .

[0156] Specifically, when the target multiplication mode is the outer product mode, it is necessary to read the elements of the left matrix from the memory column by column and to read the elements of the right matrix from the memory row by row during the execution of the matrix multiplication operator; therefore, the column-major order is used as the target layout mode of the left matrix (that is, the left matrix is ​​stored column by column in the memory), and the row-major order is used as the target layout mode of the right matrix (that is, the right matrix is ​​stored row by row in the memory). In this way, the left matrix and the right matrix can be read in a continuous memory access manner during the execution of the matrix multiplication operator.

[0157] In addition, when the original layout mode of the left matrix is ​​row-major, the original layout mode of the left matrix (ie, row-major) can be adjusted to a target layout mode (ie, column-major) through a layout adjustment operator.

[0158] Similarly, when the original layout mode of the right matrix is ​​column-major, the original layout mode of the right matrix (ie, column-major) can be adjusted to a target layout mode (ie, row-major) through a layout adjustment operator.

[0159] It should be noted that since the left matrix and the right matrix of the matrix multiplication operator are the output matrices of the predecessor operator of the matrix multiplication operator, when the present application adjusts the layout of the left matrix and the right matrix of the matrix multiplication operator, it actually adjusts the layout of the output matrix of the predecessor operator of the matrix multiplication operator based on the calculation graph (i.e., the static graph).

[0160] Step 907: output the target layout modes of the left matrix and the right matrix respectively.

[0161] In an embodiment of the present application, based on the shape information of the left matrix and the right matrix, the expected processing performance corresponding to each candidate multiplication method when performing matrix multiplication calculations on the left matrix and the right matrix is ​​pre-calculated. Then, the target multiplication method with the best performance is selected from multiple candidate multiplication methods. Then, the target layout method of the left matrix and the right matrix is ​​matched in combination with the target multiplication method. In this way, during the execution of the matrix multiplication operator, the artificial intelligence chip not only uses a continuous memory access method to read the left matrix and the right matrix from the memory, but also uses the multiplication method with the best performance to perform matrix multiplication calculations on the left matrix and the right matrix, thereby greatly improving the data reading efficiency and data calculation efficiency during the execution of the matrix multiplication operator.

[0162] Based on the same technical concept, the present application embodiment provides a schematic diagram of the structure of an optimization device for a matrix multiplication operator, such as Fig.10 As shown, the optimization device 1000 of the matrix multiplication operator includes:

[0163] Configuration module 1001, used for determining, before executing a matrix multiplication operator, a target layout mode of each of a plurality of input matrices of the matrix multiplication operator in a memory and a target multiplication mode of the matrix multiplication operator;

[0164] The target layout mode and the target multiplication mode support reading the multiple input matrices from the memory in a continuous memory access mode during the execution of the matrix multiplication operator.

[0165] Optionally, the multiple input matrices include: a left matrix and a right matrix; the target layout mode of each of the multiple input matrices and the target multiplication mode of the matrix multiplication operator are any one of the following multiple forms:

[0166] The target layout mode of the left matrix is ​​row-major order, the target layout mode of the right matrix is ​​column-major order, and the target multiplication mode is inner product mode; or,

[0167] The target layout mode of the left matrix is ​​a column-major order mode, the target layout mode of the right matrix is ​​a row-major order mode, and the target multiplication mode is an outer product mode.

[0168] Optionally, the configuration module 1001 is specifically used for:

[0169] Obtaining a target layout mode of each of a plurality of input matrices of the matrix multiplication operator in the memory;

[0170] A target multiplication method matching the target layout methods of the plurality of input matrices is selected from the plurality of candidate multiplication methods.

[0171] Optionally, the multiple candidate multiplication modes include: an inner product mode and an outer product mode; the multiple input matrices include a left matrix and a right matrix;

[0172] The configuration module 1001 is specifically used for:

[0173] When the target layout mode of the left matrix is ​​a row-major order mode, and the target layout mode of the right matrix is ​​a column-major order mode, selecting the inner product mode as the target multiplication mode;

[0174] When the target layout mode of the left matrix is ​​a column-major order mode, and the target layout mode of the right matrix is ​​a row-major order mode, the outer product mode is selected as the target multiplication mode.

[0175] Optionally, the configuration module 1001 is specifically used for:

[0176] Based on the parameter configuration information of the matrix multiplication operator, obtaining a target multiplication mode of the matrix multiplication operator from a plurality of candidate multiplication modes;

[0177] A target layout mode of each of the plurality of input matrices is selected from a plurality of candidate layout modes based on the target multiplication mode.

[0178] Optionally, the parameter configuration information of the matrix multiplication operator includes: shape information of each of the multiple input matrices;

[0179] The configuration module 1001 is specifically used for:

[0180] For the multiple candidate multiplication methods, the following operations are respectively performed: based on the shape information of each of the multiple input matrices, an estimated processing performance corresponding to when a candidate multiplication method is used to perform matrix multiplication calculations on the multiple input matrices;

[0181] Based on the estimated processing performance corresponding to each of the multiple candidate multiplication modes, a target multiplication mode of the matrix multiplication operator is obtained from the multiple candidate multiplication modes.

[0182] Optionally, the configuration module 1001 is specifically used for:

[0183] When performing matrix multiplication calculation on the multiple input matrices using the one candidate multiplication mode, obtaining a first number of clock cycles required to load the multiple input matrices based on shape information of each of the multiple input matrices; and obtaining a second number of clock cycles required to calculate the multiple input matrices based on shape information of each of the multiple input matrices;

[0184] The maximum number of clock cycles between the first number of clock cycles and the second number of clock cycles is used as the estimated processing performance corresponding to the one candidate multiplication method.

[0185] Optionally, the multiple candidate layout modes include: a row-major order mode and a column-major order mode; the multiple input matrices include a left matrix and a right matrix;

[0186] The configuration module 1001 is specifically used for:

[0187] When the target multiplication mode is an inner product mode, the row-major order mode is used as the target layout mode of the left matrix, and the column-major order mode is used as the target layout mode of the right matrix;

[0188] When the target multiplication mode is the outer product mode, the column-major order mode is used as the target layout mode of the left matrix, and the row-major order mode is used as the target layout mode of the right matrix.

[0189] In an embodiment of the present application, before executing the matrix multiplication operator, the target layout mode of each of the multiple input matrices of the matrix multiplication operator in the memory and the target multiplication mode of the matrix multiplication operator are determined, so that the target layout mode and the target multiplication mode support the use of continuous memory access to read multiple input matrices from the memory during the execution of the matrix multiplication operator. In this way, during the execution of the matrix multiplication operator, the artificial intelligence chip can use a continuous memory access method to read multiple input matrices (i.e., read data stored continuously in the memory) to perform matrix multiplication calculations; compared to reading non-continuously stored data, the artificial intelligence chip has a higher reading performance for continuously stored data, which effectively improves the data reading efficiency during the execution of the matrix multiplication operator, thereby improving the calculation efficiency of the matrix multiplication operator.

[0190] Based on the same technical concept, the embodiment of the present application provides a computer device, such as Fig.11 As shown, it includes at least one artificial intelligence chip 100 and a memory 1101 connected to the at least one artificial intelligence chip 100. The specific connection medium between the artificial intelligence chip 100 and the memory 1101 is not limited in the embodiment of the present application. Fig.11In the example, the artificial intelligence chip 100 and the memory 1101 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0191] In an embodiment of the present application, the memory 1101 stores instructions that can be executed by at least one artificial intelligence chip 100. At least one artificial intelligence chip 100 can execute the steps of the above-mentioned matrix multiplication operator optimization method by executing the instructions stored in the memory 1101.

[0192] Among them, the artificial intelligence chip 100 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and the matrix multiplication operator can be optimized by running or executing instructions stored in the memory 1101 and calling data stored in the memory 1101. Optionally, the artificial intelligence chip 100 may include one or more processing units, and the artificial intelligence chip 100 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 1101 can be implemented on the same chip, and in some embodiments, they can also be implemented separately on independent chips.

[0193] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0194] Memory 1101, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1101 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic storage, disk, optical disk, etc. Memory 1101 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 1101 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0195] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned matrix multiplication operator optimization method.

[0196] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned matrix multiplication operator optimization method.

[0197] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0199] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0200] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0201] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0202] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for optimizing a matrix multiplication operator, characterized in that: include: Before executing a matrix multiplication operator, determining a target layout mode of each of a plurality of input matrices of the matrix multiplication operator in a memory and a target multiplication mode of the matrix multiplication operator; The target layout mode and the target multiplication mode support reading the multiple input matrices from the memory in a continuous memory access mode during the execution of the matrix multiplication operator.

2. The method according to claim 1, characterized in that The multiple input matrices include: a left matrix and a right matrix; the target layout mode of each of the multiple input matrices and the target multiplication mode of the matrix multiplication operator are any one of the following multiple forms: The target layout mode of the left matrix is ​​row-major order, the target layout mode of the right matrix is ​​column-major order, and the target multiplication mode is inner product mode; or, The target layout mode of the left matrix is ​​a column-major order mode, the target layout mode of the right matrix is ​​a row-major order mode, and the target multiplication mode is an outer product mode.

3. The method according to claim 1, characterized in that Determining a target layout mode of each of the plurality of input matrices of the matrix multiplication operator in a memory and a target multiplication mode of the matrix multiplication operator includes: Obtaining a target layout mode of each of a plurality of input matrices of the matrix multiplication operator in the memory; A target multiplication method matching the target layout methods of the plurality of input matrices is selected from the plurality of candidate multiplication methods.

4. The method according to claim 3, characterized in that The multiple candidate multiplication modes include: an inner product mode and an outer product mode; the multiple input matrices include a left matrix and a right matrix; The step of selecting a target multiplication mode that matches the target layout modes of the respective input matrices from the plurality of candidate multiplication modes comprises: When the target layout mode of the left matrix is ​​a row-major order mode, and the target layout mode of the right matrix is ​​a column-major order mode, selecting the inner product mode as the target multiplication mode; When the target layout mode of the left matrix is ​​a column-major order mode, and the target layout mode of the right matrix is ​​a row-major order mode, the outer product mode is selected as the target multiplication mode.

5. The method according to claim 1, characterized in that Determining a target layout mode of each of the plurality of input matrices of the matrix multiplication operator in a memory and a target multiplication mode of the matrix multiplication operator includes: Based on the parameter configuration information of the matrix multiplication operator, obtaining a target multiplication mode of the matrix multiplication operator from a plurality of candidate multiplication modes; A target layout mode of each of the plurality of input matrices is selected from a plurality of candidate layout modes based on the target multiplication mode.

6. The method according to claim 5, characterized in that The parameter configuration information of the matrix multiplication operator includes: shape information of each of the multiple input matrices; The step of obtaining a target multiplication mode of the matrix multiplication operator from a plurality of candidate multiplication modes based on the parameter configuration information of the matrix multiplication operator includes: For the multiple candidate multiplication methods, the following operations are respectively performed: based on the shape information of each of the multiple input matrices, an estimated processing performance corresponding to when a candidate multiplication method is used to perform matrix multiplication calculations on the multiple input matrices; Based on the estimated processing performance corresponding to each of the multiple candidate multiplication modes, a target multiplication mode of the matrix multiplication operator is obtained from the multiple candidate multiplication modes.

7. The method according to claim 6, characterized in that The obtaining, based on the shape information of each of the plurality of input matrices, an estimated processing performance corresponding to matrix multiplication calculations performed on the plurality of input matrices using a candidate multiplication method comprises: When performing matrix multiplication calculation on the multiple input matrices using the one candidate multiplication mode, obtaining a first number of clock cycles required to load the multiple input matrices based on shape information of each of the multiple input matrices; and obtaining a second number of clock cycles required to calculate the multiple input matrices based on shape information of each of the multiple input matrices; The maximum number of clock cycles between the first number of clock cycles and the second number of clock cycles is used as the estimated processing performance corresponding to the one candidate multiplication method.

8. The method according to claim 5, characterized in that The multiple candidate layout modes include: a row-major order mode and a column-major order mode; the multiple input matrices include a left matrix and a right matrix; The selecting, based on the target multiplication mode, a target layout mode of each of the multiple input matrices from a plurality of candidate layout modes comprises: When the target multiplication mode is an inner product mode, the row-major order mode is used as the target layout mode of the left matrix, and the column-major order mode is used as the target layout mode of the right matrix; When the target multiplication mode is the outer product mode, the column-major order mode is used as the target layout mode of the left matrix, and the row-major order mode is used as the target layout mode of the right matrix.

9. A computer device comprising a memory, an artificial intelligence chip, and a computer program stored in the memory and executable on the artificial intelligence chip, characterized in that: When the artificial intelligence chip executes the computer program, the steps of the method as claimed in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device. When the computer program is run on the computer device, the computer device executes the steps of the method according to any one of claims 1 to 8.

11. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 8.

Images