Data processing method, chip, and computing device

Abstract

A data processing method, a chip, and a computing device. The chip can automatically detect whether a matrix operation of two matrices is a small matrix operation or a large matrix operation, without requiring active setting performed by a user, such that the computing efficiency of the chip is improved. The chip can dynamically select one type of computing units suitable for a matrix operation of two matrices, thereby avoiding the situation where different types of computing units cannot operate at an optimal operation frequency at the same moment due to a chip simultaneously running multiple types of computing units. In addition, the chip selects a computing unit suitable for a bit width of the matrix operation of the two matrices, such that the filling rate of the computing unit is high, thereby improving the computing efficiency of a processing unit.

Description

Data processing method, chip and computing device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 26, 2023, with application number 202310761257.8 and application name “A method, chip and computing device for data processing”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present invention relates to the field of computer technology, and in particular to a data processing method, chip and computing device. Background Art

[0003] The general matrix multiply (GEMM) calculation, a computation performed on dense matrices, is widely used in deep learning, a field used in modern science and technology such as weather forecasting, oil exploration, and nuclear physics. Deep learning is a method in machine learning that relies on learning to represent data. In the fields of high-performance computing (HPC) and artificial intelligence (AI), there are numerous scenarios involving small-scale, dense GEMMs. Chips that run GEMMs have multiple types of compute units. When a user calls a chip to run a GEMM, they must preselect one or more types of compute units to run the GEMM operator. Each time a chip runs a GEMM, the user must configure the chip, resulting in relatively low computational efficiency.

[0004] Summary of the Invention

[0005] In order to solve the above problems, a data processing method is provided in an embodiment of the present application. The chip can automatically detect whether the matrix operation of two matrices belongs to a small matrix operation or a large matrix operation, without the need for the user to actively set it, thereby improving the computing efficiency of the chip. In addition, the chip can dynamically select a computing unit suitable for the matrix operation of the two matrices, avoiding the chip running multiple types of computing units at the same time, so that different types of computing units cannot be at the optimal operating frequency at the same time. In addition, the present application also provides a chip and computing device corresponding to the data processing method.

[0006] To this end, the following technical solutions are adopted in the embodiments of the present application:

[0007] In a first aspect, an embodiment of the present application provides a method for data processing, which is executed by a chip, and the chip includes a first computing unit and a second computing unit; the method includes: obtaining a first matrix and a second matrix to be performed matrix multiplication; determining whether the matrix multiplication operation of the first matrix and the second matrix belongs to a large matrix operation or a small matrix operation based on the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix and the number of rows K of the second matrix, the amount of calculation of the large matrix operation is greater than the amount of calculation of the small matrix operation, and M, N and K are all positive integers greater than or equal to 1; when the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation, the first matrix and the second matrix are transferred to the first computing unit for processing; when the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation, the first matrix and the second matrix are transferred to the second computing unit for processing.

[0008] In this implementation, the chip automatically detects whether the matrix operation between the two matrices is a small or large matrix operation based on the number of rows and columns of the two matrices input, eliminating the need for user-initiated configuration and improving the chip's computational efficiency. Furthermore, the chip selects a computational unit suitable for the matrix operation between the two matrices, preventing the chip from running multiple types of computational units simultaneously, which could prevent different types of computational units from operating at optimal frequencies at the same time.

[0009] In one embodiment, the first computing unit is a Variable Vector Extension (SVE) computing unit, and the second computing unit is a Variable Matrix Extension (SME) computing unit.

[0010] In this embodiment, the SVE calculation unit is a vector calculation engine that multiplies each element of a row of a first set numbered matrix with each element of a column of a second set numbered matrix, and superimposes the products to obtain an element of the first set numbered row and the second set numbered column. The chip can use the SVE calculation unit to perform small matrix operations. The SME calculation unit is a matrix-based outer product calculation engine that calculates the outer product of each element of each column of the first matrix with each element of the row of the second matrix with the same number as the column of the first matrix, and superimposes the products of the same row and column to obtain the result matrix. The chip can use the SME calculation unit to perform large matrix operations.

[0011] In one embodiment, determining whether the matrix multiplication operation of the first matrix and the second matrix belongs to large matrix operation or small matrix operation according to the number of rows M of the first matrix, the number of columns N of the second matrix, the number of columns K of the first matrix, and the number of rows K of the second matrix includes: obtaining a threshold k from a preset mapping relationship according to M and N, where different values of M and N in the preset mapping relationship correspond to the threshold k; when K >= k, determining that the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation; when K < k, determining that the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation.

[0012] In this embodiment, a lookup table can be pre-stored inside the chip. The lookup table records the mapping relationship between different M and N and the threshold k. After the chip determines the corresponding threshold k according to M and N, it can compare K with the threshold k to determine whether the matrix operation of the two matrices belongs to small matrix operation or large matrix operation, so as to implement a calculation unit in the chip that selects a suitable matrix operation for the two matrices.

[0013] In one embodiment, determining whether the matrix multiplication operation of the first matrix and the second matrix belongs to large matrix operation or small matrix operation according to the number of rows M of the first matrix, the number of columns N of the second matrix, the number of columns K of the first matrix, and the number of rows K of the second matrix includes: calculating the product of M, N, and K; when the product of M, N, and K is greater than or equal to a threshold, determining that the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation; when the product of M, N, and K is less than the threshold, determining that the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation.

[0014] In this embodiment, the chip can determine the matrix volume after the matrix operation of the two matrices according to the product of M, N, and K, and based on the comparison between the matrix volume and the threshold, determine whether the matrix operation of the two matrices belongs to small matrix operation or large matrix operation, so as to implement a calculation unit in the chip that selects a suitable matrix operation for the two matrices.

[0015] Second aspect, an embodiment of the present application provides a chip, including: a first computing unit, a second computing unit, and a processing unit. The processing unit is configured to obtain a first matrix and a second matrix for which matrix multiplication is to be performed; determine whether the matrix multiplication operation of the first matrix and the second matrix belongs to a large matrix operation or a small matrix operation according to the number of rows M of the first matrix, the number of columns N of the second matrix, the number of columns K of the first matrix, and the number of rows K of the second matrix. The amount of computation of the large matrix operation is greater than that of the small matrix operation, and M, N, and K are all positive integers greater than or equal to 1; when the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation, transmit the first matrix and the second matrix to the first computing unit for processing; and when the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation, transmit the first matrix and the second matrix to the second computing unit for processing.

[0016] In one implementation, the first computing unit is a Scalable Vector Extension (SVE) computing unit, and the second computing unit is a Scalable Matrix Extension (SME) computing unit.

[0017] In one implementation, the processing unit is specifically configured to obtain a threshold k from a preset mapping relationship according to M and N. Different threshold values k corresponding to different values of M and N are set in the preset mapping relationship; when K >= k, it is determined that the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation; when K < k, it is determined that the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation.

[0018] In one implementation, the processing unit is specifically configured to calculate the product of M, N, and K; when the product of M, N, and K is greater than or equal to a threshold, it is determined that the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation; when the product of M, N, and K is less than the threshold, it is determined that the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation.

[0019] Third aspect, an embodiment of the present application provides a computing device, including: at least one memory, at least one chip. The at least one chip is configured to execute instructions stored in the at least one memory, so that the chip executes each possible implementation embodiment of the first aspect. [[ID=,13]]

[0020] Fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program instructions are executed by a chip, the chip executes each possible implementation embodiment of the first aspect.

[0021] In a fifth aspect, an embodiment of the present application provides a computer program product, characterized in that the computer program product stores instructions, and when the instructions are executed by a computer, the computer implements each possible implementation embodiment of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The following is a brief introduction to the drawings required for describing the embodiments or prior art.

[0023] FIG1 is a schematic diagram of a result matrix multiplication;

[0024] FIG2 is a schematic diagram of the structure of a chip provided in an embodiment of the present application;

[0025] FIG3 is a schematic diagram of an implementation flow of a data processing method provided in an embodiment of the present application;

[0026] FIG4 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0028] The term "and / or" as used herein describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " as used herein indicates that the related objects are in an "or" relationship, for example, A / B means either A or B.

[0029] The terms "first" and "second" in this specification and claims are used to distinguish different objects rather than to describe a specific order of objects. For example, "first response message" and "second response message" are used to distinguish different response messages rather than to describe a specific order of response messages.

[0030] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0031] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0032] As shown in Figure 1, the result matrix can be obtained by multiplying two matrices. Matrix A is an M×K matrix, where M and K are both positive integers greater than or equal to 1. M represents the number of rows in Matrix A, and K represents the number of columns in Matrix A. Matrix B is a K×N matrix, where N is a positive integer greater than or equal to 1. K represents the number of rows in Matrix B, and N represents the number of columns in Matrix B. The number of columns in Matrix A is the same as the number of rows in Matrix B. After multiplying Matrix A and Matrix B, the result is Matrix C, which is an M×N matrix.

[0033] The GEMM operator is generally run by a chip or a computing device equipped with a chip. The chip can be divided into multiple computing units according to the minimum unit for calculating a GEMM. Each computing unit can calculate part of a GEMM, or calculate part of a GEMM in batches. In the new generation of chips, new functions can be added to the computing units. For example, a computing unit with a scalable vector extension (SVE) function can be called an SVE computing unit. For another example, a computing unit with a scalable matrix extension (SME) function can be called an SME computing unit.

[0034] The SVE calculation unit is based on a vector calculation engine and multiplies each element of a row of a first set numbered matrix with each element of a column of a second set numbered matrix, and adds the products to obtain an element of the first set numbered row and the second set numbered column. In one embodiment, as shown in Figure 1, each element of the i-th row of matrix A is multiplied with each element of the j-th column of matrix B, and the products are added to obtain an element of the i-th row and the j-th column. Both i and j are positive integers greater than or equal to 1.

[0035] The SME calculation unit is a matrix-based outer product calculation engine that calculates the outer product of each element in each column of the first matrix with each element in the row of the second matrix with the same sequence number as the column of the first matrix, and superimposes the products of the same rows and columns to obtain the result matrix. In one embodiment, the GEMM is specifically as follows:

[0036] When calculating the result matrix, the SME calculation unit multiplies the first column of the first matrix by the first row of the second matrix, resulting in a×a for the first row and first column, a×b for the first row and second column, c×a for the second row and first column, and c×b for the second row and second column. The SME calculation unit multiplies the second column of the first matrix by the second row of the second matrix, resulting in b×c for the first row and first column, b×d for the first row and second column, d×c for the second row and first column, and d×d for the second row and second column. The SME calculation unit superimposes the products of the same row and column, resulting in a×a+b×c for the first row and first column, a×b+b×d for the first row and second column, c×a+d×c for the second row and first column, and c×b+d×d for the second row and second column.

[0037] The optimal operating frequency of the SVE compute unit is generally around 2.0 GHz. The optimal operating frequency of the SME compute unit is generally around 1.6 GHz. Because the optimal operating frequencies of the SVE and SME compute units are different, the chip cannot operate at the optimal frequencies of the SVE and SME compute units at the same time. Furthermore, when the SME compute unit runs the GEMM operator, the GEMM size is smaller than the bit width of the SME compute unit, which reduces the SME compute unit's computational efficiency.

[0038] In order to solve the defects of existing chips, the embodiments of the present application design a chip, a data processing method and a computing device. A processing unit with an automatic discrimination function can be added inside the chip. The processing unit can automatically detect whether the matrix operation of two matrices belongs to a small matrix operation or a large matrix operation, without the need for the user to actively set it, thereby improving the computing efficiency of the chip. In addition, the chip can dynamically select a computing unit suitable for the matrix operation of the two matrices, avoiding the chip running multiple types of computing units at the same time, so that different types of computing units cannot be at the optimal operating frequency at the same time. In addition, the processing unit selects a computing unit with a bit width suitable for the matrix operation of the two matrices, so that the fill rate of the computing unit is relatively high, which can improve the computing efficiency of the processing unit.

[0039] Figure 2 is a schematic diagram of the structure of a chip provided in an embodiment of the present application. As shown in Figure 2, chip 200 includes multiple first computing units 210, multiple second computing units 220, and a processing unit 230. Processing unit 230 is connected to multiple first computing units 210 and multiple second computing units 220 to implement data transmission.

[0040] In the embodiment of the present application, the chip 200 may be a central processing unit (CPU), a graphics processing unit (GPU), or other chips with processing functions. The chip 200 may be divided into a plurality of first computing units 210, a plurality of second computing units 220, and a processing unit 230 based on the execution functions.

[0041] The first calculation unit 210 can be an SVE calculation unit, a calculation unit having the function of calculating the inner product of a matrix. The first calculation unit 210 is used to multiply each element of a row of the first matrix with each element of a column of the second matrix, and superimpose each product. In an embodiment of the present application, after receiving the first matrix and the second matrix, the first calculation unit 210 can multiply each element of a row of the first set sequence number of the first matrix with each element of a column of the second set sequence number of the second matrix, and then superimpose each product to obtain an element of the first set sequence number row and the second set sequence number column. By analogy, the first calculation unit 210 can calculate each element to obtain a result matrix.

[0042] Some of the multiple first computing units 210 of chip 200 have the same bit width. Some of the multiple first computing units 210 of chip 200 have different bit widths. For a first computing unit 210, the bit width refers to the number of elements in a column of the second matrix calculated by the first computing unit 210. First computing units 210 with different bit widths can call different GEMM operators. The fill rate of the GEMM operator of the first computing unit 210 is in a fully loaded state, which can improve the computational efficiency of the first computing unit 210.

[0043] The second calculation unit 220 can be an SME calculation unit, a calculation unit with the function of calculating the outer product of a matrix. The second calculation unit 220 is used to perform outer product calculations on each element of each column of the first matrix with each element of the row of the second matrix with the same sequence number as the column of the first matrix, and superimpose the products of the same row and column to obtain a result matrix. In the embodiment of the present application, after receiving the first matrix and the second matrix, the second calculation unit 220 can perform outer product calculations on each element of a column of the first matrix with each element of the row of the second matrix with the same sequence number as the column of the first matrix, to obtain products of different rows and columns. Similarly, the second calculation unit 220 can perform outer product calculations on each element of the other columns of the first matrix with each element of the row of the second matrix with the same sequence number as the column of the first matrix, to obtain products of different rows and columns. After the second calculation unit 220 calculates each product of different rows and columns, it superimposes each product of the same row and column to obtain a result matrix.

[0044] Some of the multiple second computing units 220 of the chip 200 have the same bit width. Some of the multiple second computing units 220 of the chip 200 have the same bit width. For a second computing unit 220, the bit width refers to the larger value between the number of elements in a row of the first matrix and the number of elements in a column of the second matrix calculated by the second computing unit 220. Second computing units 220 with different bit widths can call different GEMM operators. The fill rate of the GEMM operator of the second computing unit 220 is in a fully loaded state, which can improve the computational efficiency of the second computing unit 220.

[0045] The processing unit 230 is used to receive two matrices. The two matrices are a first matrix and a second matrix. After receiving the first matrix and the second matrix, the processing unit 230 can identify the number of rows and columns of the two matrices. Generally, the number of columns of the first matrix is ​​the same as the number of rows of the second matrix. The number of rows and columns of the two matrices identified by the processing unit 230 are three numerical values, namely the number of rows of the first matrix, the number of columns of the second matrix, and the number of columns of the first matrix (or "the number of rows of the second matrix"). In a possible implementation example, as shown in Figure 1, the first matrix is ​​matrix A and the second matrix is ​​matrix B. The result matrix C is obtained by multiplying matrix A and matrix B. The processing unit 230 obtains that the number of rows of matrix A is M and the number of columns is K. The processing unit 230 obtains that the number of rows of matrix B is K and the number of columns is N. The number of rows of the first matrix obtained by the processing unit 230 is M, the number of columns of the second matrix is ​​N, and the number of columns of the first matrix is ​​K.

[0046] The processing unit 230 determines whether the matrix multiplication operation between matrix A and matrix B is a large matrix operation or a small matrix operation based on the number of rows of the first matrix (M), the number of columns of the second matrix (N), and the number of columns of the first matrix (K). In one case, if the matrix multiplication operation between matrix A and matrix B is a small matrix operation, matrix A and matrix B are transferred to the SVE calculation unit for matrix multiplication. In another case, if the matrix multiplication operation between matrix A and matrix B is a large matrix operation, matrix A and matrix B are transferred to the SME calculation unit for matrix multiplication. Large matrix operations require more computation than small matrix operations.

[0047] In an embodiment of the present application, a method for determining whether the matrix multiplication operation of the matrix A and the matrix B is a small matrix operation is provided as follows: after obtaining the number of rows M of the first matrix and the number of columns N of the second matrix, the processing unit 230 can obtain a lookup table stored in the memory, and compare the number of rows M of the first matrix and the number of columns N of the second matrix in the lookup table to determine the computing unit for running the GEMM operator. In an embodiment of the present application, the designer of the chip 200 scans the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix of two different matrices, and then obtains the running time of the GEMM operator using different computing units under this scale through actual simulation operation, and finds the dividing line of the performance threshold k corresponding to the number of rows of the first matrix and the number of columns of the second matrix. The designer of the chip 200 establishes a two-dimensional threshold boundary lookup table based on the number of rows M of the first matrix, the number of columns N of the second matrix, and the threshold k.

[0048] Table 1 Two-dimensional threshold boundary lookup table

[0049] As shown in Table 1, a mapping relationship is established between the number of rows M of each first matrix and the number of columns N of each second matrix and a threshold value k. After receiving the first matrix and the second matrix, the processing unit 230 compares the number of rows M of the first matrix and the number of columns N of the second matrix with the lookup table to determine the corresponding threshold value k.

[0050] After determining the threshold k, the processing unit 230 can compare the number of columns K of the first matrix with the threshold k to determine whether the matrix multiplication operation between the first and second matrices is a large matrix operation or a small matrix operation, and select one of the first and second computing units 210 and 220 to perform the GEMM calculation. In one case, if the processing unit 230 determines that the number of columns K of the first matrix is ​​less than the threshold k, the matrix multiplication operation between the first and second matrices can be considered a large matrix operation. The processing unit 230 can transfer the first and second matrices to the first computing unit 210, instructing the first computing unit 210 to perform the GEMM calculation. In another case, if the processing unit 230 determines that the number of columns K of the first matrix is ​​greater than or equal to the threshold k, the matrix multiplication operation between the first and second matrices can be considered a small matrix operation. The processing unit 230 can transfer the first and second matrices to the second computing unit 220, instructing the second computing unit 220 to perform the GEMM calculation.

[0051] Another method provided in an embodiment of the present application for determining whether the matrix multiplication operation between the matrix A and the matrix B is a large matrix operation or a small matrix operation is:

[0052] The processing unit 230 can compare the product of the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix with a threshold value k to determine whether the matrix multiplication operation between the first and second matrices is a large matrix operation or a small matrix operation, and select one of the first and second computing units 210 and 220 to perform the GEMM calculation. In one case, if the processing unit 230 determines that the product of the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix is ​​less than the threshold value k, the matrix multiplication operation between the first and second matrices can be considered a large matrix operation. The processing unit 230 can transmit the first and second matrices to the first computing unit 210, instructing the first computing unit 210 to perform the GEMM calculation. In another case, if the processing unit 230 determines that the product of the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix is ​​greater than or equal to the threshold value k, the matrix multiplication operation between the first and second matrices can be considered a small matrix operation. The processing unit 230 may transmit the first matrix and the second matrix to the second computing unit 220 , so that the second computing unit 220 performs GEMM calculation.

[0053] The above two implementation methods are only examples of methods for determining whether the matrix multiplication operation of the matrix A and the matrix B is a large matrix operation or a small matrix operation, but are not limited to the above two methods. Other methods that can determine whether the matrix multiplication operation of the matrix A and the matrix B is a large matrix operation or a small matrix operation are also included in the scope disclosed in this application.

[0054] In an embodiment of the present application, the processing unit 230 can detect whether the matrix multiplication operation of the first matrix and the second matrix is ​​a large matrix operation or a small matrix operation based on the product of the number of columns K of the first matrix of the two matrices or the number of rows M of the first matrix, the number of columns N of the second matrix and the number of columns K of the first matrix, and select one of the first computing unit 210 and the second computing unit 220 to perform GEMM calculation, so as to avoid the chip 200 running the first computing unit 210 and the second computing unit 220 at the same time, so that the first computing unit 210 and the second computing unit 220 cannot be at the optimal operating frequency at the same time.

[0055] FIG3 is a schematic diagram of the implementation flow of the data processing method provided in the first embodiment of the present application. As shown in FIG3 , the data processing method is generally executed by the processing unit 230 of the chip 200, and the specific implementation process is as follows:

[0056] In step S301, the processing unit 230 obtains a first matrix and a second matrix, wherein the number of columns of the second matrix is ​​the same as the number of rows of the second matrix.

[0057] In step S302 , the processing unit 230 obtains the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix.

[0058] The result matrix calculated by the GEMM operator is obtained by multiplying the first matrix and the second matrix. The first matrix and the second matrix both include a number of columns and a number of rows, and the number of columns of the first matrix is ​​the same as the number of rows of the second matrix. After obtaining the first matrix and the second matrix, the processing unit 230 identifies the number of columns and rows of the first matrix and the second matrix to obtain the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix. The number of rows M of the first matrix is ​​the number of rows of the first matrix. The number of columns N of the second matrix is ​​the number of columns of the second matrix. The number of columns K of the first matrix is ​​the number of columns of the first matrix and the number of rows of the second matrix.

[0059] In step S303 , the processing unit 230 determines a threshold k for matrix operations between the first matrix and the second matrix.

[0060] After scanning the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix of two different matrices, the designer of chip 200 obtains the running time of the GEMM operator using different computing units at this scale through actual simulation operation, and finds the dividing line of the performance threshold k corresponding to the number of rows of the first matrix and the number of columns of the second matrix. The designer of chip 200 establishes a two-dimensional threshold boundary lookup table based on the number of rows M of the first matrix, the number of columns N of the second matrix, and the threshold k. A mapping relationship is established between the number of rows M of each first matrix and the number of columns N of the second matrix and a threshold k. In the embodiment of the present application, after receiving the first matrix and the second matrix, the processing unit 230 can obtain the lookup table stored in the memory, and compare the number of rows M of the first matrix and the number of columns N of the second matrix of the two matrices with the lookup table to determine the corresponding threshold k.

[0061] In step S304, the processing unit 230 detects whether the matrix multiplication operation of the first matrix and the second matrix is ​​a large matrix operation or a small matrix operation. In one case, if the processing unit 230 determines that the matrix multiplication operation of the first matrix and the second matrix is ​​a small matrix operation, step S305 is executed. In another case, if the processing unit 230 determines that the matrix multiplication operation of the first matrix and the second matrix is ​​a large matrix operation, step S306 is executed.

[0062] In step S305 , the processing unit 230 transmits the first matrix and the second matrix to the first calculation unit 210 .

[0063] In step S306 , the processing unit 230 transmits the first matrix and the second matrix to the second calculation unit 220 .

[0064] In one implementation, after determining the threshold k, the processing unit 230 may compare the number of columns K of the first matrix with the threshold k to determine whether the matrix multiplication operation between the first and second matrices is a large matrix operation or a small matrix operation, and select one of the first and second computing units 210 and 220 to perform the GEMM calculation. In one scenario, if the processing unit 230 determines that the number of columns K of the first matrix is ​​less than the threshold k, the matrix multiplication operation between the first and second matrices may be considered a large matrix operation. The processing unit 230 may transfer the first and second matrices to the first computing unit 210, instructing the first computing unit 210 to perform the GEMM calculation. The first computing unit 210 invokes the GEMM operator, which performs the calculation on the first and second matrices to obtain a result matrix. In another scenario, if the processing unit 230 determines that the number of columns K of the first matrix is ​​greater than or equal to the threshold k, the matrix multiplication operation between the first and second matrices may be considered a small matrix operation. The processing unit 230 may transfer the first and second matrices to the second computing unit 220, instructing the second computing unit 220 to perform the GEMM calculation. The second calculation unit 220 calls the GEMM operator and allows the GEMM operator to calculate the first matrix and the second matrix to obtain a result matrix.

[0065] Each of the first computing unit 210 and the second computing unit 220 corresponds to an operator library. The operator library stores GEMM operators, allowing the first computing unit 210 and the second computing unit 220 to perform GEMM calculations. The operator library is generally a library of basic linear algebra subprograms.

[0066] The BLAS library is a function library that provides some basic linear algebra operations. The BLAS library usually contains two levels. The first level contains basic vector and matrix operations, such as vector addition, matrix multiplication, etc. The second level contains advanced matrix operations, such as LU decomposition, QR decomposition, matrix inversion, etc. The function interface of the BLAS library usually adopts the Fortran format naming convention. The GEMM function in the BLAS library is used to perform matrix multiplication operations. It accepts four parameters, namely matrix A, matrix B, result matrix C, and the sizes of matrix A, matrix B, and matrix C. The GEMM function in the BLAS library is a highly optimized matrix multiplication implementation that can quickly perform matrix multiplication operations on multiple CPU architectures and hardware accelerators.

[0067] In a second embodiment of the present application, the processing unit 230 can compare the product of the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix with a threshold value k to detect whether the matrix multiplication operation between the first and second matrices is a large matrix operation or a small matrix operation, and select one of the first and second computing units 210 and 220 to perform a GEMM calculation. In one embodiment, when the processing unit 230 determines that the product of the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix is ​​less than the threshold value k, the matrix multiplication operation between the first and second matrices can be considered to be a large matrix operation. The processing unit 230 can transmit the first and second matrices to the first computing unit 210, instructing the first computing unit 210 to perform the GEMM calculation. The first computing unit 210 calls the GEMM operator and has the GEMM operator perform calculations on the first and second matrices to obtain a result matrix. In another case, when the processing unit 230 determines that the product of the number of rows M of the first matrix, the number of columns N of the second matrix, and the number of columns K of the first matrix is ​​greater than or equal to the threshold k, the matrix multiplication operation of the first matrix and the second matrix can be considered to be a small matrix operation. The processing unit 230 can transmit the first matrix and the second matrix to the second computing unit 220, and have the second computing unit 220 perform a GEMM calculation. The second computing unit 220 calls the GEMM operator and has the GEMM operator calculate the first matrix and the second matrix to obtain a result matrix.

[0068] Typically, the chip 200 has multiple first computing units 210 and second computing units 220 with different bit widths, which can run different GEMM operators. The bit widths of the first computing unit 210 and the second computing unit 220 are related to the number of columns K of the first matrix of the two matrices. In the embodiment of the present application, the processing unit 230 can scan the bit widths of the multiple first computing units 210 and the bit widths of the multiple second computing units 220, and pre-store the bit widths of the multiple first computing units 210 and the bit widths of the multiple second computing units 220 in the memory.

[0069] When the processing unit 230 determines to transmit the first matrix and the second matrix to the first computing unit 210, it can compare the number of columns K of the first matrix with the bit widths of multiple first computing units 210 to determine a first computing unit 210 with a bit width that is less than the number of columns K of the first matrix and close to the number of columns K of the first matrix, thereby obtaining a first computing unit 210 with an appropriate bit width. The processing unit 230 inputs the first matrix and the second matrix into the first computing unit 210 with the appropriate bit width, ensuring that the GEMM operator of the first computing unit 210 is in a fully loaded state as much as possible, thereby improving the computational efficiency of the first computing unit 210.

[0070] Similarly, when the processing unit 230 determines to transmit the first matrix and the second matrix to the second computing unit 220, it can compare the number of columns K of the first matrix with the bit widths of multiple second computing units 220 to determine a second computing unit 220 with a bit width that is smaller than the number of columns K of the first matrix and close to the number of columns K of the first matrix, thereby obtaining a second computing unit 220 with an appropriate bit width. The processing unit 230 inputs the first matrix and the second matrix into the second computing unit 220 with the appropriate bit width, ensuring that the GEMM operator of the second computing unit 220 is as fully loaded as possible, thereby improving the computational efficiency of the second computing unit 220.

[0071] It should be noted that the processing unit 230 can select a computing unit with an appropriate bit width and select the first computing unit or the second computing unit in any order. In one case, the processing unit 230 can preferentially select the first computing unit 210 and the second computing unit 220 with an appropriate bit width based on the number of columns K of the first matrix. Then, the processing unit 230 detects whether the matrix multiplication operation of the first matrix and the second matrix is ​​a large matrix operation or a small matrix operation, and selects one of the first computing unit 210 and the second computing unit 220 to perform the GEMM calculation. In another case, the processing unit 230 detects whether the matrix multiplication operation of the first matrix and the second matrix is ​​a large matrix operation or a small matrix operation, and selects one of the first computing unit 210 and the second computing unit 220 to perform the GEMM calculation. Then, the processing unit 230 can select the first computing unit 210 and the second computing unit 220 with an appropriate bit width based on the number of columns K of the first matrix.

[0072] In the embodiment of the present application, the processing unit 230 can automatically detect whether the matrix operation of the two matrices is a small matrix operation or a large matrix operation, and dynamically select a computing unit suitable for the matrix operation of the two matrices. This prevents the chip 200 from running multiple types of computing units simultaneously, which would prevent different types of computing units from being at the optimal operating frequency at the same time. In addition, the processing unit 230 selects a computing unit with a bit width suitable for the matrix operation of the two matrices, thereby ensuring a relatively high fill rate of the computing unit and improving the computing efficiency of the processing unit.

[0073] Figure 4 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. As shown in Figure 4, computing device 400 includes a bus 410, a chip 420, a memory 430, and a communication interface 440. Chip 420, memory 430, and communication interface 440 communicate with each other via bus 410. Computing device 400 can be a server, a computer, a portable notebook, a cabinet, etc. It should be understood that this application does not limit the number of chips and memories in computing device 400.

[0074] Bus 410 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG4 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 410 may include a path for transmitting information between various components of computing device 400 (e.g., chip 420, memory 430, and communication interface 440).

[0075] The chip 420 may be the aforementioned chip 200 .

[0076] The memory 430 may include a volatile memory, such as a random access memory (RAM). The memory 430 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0077] The memory 430 stores executable program code, such as a program for the processing unit, a program for the first computing unit, and a program for the second computing unit. The program for the first computing unit and the program for the second computing unit are, for example, GEMM operators. The program for the processing unit is, for example, a program with a discrimination function. The chip 420 executes the executable program code to implement the functions of the aforementioned first computing unit 210, second computing unit 220, and processing unit 230, respectively, thereby implementing the data processing method. That is, the memory 430 stores instructions for executing the data processing method.

[0078] The communication interface 440 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 400 and other devices or a communication network.

[0079] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data processing method.

[0080] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a high-density digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute a data processing method.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for data processing, characterized in that: The method is executed by a chip, and the chip includes a first computing unit and a second computing unit; the method includes: Obtaining a first matrix and a second matrix for which matrix multiplication is to be performed; Determining whether the matrix multiplication operation of the first matrix and the second matrix belongs to a large matrix operation or a small matrix operation according to the number of rows M of the first matrix, the number of columns N of the second matrix, the number of columns K of the first matrix, and the number of rows K of the second matrix, wherein the amount of computation of the large matrix operation is greater than that of the small matrix operation, and M, N, and K are all positive integers greater than or equal to 1; When the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation, transmitting the first matrix and the second matrix to the first computing unit for processing; When the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation, transmitting the first matrix and the second matrix to the second computing unit for processing.

2. The method according to claim 1, characterized in that The first computing unit is a scalable vector extension (SVE) computing unit, and the second computing unit is a scalable matrix extension (SME) computing unit.

3. The method according to claim 1 or 2, characterized in that: The determining whether the matrix multiplication operation of the first matrix and the second matrix belongs to a large matrix operation or a small matrix operation according to the number of rows M of the first matrix, the number of columns N of the second matrix, the number of columns K of the first matrix, and the number of rows K of the second matrix includes: Obtaining a threshold value k from a preset mapping relationship according to M and N, and different threshold values k corresponding to different values of M and N are set in the preset mapping relationship; When K >= k, determining that the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation; When K < k, determining that the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation.

4. The method according to claim 1 or 2, characterized in that: The determining whether the matrix multiplication operation of the first matrix and the second matrix belongs to a large matrix operation or a small matrix operation according to the number of rows M of the first matrix, the number of columns N of the second matrix, the number of columns K of the first matrix, and the number of rows K of the second matrix includes: Calculating the product of M, N, and K; When the product of M, N, and K is greater than or equal to a threshold value, determining that the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation; When the product of M, N, and K is less than the threshold value, determining that the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation.

5. A chip, characterized in that: Includes: A first computing unit (210), a second computing unit (220), and a processing unit (230), where the processing unit is configured to obtain a first matrix and a second matrix for which matrix multiplication is to be performed; Determining whether the matrix multiplication operation of the first matrix and the second matrix belongs to a large matrix operation or a small matrix operation according to the number of rows M of the first matrix, the number of columns N of the second matrix, the number of columns K of the first matrix, and the number of rows K of the second matrix, wherein the amount of computation of the large matrix operation is greater than that of the small matrix operation, and M, N, and K are all positive integers greater than or equal to 1; When the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation, the first matrix and the second matrix are transmitted to the first computing unit for processing; And When the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation, the first matrix and the second matrix are transmitted to the second computing unit for processing.

6. The chip according to claim 5, characterized in that: The first computing unit is a scalable vector extension (SVE) computing unit, and the second computing unit is a scalable matrix extension (SME) computing unit.

7. The chip according to claim 5 or 6, characterized in that: The processing unit is specifically configured to obtain a threshold k from a preset mapping relationship according to M and N, and different values of M and N are set with corresponding thresholds k in the preset mapping relationship; When K >= k, it is determined that the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation; When K < k, it is determined that the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation.

8. The chip according to claim 5 or 6, characterized in that: The processing unit is specifically configured to calculate the product of M, N, and K; When the product of M, N, and K is greater than or equal to the threshold, it is determined that the matrix multiplication operation of the first matrix and the second matrix belongs to the large matrix operation; When the product of M, N, and K is less than the threshold, it is determined that the matrix multiplication operation of the first matrix and the second matrix belongs to the small matrix operation.

9. A computing device, characterized in that Comprising: At least one memory, At least one chip, where the at least one chip is configured to execute instructions stored in the at least one memory, so that the chip executes the method according to any one of claims 1-4.

10. A computer-readable storage medium, characterized in that: Comprising computer program instructions, when the computer program instructions are executed by a chip, the chip executes the method according to any one of claims 1-4.

Images