Method for calculating matrix multiplication, artificial intelligence chip, calculation device, medium and program product

Abstract

The invention relates to a method for calculating matrix multiplication, an artificial intelligence chip, a calculation device, a medium and a program product. The method comprises the following steps of: accumulating a calculation result of a current cycle calculation and a calculation result of a previous cycle calculation of matrix multiplication executed in a thread bundle group granularity by utilizing a buffer which is configured in a calculation core and is used for accumulation operation; determining whether the last cycle calculation of the matrix multiplication performed at the thread bundle group granularity is completed; and in response to determining that the last cycle computation of the matrix multiplication performed at the thread bundle group granularity is completed, writing the computation results accumulated via the buffer to the register file. According to the invention, the write bandwidth of the register and the occupation of the register space can be obviously reduced.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of artificial intelligence technology, and more specifically to a method for calculating matrix multiplication, an artificial intelligence chip, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In existing methods for calculating matrix multiplication, such as with input left and right matrices to be multiplied, when performing matrix multiplication using existing matrix multiplication instructions (e.g., but not limited to WGMMA instructions), each loop can only calculate the multiplication of a fixed column dimension of the corresponding left matrix block or a fixed row dimension of the right matrix block. When the column dimension of the input left matrix to be multiplied is greater than the aforementioned fixed column dimension of the left matrix block, the computation unit must write data to the register after each loop calculation, thus significantly consuming register write bandwidth. Furthermore, in the above method, since the computation unit must write data to the register after each loop calculation, if it is desired to accumulate the column dimension of the right matrix, a large amount of register space is required to store the output results of each loop and accumulate the column dimension of the right matrix.

[0003] In summary, the shortcomings of existing methods for calculating matrix multiplication are: they significantly consume register write bandwidth, and for cases where it is necessary to accumulate the column dimensions of the right matrix, they require a large amount of register space. Summary of the Invention

[0004] This invention provides a method for calculating matrix multiplication, an artificial intelligence chip, a computing device, a computer-readable storage medium, and a computer program product, which can significantly reduce register write bandwidth and register space occupancy.

[0005] According to a first aspect of the present invention, a method for calculating matrix multiplication is provided. The method includes: accumulating the result of the current iteration of a matrix multiplication performed at the thread bundle group granularity with the result of the previous iteration using a buffer configured in a computation kernel for accumulation operations; determining whether the last iteration of the matrix multiplication performed at the thread bundle group granularity has been completed; and writing the calculation result accumulated via the buffer to a register file in response to determining that the last iteration of the matrix multiplication performed at the thread bundle group granularity has been completed.

[0006] According to a second aspect of the present invention, an artificial intelligence chip is provided, comprising: a tensor core including: a computation unit configured for performing matrix multiplication at the thread bundle granularity based on matrix multiplication instructions; a buffer for accumulation operations configured to accumulate the calculation result of the current iteration of the matrix multiplication performed at the thread bundle granularity with the calculation result of the previous iteration, and, in response to determining that the last iteration of the matrix multiplication performed at the thread bundle granularity has been completed, write the calculation result accumulated via the buffer to a register file, wherein the thread bundle includes a plurality of consecutive thread bundles. According to a third aspect of the present invention, a computing device is also provided. The computing device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the computing device to perform the method of the first aspect of the present invention.

[0007] According to a fourth aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method of the first aspect of the present invention.

[0008] According to a fifth aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a machine, performs the method of the first aspect of the present invention.

[0009] In some embodiments, the buffer for accumulation is configured in a tensor kernel. Accumulating the calculation result of the current iteration of matrix multiplication performed at the thread bundle granularity with the calculation result of the previous iteration includes: using a matrix multiplication instruction to cache the calculation result of the current iteration of matrix multiplication performed at the thread bundle granularity in the buffer configured in the tensor kernel, wherein the current iteration of matrix multiplication performed at the thread bundle granularity is a multiplication between the current left matrix block of the input left matrix and the current right matrix block of the input right matrix; and obtaining the calculation result of the previous iteration cached in the buffer so as to accumulate the calculation result of the current iteration with the calculation result of the previous iteration, and caching the accumulated result in the buffer configured in the tensor kernel.

[0010] In some embodiments, the third dimension of the computation size of the matrix multiplication instruction is variable, and the third dimension is either the row dimension of the input right matrix or the column dimension of the input left matrix. The first dimension of the computation size of the matrix multiplication instruction is the row dimension of the input left matrix, and the second dimension of the computation size of the matrix multiplication instruction is either the column dimension of the input right matrix or the column dimension of the output matrix.

[0011] In some embodiments, determining that the last loop computation of a matrix multiplication performed at the thread bundle granularity has been completed includes: determining that the last loop computation of a matrix multiplication performed at the thread bundle granularity has been completed in response to determining that all inner products of the column dimensions of the input right matrix have been processed.

[0012] In some embodiments, accumulating the calculation result of the current iteration of a matrix multiplication performed at the thread bundle granularity with the calculation result of the previous iteration includes: in response to determining that the third dimension of the calculation size of the matrix multiplication instruction is greater than the third dimension of the calculation size of the matrix multiplication performed at the thread bundle granularity, performing a matrix multiplication of a predetermined size at the thread bundle granularity using each iteration of the calculation, and accumulating the calculation result of the current iteration with the calculation result of the previous iteration.

[0013] In some embodiments, accumulating the calculation result of the current iteration of matrix multiplication performed at the thread bundle group granularity with the calculation result of the previous iteration further includes: in response to determining that the third dimension of the calculation size of the matrix multiplication instruction is less than or equal to the third dimension of the calculation size of the matrix multiplication performed at the thread bundle group granularity, performing matrix multiplication of a predetermined size cyclically at the thread bundles included in the thread bundle group as the granularity.

[0014] In some embodiments, the computation size of a matrix multiplication instruction is 64. 16 16k, one of the thread bundles comprises four consecutive thread bundles, each of the thread bundles performing matrix multiplication computations with a size of 16. 16 16k, where k is a natural number.

[0015] In some embodiments, the tensor kernel further includes a post-processing unit, including a transpose unit configured to transpose the computation result accumulated via a buffer in response to determining that the last loop computation of a matrix multiplication performed at the thread bundle granularity has been completed, so as to write it to a register file.

[0016] In some embodiments, the buffer used for accumulation operations does not round the mantissa of the accumulated result when accumulating the result of the current iteration of matrix multiplication performed at the thread bundle granularity with the result of the previous iteration. This invention, by utilizing a buffer configured in the computation core for accumulation operations during the execution of matrix multiplication iterations at the thread bundle granularity, conveniently stores and accumulates intermediate calculation results; and after the final iteration is completed, writes the calculation result accumulated via the buffer in the computation core to the register all at once; thereby significantly reducing register write bandwidth and register space usage.

[0017] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements.

[0019] Figure 1 The diagram illustrates a computing apparatus for implementing a method for calculating matrix multiplication according to an embodiment of the present invention.

[0020] Figure 2 A flowchart of a method for calculating matrix multiplication according to some embodiments of the present invention is shown.

[0021] Figure 3 A schematic diagram of the data flow for a method of calculating matrix multiplication according to some embodiments of the present invention is shown.

[0022] Figure 4 A schematic diagram of a method for calculating matrix multiplication according to some embodiments of the present invention is shown.

[0023] Figure 5 A schematic diagram of a method for a single-loop calculation of a single thread bundle according to some embodiments of the present invention is shown.

[0024] Figure 6 A schematic diagram of a general-purpose graphics processor according to some embodiments of the present invention is shown.

[0025] Figure 7 A schematic diagram of a computing device according to some embodiments of the present invention is shown.

[0026] Figure 8 A flowchart of a method for accumulating calculation results for loop calculations according to some embodiments of the present invention is shown.

[0027] Figure 9 Schematic diagrams of methods for calculating matrix multiplication according to other embodiments of the present invention are shown. In the various figures, the same or corresponding reference numerals denote the same or corresponding parts. Detailed Implementation

[0028] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0029] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0030] As described above, the traditional methods for calculating matrix multiplication have the following drawbacks: they significantly consume register write bandwidth and require a large amount of register space for cases where the column dimensions of the right matrix need to be accumulated.

[0031] In existing methods for calculating matrix multiplication, the size of the input left matrix is, for example, M. K. The size of the input right matrix is, for example, K. N (where M, K, and N are all natural numbers). For example, the input left matrix is ​​divided into four left matrix blocks: A0, A1, A2, and A3. The input right matrix is ​​divided into four right matrix blocks: B0, B1, B2, and B3. The arithmetic unit used for matrix multiplication performs iterative calculations based on the left and right matrix blocks. For example, the intermediate result of the first iterative calculation is D0 = A0. After B0 and D0 are generated, they are output from the computation unit to the register of the scalar core. The intermediate result of the second loop calculation is D1=A0. After B1 and D1 are generated, they are output to the register by the calculation unit. The intermediate result of the third loop calculation is D2=A0. After B2 and D2 are generated, they are output from the computation unit to the registers of the scalar core. The intermediate result of the fourth loop calculation is D3=A1. After B2 and D3 are generated, they are output to registers by the computation unit. It should be understood that if the output matrix from each loop calculation is written back to the register file in blocks, the register write bandwidth becomes a significant performance bottleneck. Furthermore, to achieve N-dimensional accumulation of the right matrix, a large amount of register space is required to store the intermediate calculation results from each loop calculation and perform N-dimensional accumulation. Therefore, the traditional method for calculating matrix multiplication has the following shortcomings: it significantly consumes register write bandwidth, and for cases requiring column-dimensional accumulation of the right matrix, it requires a large amount of register space.

[0032] To at least partially address one or more of the aforementioned problems and other potential issues, exemplary embodiments of the present invention propose a method for calculating matrix multiplication. In this method, a buffer configured in the computation core for accumulation operations is used to accumulate the calculation result of the current iteration of matrix multiplication performed at the thread bundle group granularity with the calculation result of the previous iteration; it is determined whether the final iteration of matrix multiplication performed at the thread bundle group granularity has been completed; and in response to determining that the final iteration of matrix multiplication performed at the thread bundle group granularity has been completed, the calculation result accumulated via the buffer is written to a register file. The present invention significantly reduces register write bandwidth and register space usage by conveniently storing and accumulating intermediate calculation results during the execution of matrix multiplication iterations at the thread bundle group (including multiple consecutive thread bundles) granularity, utilizing a buffer configured in the computation core for accumulation operations; and by writing the calculation result accumulated via the buffer within the computation core to the register all at once after the final iteration is completed.

[0033] Figure 1 A schematic diagram illustrates a computing apparatus 100 implementing a method for calculating matrix multiplication according to an embodiment of the present invention. Figure 1 As shown, the computing device 100 may have one or more processing units, including dedicated processing units such as graphics processing units (GPUs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), general-purpose computing on graphics processing units (GPGPUs), and general-purpose processing units such as CPUs. The computing device 100 also includes at least: a calculation result accumulation module 110, a loop calculation completion determination module 120, and a writing module 130.

[0034] Regarding the calculation result accumulation module 110, it is used to accumulate the calculation result of the current loop calculation of matrix multiplication performed at the thread bundle group granularity with the calculation result of the previous loop calculation by using the buffer configured in the calculation core for accumulation operation.

[0035] Regarding the loop calculation completion determination module 120, it is used to determine whether the last loop calculation of the matrix multiplication performed at the thread bundle granularity has been completed.

[0036] Regarding the write module 130, in response to determining that the last loop calculation of the matrix multiplication performed at the thread bundle granularity has been completed, it writes the calculation result accumulated via the buffer to the register file.

[0037] The following will combine Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 7 A method 200 for calculating matrix multiplication, according to an embodiment of the present invention, is described. Figure 2 A flowchart of a method 200 for calculating matrix multiplication according to some embodiments of the present invention is shown. Figure 3 A schematic diagram of a data flow 300 for calculating matrix multiplication according to some embodiments of the present invention is shown. Figure 4 A schematic diagram of a method for calculating matrix multiplication according to some embodiments of the present invention is shown. Figure 5 A schematic diagram of a method 500 for a single-loop calculation of a single thread bundle according to some embodiments of the present invention is shown. Figure 7 A schematic diagram of a computing device according to some embodiments of the present invention is shown. It should be understood that method 200 can, for example, be implemented in... Figure 1 The described computing device 100 performs the operation. Method 200 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0038] At step 202, the computing device 100 uses a buffer configured in the computing core for accumulation operations to accumulate the calculation result of the current loop calculation of matrix multiplication performed at the thread bundle group granularity with the calculation result of the previous loop calculation. The thread bundle group includes multiple consecutive thread bundles.

[0039] Regarding computational kernels, examples include tensor kernels. Tensor kernels (such as...) Figure 3As indicated by 330 in the diagram, it includes, for example, an operation unit 332 for performing matrix multiplication, which is, for example, a General Matrix Multiplication (GEMM) operation unit. The operation unit 332 is, for example, based on data acquired from shared memory (which is, for example,...). Figure 3 The right input matrix of the global shared memory (GSM) 320 shown (i.e., Figure 3 Matrix B 322 in the matrix), and from shared memory or registers (such as... Figure 3 The left input matrix (i.e., as indicated by the middle mark 310) Figure 3 Matrix A 312 in the dataset). In some embodiments, the computational kernel further includes a buffer for accumulation operations (...). Figure 3 (Not shown in the image). It should be understood that the output matrix of matrix multiplication (i.e., ...) Figure 3 The matrix D (314) is stored, for example, in a register. In some embodiments, this register is, for example, a register in the scalar kernel. In some embodiments, this register is, for example, a thread register. In some embodiments, the computation kernel further includes a preprocessing unit and a post-processing unit. Figure 7 As shown, the computational kernel includes a tensor kernel 2 700 comprising a general matrix multiplication unit 2 720, a preprocessing unit 2 710, a postprocessing unit 2 730, and a cache 2 740.

[0040] Regarding the buffer used for accumulation, it is configured within the tensor kernel. In some embodiments, the buffer for accumulation is configured within the computational unit included in the tensor kernel, for example, within a general matrix multiplication unit. In some embodiments, the buffer for accumulation is configured outside the computational unit, for example, as... Figure 7 As shown, the buffer used for accumulation operations can be a cache 2740 on the tensor kernel 2700, and the cache 2740 interacts with the general matrix multiplication unit 2720. In some embodiments, to reduce buffer usage, the accumulation result in the buffer used for accumulation can be continuously overwritten in multiple loops. For example, the calculation result of the current loop is accumulated with the calculation result of the previous loop, and the accumulated result overwrites the calculation result of the previous loop. In some embodiments, the buffer used for accumulation only accumulates the loop calculation results in the third dimension of the calculation size of the matrix multiplication performed at the thread bundle granularity.

[0041] A warp is, for example, the most basic unit of task execution. Each warp consists of a group of threads, where a thread is the smallest unit of execution using hardware. In some embodiments, a warp may contain 32 threads. The threads within a warp may execute in parallel, for example.

[0042] A warp group comprises multiple consecutive warps. In some embodiments, a warp group consists of four consecutive warps. For example, a warp group may contain 128 threads. Each warp, for example, can achieve a computation size of 16 per loop computation. 16 Matrix multiplication with a power of 16. (See reference) Figure 5 For a single loop calculation within a single thread bundle, the matrix multiplication of matrix block A0 and matrix block B0 results in matrix block D0 (e.g., from "A0"). (As shown in "B0->D0"); the result of matrix multiplication of matrix block A0 and matrix block B1 is, for example, matrix block D1 (e.g., from "A0"). (As shown in "B1->D1"); the result of matrix multiplication of matrix block A1 and matrix block B0 is, for example, matrix block D2 (e.g., from "A1"). (As shown in "B0->D2"); the result of matrix multiplication of matrix block A1 and matrix block B1 is, for example, matrix block D3 (e.g., from "A1"). (As shown in "B1->D3"). Here, D0, D1, D2, and D3 are four matrix blocks in the result matrix of one loop calculation of a single thread bundle. Matrix block A0 and matrix block A1 are matrix blocks of the input left matrix, and their sizes are, for example, 8... 16. Matrix blocks B0 and B1 are matrix blocks of the input right matrix, with sizes of, for example, 16. 8. The size of the resulting matrix from one loop calculation of a single thread bundle is 16. 16. For example Figure 5 As shown, each thread bundle, for example, each loop computation, can achieve a computation size of 16. 16 16.

[0043] like Figure 4 As shown, thread bundle 440 includes four threads: thread bundle 0, thread bundle 1, thread bundle 2, and thread bundle 3. Therefore, each thread bundle, for example, can achieve a computation size of 16 for each loop computation. 16 16. A thread bundle group consisting of four consecutive thread bundles, each loop calculating, can, for example, achieve a computation size of 64. 16 16. It should be understood that more efficient data exchange and synchronization are achieved between thread bundles within a thread bundle group. Computational tasks are allocated at the thread bundle group level. This invention, through cyclical calculations of matrix multiplication performed at the thread bundle group level, facilitates the collaborative processing of larger-scale matrix blocks.

[0044] The method for accumulating the calculation result of the current iteration of matrix multiplication performed at the thread bundle granularity with the calculation result of the previous iteration is as follows: The computing device 100 uses a matrix multiplication instruction to cache the calculation result of the current iteration of matrix multiplication performed at the thread bundle granularity in the buffer configured in the tensor kernel. The current iteration of matrix multiplication performed at the thread bundle granularity is a multiplication between the current left matrix block of the input left matrix and the current right matrix block of the input right matrix. The computing device 100 also retrieves the calculation result of the previous iteration cached in the buffer so as to accumulate the calculation result of the current iteration with the calculation result of the previous iteration, and caches the accumulated result in the buffer configured in the tensor kernel.

[0045] like Figure 4 As shown, thread bundle 440 is used to execute matrix multiplication instructions, such as instructing matrix multiplication based on the input left matrix 1410 (i.e., matrix A) and the input right matrix 1420 (i.e., matrix B). Figure 4 The marker 430 indicates the output matrix 1 (i.e., matrix D) of the matrix multiplication. The computational size of the above matrix multiplication is 64. 16 16k, where k is a natural number. Specifically, the size of the input left matrix 1 410 (i.e., matrix A) is 64. 16k, the size of the input right matrix 1 420 (i.e., matrix B) is 16k. 16. The output matrix 1430 (i.e., matrix D) has a size of 64. 16. Thread bundle group 440 includes thread bundle 0, thread bundle 1, thread bundle 2 and thread bundle 3. Figure 4 The size of the matrix multiplication performed by each thread bundle is 16. 16 16k. In some embodiments, each thread bundle executes the inner product of each left matrix block of the input left matrix, that is, the thread bundles of thread bundle group 440 equally divide the row dimension of the input left matrix. For example, thread bundle 0 executes the inner product of the first row of the input left matrix blocks (i.e., the elements of rows 0-15), thread bundle 1 executes the inner product of the second row of the input left matrix blocks (i.e., the elements of rows 16-31), and so on. The column dimension of the input left matrix, i.e., the row dimension of the input right matrix, is, for example, variable. In some embodiments, the column dimension of the input left matrix / row dimension of the input right matrix may be equally divided multiple times. For example, refer to Figure 4 For the output matrix block D0 of output matrix 1430, it is equal to the sum of the corresponding products of the k left matrix blocks (i.e., rows 0-15) of the first row of the input left matrix 1410 (i.e., the block corresponding to thread bundle 0) and the k right matrix blocks of the input right matrix 1420. Therefore, in the current iteration of the k-th iteration, the result of multiplying the corresponding left and right matrix blocks is temporarily stored at position D0 in the buffer configured within the computation kernel. Then, the result of the current iteration is summed with the result of the previous iteration, and this sum replaces the data at position D0 after the previous iteration.

[0046] It should be understood that, unlike existing technologies that write intermediate results of each loop calculation back to the registers of the scalar core, this invention utilizes a buffer configured within the computation core to temporarily store and accumulate intermediate results during the iterative calculation process. Frequent accumulation operations are performed on a buffer with extremely high access speed within the computation core, making the overhead of accumulating intermediate results almost negligible. Furthermore, it eliminates the need to output intermediate results of each loop calculation to registers outside the computation core for accumulation; therefore, this invention significantly reduces register write bandwidth and register space usage. Regarding matrix multiplication, it multiplies and accumulates the elements of the corresponding row in the input left matrix and the corresponding column in the input right matrix to obtain the element at the corresponding row and column of the output matrix. It can be understood that the column dimension of the input left matrix is ​​the same as the row dimension of the input right matrix.

[0047] In some embodiments, the thread bundle performs matrix multiplications of a predetermined size in each loop computation. For example, the computation size of the matrix multiplication performed at the thread bundle granularity is m. n k, where m is the first dimension, n is the second dimension, and k is the third dimension, where the size of the left input matrix for matrix multiplication performed at the thread bundle granularity is m. k, where the size of the input right matrix is ​​k n.

[0048] A method for accumulating the calculation result of the current iteration of a matrix multiplication performed at the thread-bundle granularity with the calculation result of the previous iteration using a buffer includes, for example: using a matrix multiplication instruction, caching the calculation result of the current iteration of a matrix multiplication performed at the thread-bundle granularity in the buffer configured in the tensor kernel, wherein the current iteration of the matrix multiplication performed at the thread-bundle granularity is a multiplication between the current left matrix block of the input left matrix and the current right matrix block of the input right matrix; and obtaining the calculation result of the previous iteration cached in the buffer, so as to accumulate the calculation result of the current iteration with the calculation result of the previous iteration, and caching the accumulated result in the buffer configured in the tensor kernel.

[0049] At step 204, the computing device 100 determines whether the last loop calculation of the matrix multiplication performed at the thread bundle granularity has been completed.

[0050] Regarding the final loop calculation, this is, for example, the last loop of a matrix multiplication loop calculation performed at the thread bundle granularity. In some embodiments, the number of loop calculations required for matrix multiplication performed at the thread bundle granularity depends at least on the computational size of the matrix multiplication instructions and the computational size of the thread array. In some embodiments, the computational size of the matrix multiplication instructions is, for example, 64. 16 16k, the computational size of a thread array (which, for example, consists of 4 consecutive thread bundles) is, for example, 64. 16 If 16 is the number of iterations required for the loop calculation of matrix multiplication performed at the thread bundle granularity, then k is a natural number.

[0051] In some embodiments, the buffer used for accumulation only accumulates the results of loop calculations in the third dimension of the matrix multiplication computation size performed at the thread bundle granularity. Therefore, the number of cycles required for the aforementioned loop calculation is the number of cycles of the matrix multiplication computation in the third dimension performed at the thread bundle granularity. It should be understood that in the above embodiments, the buffer used for accumulation temporarily stores the results of loop calculations of the matrix multiplication computation in the first or second dimension performed at the thread bundle granularity, without accumulating them.

[0052] Regarding the computation size of matrix multiplication instructions, for example, it indicates a computation size of M. N Matrix multiplication of K. The size of the input left matrix is ​​M. K, where the size of the input right matrix is ​​K. N, and the size of the output matrix is ​​M. N; that is, the first dimension of the computational size of the matrix multiplication instruction is the row dimension of the input left matrix; the second dimension of the computational size of the matrix multiplication instruction is the column dimension of the input right matrix or the column dimension of the output matrix; the third dimension is the row dimension of the input right matrix or the column dimension of the input left matrix. It should be understood that the computational size of the matrix multiplication instruction (this computational size is, for example, "64") 16 The third dimension of "16k" (e.g., "16k") is variable. It should be understood that in some embodiments, the third dimension of the computational size of a matrix multiplication instruction can be greater than the third dimension of the computational size of a matrix multiplication performed at the thread bundle granularity. The third dimension of the computational size of a matrix multiplication instruction can also be less than or equal to the third dimension of the computational size of a matrix multiplication performed at the thread bundle granularity.

[0053] Regarding the method for determining whether the last loop calculation has been completed, the computing device 100, for example, determines whether all inner products of the column dimensions of the input right matrix have been processed. In response to determining that all inner products of the column dimensions of the input right matrix have been processed, it determines that the last loop calculation of the matrix multiplication executed at the thread bundle granularity has been completed.

[0054] At step 206, if the computing device 100 determines that the last loop calculation of the matrix multiplication performed at the thread bundle granularity has been completed, the calculation result accumulated via the buffer is written to the register file.

[0055] Regarding register files, in some embodiments, they are located, for example, in the thread local register (TLR). In some embodiments, they are located outside the tensor kernel, for example, in a register within the scalar kernel.

[0056] like Figure 3 As shown, for example, according to the matrix multiplication instruction, matrix multiplication is performed on matrices A 312 and B 322 to obtain matrix D 314. Figure 3 As shown, matrix A 312 is stored in thread register 310, and matrix B 322 is stored in global shared memory 320. During matrix multiplication, the matrix blocks of matrix A 312 and matrix B 322 are processed in the current iteration of the matrix multiplication within the operation unit 332 of the tensor core 330. Since the tensor core 330 is equipped with a buffer (not shown) for accumulation, the result of the previous iteration is cached in the buffer and accumulated with the result of the current iteration. After the final iteration is completed, the final accumulated result in the buffer, i.e., matrix D 314, is finally written into the thread register.

[0057] In the above scheme, the present invention utilizes a buffer configured in the computing core for accumulation operations to accumulate the calculation result of the current loop calculation of matrix multiplication performed at the thread bundle group granularity (the thread bundle group includes multiple consecutive thread bundles) with the calculation result of the previous loop calculation; determines whether the last loop calculation of matrix multiplication performed at the thread bundle group granularity has been completed; and in response to determining that the last loop calculation of matrix multiplication performed at the thread bundle group granularity has been completed, writes the calculation result accumulated via the buffer to the register file; the present invention can significantly reduce the write bandwidth and register space occupation of the register by utilizing the buffer configured in the computing core for accumulation operations during the matrix multiplication loop calculation performed at the thread bundle group granularity; and writes the calculation result accumulated via the buffer in the computing core to the register all at once after the last loop calculation is completed.

[0058] The following will combine Figure 8 A method 800 for accumulating calculation results for loop calculations, according to an embodiment of the present invention, is described. Figure 8 A flowchart of a method 800 for accumulating calculation results for loop calculations according to some embodiments of the present invention is shown. It should be understood that method 800 can, for example, be used in... Figure 1 The described computing device 100 performs the operation. Method 800 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0059] At step 802, the computing device 100 determines whether the third dimension of the computation size of the matrix multiplication instruction is less than or equal to the third dimension of the computation size of the matrix multiplication executed at the thread bundle granularity.

[0060] Regarding the computation size of matrix multiplication instructions, it is, for example, M. N K. Here, M represents the first dimension of the matrix multiplication instruction's computational size, which is equal to the row dimension of the input left matrix. N represents the second dimension of the matrix multiplication instruction's computational size, which is equal to the column dimension of the input right matrix or the column dimension of the output matrix. K represents the third dimension of the matrix multiplication instruction's computational size, which is either the row dimension of the input right matrix or the column dimension of the input left matrix. It should be understood that the third dimension of the matrix multiplication instruction's computational size can be greater than the third dimension of the matrix multiplication executed at the thread bundle group granularity; it can also be less than or equal to the third dimension of the matrix multiplication executed at the thread bundle group granularity.

[0061] For example, computing device 100 compares the third dimension of the computation size of a matrix multiplication instruction with the third dimension of the computation size of a matrix multiplication performed at the thread bundle granularity to determine whether the third dimension of the computation size of the matrix multiplication instruction is less than or equal to the third dimension of the computation size of the matrix multiplication performed at the thread bundle granularity.

[0062] At step 804, if the computing device 100 determines that the third dimension of the computation size of the matrix multiplication instruction is less than or equal to the third dimension of the computation size of the matrix multiplication executed at the thread bundle group granularity, the matrix multiplication of the predetermined size is cyclically calculated at the thread bundles included in the thread bundle group as the granularity.

[0063] For example, the computation size of a matrix multiplication instruction is, for example, 64. 16 8, with its third dimension being 8. The computational size of matrix multiplication performed at the thread bundle granularity is, for example, 64. 16 16. Its third dimension is 16. If the third dimension "8" of the computation size of the matrix multiplication instruction of the computing device 100 is less than the third dimension "16" of the computation size of the matrix multiplication executed at the thread bundle group granularity, then the computing device 100 performs a cyclic computation of a predetermined size matrix multiplication at the thread bundles included in the thread bundle group as the granularity, without using a buffer to accumulate the cyclic computation results.

[0064] At step 806, if the computing device 100 determines that the third dimension of the computation size of the matrix multiplication instruction is greater than the third dimension of the computation size of the matrix multiplication executed at the thread bundle granularity, it performs a matrix multiplication of a predetermined size at the thread bundle granularity using each loop calculation, and accumulates the calculation result of the current loop calculation with the calculation result of the previous loop calculation.

[0065] For example, the computation size of a matrix multiplication instruction is, for example, 64. 16 64, with its third dimension being 64. The computational size of matrix multiplication performed at the thread bundle granularity is, for example, 64. 16 The third dimension of the matrix multiplication instruction of computing device 100 is 16. If the third dimension "64" of the computation size of the matrix multiplication instruction is greater than the third dimension "16" of the computation size of the matrix multiplication performed at the thread bundle granularity, then computing device 100 performs a predetermined size (for example, 64) at the thread bundle granularity using each loop computation. 16 16) matrix multiplication is performed, and the calculation results of each loop calculation are temporarily stored in a buffer, and the calculation results of the current loop calculation are accumulated with the calculation results of the previous loop calculation. It should be understood that if the computing device 100 determines that the last loop calculation of the matrix multiplication performed at the thread bundle granularity has been completed, the calculation results accumulated via the buffer are written to the register file.

[0066] In the above scheme, the present invention can realize the accumulation of asynchronous execution and thread bundle execution granularity of matrix multiplication in the third dimension of cyclic calculation, thereby conveniently realizing matrix multiplication instructions with variable third dimension of calculation size.

[0067] Figure 6 A schematic diagram of a general-purpose graphics processor (or "GPGPU") 600 according to some embodiments of the present invention is shown. Figure 6 As shown, the general-purpose graphics processor 600 includes: shared memory 1610, tensor core 1620, register 1630, execution unit 640, control unit 650, memory access unit 660, secondary cache (e.g., L2 cache 670), and global memory 680. The tensor core 1620 includes, for example, a preprocessing unit 1622, a cache 1624, a general-purpose matrix multiplication unit 1626, and a post-processing unit 1628. The memory access unit 660 includes an address calculation unit 662 and a primary cache (e.g., L1 cache 664). Specifically, the tensor core 1620 is used to execute the method for calculating matrix multiplication according to the present invention. For example, the left matrix (i.e., matrix A) and matrix C are input from register 1630, the right matrix (i.e., matrix B) is input from shared memory 1610, and matrix multiplication is performed by the general-purpose matrix multiplication unit 1626. The output matrix (i.e., matrix D) is written to register 1630. In some embodiments, cache 1624 is configured as a buffer for accumulation operations. In other embodiments, the buffer for accumulation operations may be configured within a general matrix multiplication unit 1626. In some embodiments, the expression D = A B + C represents the matrix operation rules. In some embodiments, matrix C is, for example, a matrix of historical calculation results. It should be understood that in the iterative calculation of matrix multiplication at the thread bundle granularity, cache 1624 stores, for example, the sum of the calculation result of the current iteration and the calculation result of the previous iteration. Tensor kernel 1620 is also configured to utilize a cache for accumulation to implement the sum of the calculation result of the current iteration and the calculation result of the previous iteration, and to write the calculation result accumulated via the buffer to a register if it is determined that the last iteration of matrix multiplication performed at the thread bundle granularity has been completed.

[0068] It should be understood that in some embodiments, matrix C originates from register 1630. In some embodiments, such as when processing matrix multiplication with a large accumulation dimension, the accumulation dimension needs to be divided into multiple sub-matrices for multiple calculations, and a portion of the result obtained from each sub-matrix calculation is output to register 1630. In the next sub-matrix calculation, this portion of the result is retrieved from register 1630 for accumulation. In some embodiments, "accumulation or addition of matrix C" (hereinafter referred to as "+C") is only processed in the first few rounds of matrix multiplication of the corresponding sub-matrix.

[0069] It should be understood that in some embodiments, matrix C is, for example, a bias matrix. This corresponds to post-processing of the convolution result in a convolution operation, such as adding a bias term to the convolution result. Typically, the bias term needs to be calculated by the execution unit after the convolution result is output to a register, for example. According to some embodiments of this application, the addition of the bias matrix, i.e., "+C", can be performed directly in the tensor kernel 1620, thereby making the computation execution more efficient.

[0070] It should be understood that in some embodiments, "accumulation or addition of matrix C" is optional. When "accumulation or addition of matrix C" is not performed, it is not necessary to read matrix C from register 1630.

[0071] Figure 7A schematic diagram of a tensor core according to some embodiments of the present invention is shown. Tensor core 2 700 includes a computation core in an artificial intelligence chip. Tensor core 2 700 includes: an arithmetic unit (e.g., a general matrix multiplication arithmetic unit 2 720) and a buffer for accumulation operations. The arithmetic unit is configured for matrix multiplication performed at the thread bundle granularity based on matrix multiplication instructions. The buffer for accumulation operations is configured to accumulate the result of the current iteration of the matrix multiplication performed at the thread bundle granularity with the result of the previous iteration, and to write the accumulated result via the buffer to a register file in response to determining that the last iteration of the matrix multiplication performed at the thread bundle granularity has been completed, wherein the thread bundle includes a plurality of consecutive thread bundles. In some embodiments, tensor core 2 700 further includes a post-processing unit 2 730, which includes, for example, a second transpose unit 732. The second transpose unit 732 is configured, for example, to transpose the computation result accumulated via a buffer in response to determining that the last loop computation of a matrix multiplication performed at the thread bundle granularity has been completed, so as to write it to a register file. In some embodiments, the tensor kernel 2 700 further includes a preprocessing unit 2 710 and a cache 2 740. The preprocessing unit 2 710 includes a first transpose unit 712 and a first data type conversion unit 714. The postprocessing unit 2 730 further includes, for example, a second data type conversion unit 734. In some embodiments, matrix A comes from register 2 750, matrix B comes from shared memory 2 760, matrix C comes from a register, and the general matrix multiplication unit 2 720 performs D = A. The operation B + C results in a final value written to register 2750. In the tensor core 2700, cache 2740 is configured as a buffer for accumulation operations. During loop calculations, the general matrix multiplication unit 2720 sends the result of the current loop calculation (i.e., the intermediate result) to cache 2740 to accumulate the result of the current loop calculation with the result of the previous loop calculation, obtaining matrix C as the historical calculation result matrix, and caching the historical calculation result matrix. In some embodiments, cache 2740 also sends the historical calculation result matrix to the general matrix multiplication unit 2720. In some embodiments, for example, when processing matrix multiplication with a large accumulation dimension, the accumulation dimension needs to be divided into multiple sub-matrices for multiple calculations, and a portion of the result obtained from each sub-matrix calculation is output to cache 2740. In the next sub-matrix calculation, this portion of the result is retrieved from cache 2740 and sent to the general matrix multiplication unit 2720 for accumulation. In some embodiments, matrix C is, for example, a bias matrix, which corresponds to the post-processing of the convolution result in the convolution operation, such as adding a bias term to the convolution result. According to some embodiments of this application, the addition of the bias matrix, i.e., "+C", can be performed directly in the tensor kernel 2 700, thereby making the computation more efficient. The general matrix multiplication unit 2 720 sends the final calculation result, i.e., matrix D, to the post-processing unit 2 730 for subsequent calculations. The post-processing unit 2 730, for example, uses the second transpose unit 732 to transpose matrix D so that the transposed output matrix is ​​provided to the register.

[0072] In the above scheme, compared with the typical matrix multiplication method, the present invention utilizes a buffer configured in the computing core for accumulation operation to store the intermediate calculation results of multiple loops of matrix multiplication. The intermediate calculation results are accumulated to obtain the final calculation result, which conveniently realizes the accumulation in the third dimension of the calculation size of matrix multiplication. In addition, the final calculation result is output to the register at once, which significantly reduces the occupation of the register write bandwidth and alleviates the pressure on the register space.

[0073] The following will combine Figure 9 This invention describes a method 900 for accumulating the results of a cyclic calculation according to an embodiment of the invention, and specifically describes the number of accumulation rounds and the data structure in the method 900. Figure 9 A schematic diagram of a method 900 for calculating matrix multiplication according to other embodiments of the present invention is shown. It should be understood that method 900 can, for example, be used in... Figure 1The described computing device 100 performs the operation. Method 900 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0074] like Figure 9 As shown, for example, matrix multiplication is performed on the input left matrix 2910 (i.e., matrix A) and the input right matrix 2920 (i.e., matrix B). Figure 9 The marker 930 indicates the output matrix 2 (i.e., matrix D) of the matrix multiplication. The computational size of the above matrix multiplication is M. N K, where M, N, and K are natural numbers. Specifically, the size of the input left matrix 2910 is M. K, the size of the input right matrix 2920 is K. N, the size of the output matrix 2930 is M. N.

[0075] In the hardware architecture, for the above calculations, the size of the data buffered in the buffer is M. N. The computation size of a single operation in a Generalized Matrix Multiplication (GEMM) unit is subM. subN subK, where subM, subN, and subK are natural numbers. Based on the computational size of the general matrix multiplication unit, the input left matrix 2910 and the input right matrix 2920 are divided into multiple matrix blocks, where the size of each left matrix block of the input left matrix 2910 is subM. subK, where subK is the size of each block of the right matrix 2920. subN, the size of the output matrix block is subM subN. In some embodiments, M=16, N=32, subM=8, subN=8, and subK is determined according to the data type of the specific operation.

[0076] In Method 900, the inputs and operations performed in each round are shown in Table 1 below.

[0077] Table 1

[0078] As shown in the table above, since N=32 and subN=8, meaning N is 4 times subN, the input left matrix block can be sent every 4 rounds, and each data block can be reused 4 times. Therefore, the register read bandwidth utilization rate is 25%. The larger N is, the more times the input left matrix block is reused, and the less register read bandwidth is utilized. Furthermore, starting from round 8, for example, a corresponding data block is fetched from the buffer by the general matrix multiplication unit for accumulation.

[0079] It should be understood that, in some embodiments, method 900 is also used to execute the expression D = A B + C represents the matrix operation rules. The inputs and operations performed in each round are shown in Table 2 below.

[0080] Regarding matrix C, corresponding to matrix D which includes multiple output matrix blocks, matrix C includes multiple matrix blocks: C0, C1, C2, C3, C4, C5, C6, and C7. Among them, matrix block C0 is in the same position as output matrix block D0, matrix block C1 is in the same position as output matrix block D1, and so on.

[0081] In some embodiments, such as when processing matrix multiplication with a large accumulation dimension, the accumulation dimension needs to be divided into multiple sub-matrices for multiple calculations. In the next sub-matrix calculation, a portion of the result obtained from the previous sub-matrix calculation needs to be retrieved and accumulated. In some embodiments, "+C" is only executed in the first few rounds of the sub-matrix matrix multiplication. As shown in Table 2, for example, in the first 8 rounds, matrix blocks C0 to C7 of matrix C are read from the corresponding storage device, such as from a register, to perform "+C". From round 8 onwards, Table 2 is consistent with Table 1, where the "+C" operation is not performed, therefore, it will not be described again in Table 2.

[0082] It should be understood that when reading matrix C from the register in the first few rounds, performing the "+C" operation will increase the bandwidth utilization of the register to some extent.

[0083] In some embodiments, matrix C is, for example, a bias matrix. It corresponds to the post-processing of the convolution result in the convolution operation, such as adding a bias term to the convolution result.

[0084] Table 2

[0085] In traditional matrix multiplication operations, the output results are typically in fp32 format. Since the mantissa of a typical fp32 format occupies 23 bits, there is a rounding process from a high-precision intermediate result to a mantissa occupying 23 bits when outputting the result, which results in a loss of precision.

[0086] Therefore, in some embodiments of the present invention, the buffer used for accumulation operations does not round the mantissa of the accumulated result when accumulating the calculation result of the current iteration of matrix multiplication performed at the thread bundle granularity with the calculation result of the previous iteration. For example, the data stored in the buffer is represented using high precision as 940. Figure 9 As shown, in the high-precision representation 940, the sign bit 941 occupies 1 bit, the exponent bit 942 occupies 9 bits, and the mantissa bit 943 occupies 40 bits. This high-precision representation can improve the overall calculation accuracy, thus effectively mitigating the precision loss problem caused by type conversion during frequent data transfer of floating-point data.

[0087] The various processes and procedures described above, such as methods 200, 400, 500, 800, and 900, can be executed at a computing device. This computing device may include, for example, at least one processor (at least one graphics processor and at least one central processing unit); and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor. In some embodiments, methods 200, 400, 500, 800, and 900 may be implemented as a computer software program or program product tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on the computing device via read-only memory (ROM) and / or a communication unit. When the computer program is loaded into random-access memory (RAM) and executed by the GPU and CPU, one or more actions of methods 200, 400, 500, 800, and 900 described above can be performed.

[0088] This invention can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention. The computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0089] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. Various aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0090] These computer-readable program instructions can be provided to the central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the central processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0091] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0092] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps loaded in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0093] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors.

Claims

1. A method for calculating matrix multiplication, characterized in that, The method includes: Using a buffer configured in the computing core for accumulation operations, the calculation result of the current loop calculation of matrix multiplication performed at the thread bundle group granularity is accumulated with the calculation result of the previous loop calculation, wherein the thread bundle group includes multiple consecutive thread bundles. Determine whether the final loop calculation of the matrix multiplication performed at the thread bundle granularity has been completed; and In response to the completion of the last loop calculation of the matrix multiplication performed at the thread bundle granularity, the calculation result accumulated via the buffer is written to the register file.

2. The method according to claim 1, characterized in that, The buffer used for accumulation operations is configured in the tensor kernel, and the accumulation of the calculation result of the current iteration of matrix multiplication performed at the thread bundle granularity with the calculation result of the previous iteration includes: Using matrix multiplication instructions, the calculation result of the current iteration of matrix multiplication performed at the thread bundle granularity is cached in the buffer configured in the tensor kernel. The current iteration of matrix multiplication performed at the thread bundle granularity is a multiplication between the current left matrix block of the input left matrix and the current right matrix block of the input right matrix. as well as Retrieve the calculation result of the previous loop from the buffer so that the calculation result of the current loop can be accumulated with the calculation result of the previous loop, and the accumulated result can be cached in the buffer configured in the tensor kernel.

3. The method according to claim 2, characterized in that, The third dimension of the computation size of the matrix multiplication instruction is variable. The third dimension is either the row dimension of the input right matrix or the column dimension of the input left matrix. The first dimension of the computation size of the matrix multiplication instruction is the row dimension of the input left matrix, and the second dimension of the computation size of the matrix multiplication instruction is either the column dimension of the input right matrix or the column dimension of the output matrix.

4. The method according to claim 3, characterized in that, In response to determining that the final loop computation of matrix multiplication performed at the thread bundle granularity has been completed, the following steps are included: In response to determining that all inner products of the column dimensions of the input right matrix have been processed, it is determined that the last loop calculation of the matrix multiplication performed at the thread bundle granularity has been completed.

5. The method according to claim 1, characterized in that, The calculation result of the current iteration of matrix multiplication performed at the thread bundle granularity is accumulated with the calculation result of the previous iteration, including: In response to the determination that the third dimension of the computation size of the matrix multiplication instruction is greater than the third dimension of the computation size of the matrix multiplication executed at the thread bundle granularity, the matrix multiplication of the predetermined size is performed at the thread bundle granularity using each loop calculation, and the calculation result of the current loop calculation is accumulated with the calculation result of the previous loop calculation.

6. The method according to claim 5, characterized in that, Adding the result of the current iteration of matrix multiplication performed at the thread bundle granularity to the result of the previous iteration also includes: In response to the condition that the third dimension of the computation size of a matrix multiplication instruction is less than or equal to the third dimension of the computation size of a matrix multiplication performed at the thread bundle group granularity, matrix multiplications of a predetermined size are computed iteratively at the thread bundles included in the thread bundle group.

7. The method according to claim 3, characterized in that, The computation size of the matrix multiplication instruction is 64. 16 16k, one of the thread bundles comprises four consecutive thread bundles, each of the thread bundles performing matrix multiplication computations with a size of 16. 16 16k, where k is a natural number.

8. An artificial intelligence chip, characterized in that, The artificial intelligence chip includes: Tensor kernels include: The arithmetic unit is configured for matrix multiplication based on matrix multiplication instructions, executed at the thread bundle granularity; and A buffer for accumulation operations is configured to accumulate the result of the current loop calculation of a matrix multiplication performed at the thread bundle granularity with the result of the previous loop calculation, and to write the calculation result accumulated via the buffer to a register file in response to determining that the last loop calculation of the matrix multiplication performed at the thread bundle granularity has been completed, wherein the thread bundle includes a plurality of consecutive thread bundles.

9. The artificial intelligence chip according to claim 8, characterized in that, Tensor kernels also include: The post-processing unit includes a transpose unit configured to transpose the computation result accumulated via a buffer in response to determining that the last loop computation of a matrix multiplication performed at the thread bundle granularity has been completed, so as to write it to a register file.

10. The artificial intelligence chip according to claim 8, characterized in that, The buffer used for accumulation operations does not round the mantissa of the accumulated result when accumulating the result of the current iteration of matrix multiplication performed at the thread bundle granularity with the result of the previous iteration.

11. A computing device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method according to any one of claims 1-7.

13. A computer program product, characterized in that, Includes a computer program, which, when executed by a machine, performs the method according to any one of claims 1-7.

Images