Device and method for realizing rapid calculation of continuous matrix multiplication
By using the "row bank first" and "column bank first" data placement methods in deep learning acceleration kernel NNA, the data input direction is adjusted, and the problem of low efficiency of continuous matrix multiplication calculation in the existing technology is solved, and fast calculation and flexible output mode switching is achieved.
Patent Information
- Application Number
- CN202510394574.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-31
- Publication Date
- 2025-08-01
AI Technical Summary
When performing continuous matrix multiplication calculations, the prior art requires explicit data placement conversion or adding hardware connections, resulting in low computing efficiency, especially in Transformer-like models, the calculation cost is rapidly increasing.
Deep learning acceleration kernel NNA and first-level cache L1_buffer are used to adjust the direction of data being sent to the MAC array through two data placement methods: "rows are preferred by bank" and "columns are preferred by bank" to adjust the direction of data being sent to the MAC array, and realize the switching of the output matrix between the two data placement methods, avoiding data format conversion and hardware connection adjustment.
The rapid calculation of continuous matrix multiplication is realized, which reduces the consumption of hardware resources, improves the calculation efficiency, and can flexibly switch the output mode according to the next step of matrix multiplication requirements.
Smart Images
Figure CN120408006A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a deep learning acceleration core NNA, and particularly to an apparatus and method for implementing fast calculation of continuous matrix multiplication. Background Art
[0002] Transformer has achieved a leapfrog development in the field of deep learning models. In terms of performance and performance for various tasks, it far exceeds deep learning models centered on convolutional neural networks. At the same time, large models centered on Transformer, such as ChatGPT, have further brought deep learning and artificial intelligence into people's daily lives. Large models will also bring huge economic value. According to McKinsey's research report on estimates for the enterprise side and global banks, large models will bring revenue increases of 2.6 - 4.4 trillion US dollars and 2.8% - 4.7% respectively. In China, large models are also in a stage of rapid development, with broad development potential and market prospects.
[0003] In Transformer and large models, continuous matrix multiplication calculation plays a core role. Taking Transformer as an example, the core structure of the Transformer model includes a self-attention mechanism (Self-Attention) and a feedforward neural network (Feedforward Neural Network). In the self-attention mechanism, it is necessary to calculate matrices of queries, keys, and values, and calculate attention scores and weighted sums through continuous matrix multiplication. This part of the calculation occupies the vast majority of the computational workload, and it is not a simple matrix multiplication, but continuous multiple matrix multiplications, that is, the calculation result of the previous matrix multiplication is usually used as the input for the next matrix multiplication. In multi-head attention (Multi-Head Attention), the computational workload of continuous matrix multiplication further increases. In the feedforward neural network, it is usually composed of linear transformation layers, and each linear transformation layer also involves matrix multiplication calculation. Combining relevant research and experimental data, the computational workload of continuous matrix multiplication accounts for more than 90% in Transformer.
[0004] Therefore, providing a hardware solution capable of fast calculation of continuous matrix multiplication is a necessary condition for implementing the deployment of Transformer and large models centered on Transformer, and at the same time, the operation speed will directly affect the inference performance of the model.
[0005] Due to the different requirements for the data layout of the left matrix and the right matrix in matrix multiplication calculations, after the hardware architecture completes a matrix multiplication calculation, if it is necessary to perform another matrix multiplication calculation on the result of this calculation, the data layout of these two matrices must be different. To achieve the above purpose, the prior art usually requires explicit data layout conversion or add wiring on the hardware so that the calculation result can be arranged in either of the two data layouts through hardware configuration selection. However, both of the above two solutions will bring additional costs, especially in models such as Transformer that are centered on continuous matrix multiplication calculations, the costs will increase rapidly. Summary of the Invention
[0006] (1) Technical problems to be solved
[0007] In view of the above-mentioned shortcomings of the prior art, the present invention provides a device and method for realizing fast calculation of continuous matrix multiplication, which can effectively overcome the defect that it is difficult to efficiently realize continuous matrix multiplication calculation in the prior art.
[0008] (2) Technical solutions
[0009] To achieve the above object, the present invention is realized through the following technical solutions:
[0010] A device for realizing fast calculation of continuous matrix multiplication includes a deep learning acceleration core NNA and a first-level cache L1_buffer. The first-level cache L1_buffer includes an input cache input_buffer, a weight cache weight_buffer, and an output cache output_buffer; [[ID=!8]]
[0011] The deep learning acceleration core NNA includes a MAC array for executing Matrix instructions. The MAC array includes multiple processing units PE for performing calculations;
[0012] The input cache input_buffer stores the left matrix and the right matrix in the data layout of "row bank priority" and "column bank priority", and provides data to the MAC array in the row direction;
[0013] The weight cache weight_buffer stores the right matrix and the left matrix in the data layout of "column bank priority" and "row bank priority", and provides data to the MAC array in the column direction;
[0014] The output cache output_buffer stores the output matrix calculated by the MAC array in the data layout of "row bank priority" or "column bank priority".
[0015] Preferably, when the input buffer stores the left matrix in a data layout of "row-major per bank" and provides data to the MAC array in the row direction, and the weight buffer stores the right matrix in a data layout of "column-major per bank" and provides data to the MAC array in the column direction, the output buffer will store the output matrix calculated by the MAC array in a data layout of "row-major per bank".
[0016] When the weight buffer stores the left matrix in a data layout of "row-major per bank" and provides data to the MAC array in the column direction, and the input buffer stores the right matrix in a data layout of "column-major per bank" and provides data to the MAC array in the row direction, the output buffer will store the output matrix calculated by the MAC array in a data layout of "column-major per bank".
[0017] Preferably, the input buffer, weight buffer, and output buffer all contain multiple banks. The data of each bank in the input buffer is broadcast to a row of the MAC array, the data of each bank in the weight buffer is broadcast to a column of the MAC array, and each processing element PE in the MAC array performs multiply-accumulate calculations on the input data in the row direction and the input data in the column direction. The calculation results of one row or one column of the processing elements PE are concatenated and output to the corresponding bank of the output buffer.
[0018] Preferably, the data layout of "row-major per bank" is as follows:
[0019] Each row of data of the left matrix is folded by P bytes and placed in one bank. After one row of data is placed, the next row of data starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of each row of data, if it does not reach the bank boundary, 0 is filled until the bank boundary.
[0020] The data layout of "column-major per bank" is as follows:
[0021] The data of each column of the right matrix is folded by P bytes and placed in one bank. After one column of data is placed, the next column of data starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of each column of data, if it does not reach the bank boundary, 0 is filled until the bank boundary is reached.
[0022] Preferably, the number of rows of the processing element PE in the MAC array is R, and the number of columns is C. The input data in the row direction and the input data in the column direction of each processing element PE are both P * 8 bits. Each processing element PE outputs an 8-bit calculation result per cycle. Each cycle, a row of processing elements PE outputs C * 8-bit calculation results to the corresponding bank of the output buffer output_buffer, or each cycle, a column of processing elements PE outputs R * 8-bit calculation results to the corresponding bank of the output buffer output_buffer;
[0023] The number of banks of the input buffer input_buffer is the same as the number of rows R of the processing element PE in the MAC array. The bit width of each bank in the input buffer input_buffer is P * 8, and the total bit width is R * P * 8;
[0024] The number of banks of the weight buffer weight_buffer is the same as the number of columns C of the processing element PE in the MAC array. The bit width of each bank in the weight buffer weight_buffer is P * 8, and the total bit width is C * P * 8;
[0025] When the output buffer output_buffer stores the output matrix calculated by the MAC array in the data placement method of "row bank priority", the number of banks of the output buffer output_buffer is the same as the number of rows R of the processing element PE in the MAC array. The bit width of each bank in the output buffer output_buffer is C * 8, and the total bit width is R * C * 8;
[0026] When the output buffer output_buffer stores the output matrix calculated by the MAC array in the data placement method of "column bank priority", the number of banks of the output buffer output_buffer is the same as the number of columns C of the processing element PE in the MAC array. The bit width of each bank in the output buffer output_buffer is R * 8, and the total bit width is C * R * 8.
[0027] A method for realizing fast calculation of continuous matrix multiplication, based on the characteristics of two data placement methods of "row bank priority" and "column bank priority", adjusts the direction of data sent to the MAC array to control the switching of the output matrix between the two data placement methods of "row bank priority" and "column bank priority", thereby realizing fast calculation of continuous matrix multiplication and finally realizing model acceleration inference.
[0028] Preferably, when the output matrix is used as the left matrix in the next matrix multiplication calculation, the left matrix in the current matrix multiplication calculation is stored in the input buffer input_buffer in the data placement method of "row bank priority", and the input buffer input_buffer provides data to the MAC array in the row direction. At the same time, the right matrix in the current matrix multiplication calculation is stored in the weight buffer weight_buffer in the data placement method of "column bank priority", and the weight buffer weight_buffer provides data to the MAC array in the column direction, so as to control the output matrix calculated by the MAC array to be stored in the output buffer output_buffer in the data placement method of "row bank priority".
[0029] When the output matrix is used as the right matrix in the next matrix multiplication calculation, the left matrix in the current matrix multiplication calculation is stored in the weight buffer weight_buffer in the data placement method of "row bank priority", and the weight buffer weight_buffer provides data to the MAC array in the column direction. At the same time, the right matrix in the current matrix multiplication calculation is stored in the input buffer input_buffer in the data placement method of "column bank priority", and the input buffer input_buffer provides data to the MAC array in the row direction, so as to control the output matrix calculated by the MAC array to be stored in the output buffer output_buffer in the data placement method of "column bank priority".
[0030] Preferably, the input buffer input_buffer, the weight buffer weight_buffer, and the output buffer output_buffer all include multiple banks. The data of each bank of the input buffer input_buffer is broadcast to a row of the MAC array, the data of each bank of the weight buffer weight_buffer is broadcast to a column of the MAC array, and each processing element PE in the MAC array performs multiply-accumulate calculation on the input data in the row direction and the input data in the column direction. The calculation results of one row or one column of the processing elements PE are concatenated and output to the corresponding bank of the output buffer output_buffer.
[0031] Preferably, the data arrangement method of "rows prioritizing banks" is as follows:
[0032] The data of each row of the left matrix is folded by P bytes and placed in one bank. After the data of one row is placed, the data of the next row starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of the data of each row, if it does not reach the bank boundary, 0 is filled until the bank boundary.
[0033] The data arrangement method of "columns prioritizing banks" is as follows:
[0034] The data of each column of the right matrix is folded by P bytes and placed in one bank. After the data of one column is placed, the data of the next column starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of the data of each column, if it does not reach the bank boundary, 0 is filled until the bank boundary.
[0035] (III) Advantageous Effects
[0036] Compared with the prior art, an apparatus and method for implementing fast calculation of continuous matrix multiplication provided by the present invention have the following advantageous effects:
[0037] 1) Based on the two data arrangement methods of "rows prioritizing banks" and "columns prioritizing banks" and the MAC array in the deep learning acceleration core NNA, by adjusting the direction of data sent into the MAC array, the controllability of the output mode is achieved, and both the output matrix in the form of "rows prioritizing banks" and the output matrix in the form of "columns prioritizing banks" can be output.
[0038] 2) Using the solution proposed by the present invention, fast calculation of continuous matrix multiplication can be realized, that is, according to the left and right matrix formats required for the next matrix multiplication calculation, the output mode is specified. If the left matrix is required for the next matrix multiplication calculation, the output matrix is controlled to be stored in the data arrangement method of "rows prioritizing banks"; if the right matrix is required for the next matrix multiplication calculation, the output matrix is controlled to be stored in the data arrangement method of "columns prioritizing banks", thereby realizing fast calculation of continuous matrix multiplication without data format conversion and adjustment.
[0039] 3) Using the solution proposed by the present invention, only one hardware wiring design is required to control the switching of the output matrix between the two data arrangement methods of "rows prioritizing banks" and "columns prioritizing banks", effectively saving hardware wiring resources. Description of the Drawings
[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0041] Figure 1 This is the hardware architecture diagram of the output buffer output_buffer in the present invention for storing the output matrix in the data placement manner of "row by bank priority".
[0042] Figure 2 This is the hardware architecture diagram of the output buffer output_buffer in the present invention for storing the output matrix in the data placement manner of "column by bank priority".
[0043] Figure 3 This is the schematic diagram of storing the left matrix in the data placement manner of "row by bank priority" in the present invention.
[0044] Figure 4 This is the schematic diagram of storing the right matrix in the data placement manner of "column by bank priority" in the present invention.
[0045] Figure 5 This is the schematic diagram of the output buffer output_buffer in the present invention for storing the output matrix in the data placement manner of "row by bank priority".
[0046] Figure 6 This is the schematic diagram of the output buffer output_buffer in the present invention for storing the output matrix in the data placement manner of "column by bank priority".
[0047] Figure 7 This is an example diagram for performing fast calculation of continuous matrix multiplication using the technical solution of the present invention. Detailed implementation manners
[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, rather than all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.
[0049] A device for realizing fast calculation of continuous matrix multiplication, such as Figure 1 And Figure 2As shown, it includes a deep learning acceleration core NNA and a first-level cache L1_buffer. The first-level cache L1_buffer includes an input cache input_buffer, a weight cache weight_buffer, and an output cache output_buffer;
[0050] The deep learning acceleration core NNA includes a MAC array for executing Matrix instructions. The MAC array includes multiple processing units PE for performing calculations;
[0051] The input cache input_buffer stores the left matrix and the right matrix in the data layout of "row-major per bank" and "column-major per bank", and provides data to the MAC array in the row direction;
[0052] The weight cache weight_buffer stores the right matrix and the left matrix in the data layout of "column-major per bank" and "row-major per bank", and provides data to the MAC array in the column direction;
[0053] The output cache output_buffer stores the output matrix calculated by the MAC array in the data layout of "row-major per bank" or "column-major per bank".
[0054] The input cache input_buffer, the weight cache weight_buffer, and the output cache output_buffer all include multiple banks. The data of each bank in the input cache input_buffer is broadcast to a row of the MAC array, and the data of each bank in the weight cache weight_buffer is broadcast to a column of the MAC array. Each processing unit PE in the MAC array performs multiply-accumulate calculations on the input data in the row direction and the input data in the column direction. The calculation results of a row or a column of processing units PE are concatenated and output to the corresponding bank of the output cache output_buffer.
[0055] In the technical solution of this application, as Figure 3 (assuming the number of banks is 16, P is 16, and a schematic diagram of storing a 64*64 matrix of fp16 in the data layout of "row-major per bank", where one fp16 data occupies 2 bytes), the data layout of "row-major per bank" is as follows:
[0056] The data of each row of the left matrix is folded by P bytes and placed in one bank. After one row of data is placed, the data of the next row starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of each row of data, if it does not reach the bank boundary, 0 is filled until the bank boundary;
[0057] As Figure 4 (assuming the number of banks is 16, P is taken as 16, a schematic diagram of storing a 64*64 matrix of fp16 in the data layout of "column-major by bank", where one fp16 data occupies 2 bytes), the data layout of "column-major by bank" is as follows:
[0058] The data of each column of the right matrix is folded by P bytes and placed in one bank. After one column of data is placed, the next column of data starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of each column of data, if it does not reach the bank boundary, 0 is filled until the bank boundary.
[0059] As Figure 1 and Figure 2 shown, the number of rows of the processing element PE in the MAC array is R, and the number of columns is C (the values of R and C can be equal). The input data in the row direction and the input data in the column direction of each processing element PE are both P*8 bits. Each processing element PE outputs an 8-bit calculation result per cycle. Each cycle, one row of processing elements PE outputs C*8-bit calculation results to the corresponding bank of the output buffer output_buffer, or each cycle, one column of processing elements PE outputs R*8-bit calculation results to the corresponding bank of the output buffer output_buffer;
[0060] The number of banks in the input buffer input_buffer is the same as the number of rows R of the processing element PE in the MAC array. The bit width of each bank in the input buffer input_buffer is P*8, and the total bit width is R*P*8;
[0061] The number of banks in the weight buffer weight_buffer is the same as the number of columns C of the processing element PE in the MAC array. The bit width of each bank in the weight buffer weight_buffer is P*8, and the total bit width is C*P*8;
[0062] When the output buffer output_buffer stores the output matrix calculated by the MAC array in the data layout of "row-major by bank", the number of banks in the output buffer output_buffer is the same as the number of rows R of the processing element PE in the MAC array. The bit width of each bank in the output buffer output_buffer is C*8, and the total bit width is R*C*8;
[0063] When the output buffer output_buffer stores the output matrix calculated by the MAC array in the data arrangement mode of "bank - priority by column", the number of banks of the output buffer output_buffer is the same as the number of columns C of the processing elements PE in the MAC array, the bit - width of each bank in the output buffer output_buffer is R * 8, and the total bit - width is C * R * 8.
[0064] As Figure 1 and Figure 5 shown, when the input buffer input_buffer stores the left matrix in the data arrangement mode of "bank - priority by row" and provides data to the MAC array in the row direction, and the weight buffer weight_buffer stores the right matrix in the data arrangement mode of "bank - priority by column" and provides data to the MAC array in the column direction, the output buffer output_buffer will store the output matrix calculated by the MAC array in the data arrangement mode of "bank - priority by row";
[0065] As Figure 2 and Figure 6 shown, when the weight buffer weight_buffer stores the left matrix in the data arrangement mode of "bank - priority by row" and provides data to the MAC array in the column direction, and the input buffer input_buffer stores the right matrix in the data arrangement mode of "bank - priority by column" and provides data to the MAC array in the row direction, the output buffer output_buffer will store the output matrix calculated by the MAC array in the data arrangement mode of "bank - priority by column".
[0066] As Figure 5 shown, when the left matrix is stored in the input buffer input_buffer in the data arrangement mode of "bank - priority by row" and sent into the MAC array in the row direction, and at the same time the right matrix is stored in the weight buffer weight_buffer in the data arrangement mode of "bank - priority by column" and sent into the MAC array in the column direction, the output matrix calculated by the MAC array is stored in the output buffer output_buffer in the data arrangement mode of "bank - priority by row".
[0067] The specific operation process can be combined with Figure 1 , the first bank in the input buffer input_buffer stores the 1st row, the (R + 1)th row... the (nR + 1)th row of the left matrix, the second bank stores the 2nd row, the (R + 2)th row... the (nR + 2)th row of the left matrix, and so on. Specifically, it can be referred to Figure 3, if R = 16 (consistent with the number of banks), the data of each bank is broadcast to a row of the MAC array. Similarly, the first bank in the weight buffer weight_buffer stores the 1st column, (C + 1)th column, ..., (nC + 1)th column of the right matrix, the second bank stores the 2nd column, (C + 2)th column, ..., (nC + 2)th column of the right matrix, and so on. For details, please refer to Figure 4 , if C = 16 (consistent with the number of banks), the data of each bank is broadcast to a column of the MAC array. As Figure 1 shown, the calculation results of the processing elements PE in the first row of the MAC array are directly stored in the first bank of the output buffer output_buffer after stitching. Then, the first bank will store all the data of the 1st row, (R + 1)th row, ..., (nR + 1)th row of the output matrix, that is, the result of multiplying the data of the first bank in the input buffer input_buffer in the row direction by the data of all C banks in the weight buffer weight_buffer in the column direction. And so on, finally, the output matrix is stored in the output buffer output_buffer in the data layout mode of "bank-first by row".
[0068] As Figure 6 shown, when the left matrix is stored in the weight buffer weight_buffer in the data layout mode of "bank-first by row" and sent into the MAC array from the column direction, and at the same time the right matrix is stored in the input buffer input_buffer in the data layout mode of "bank-first by column" and sent into the MAC array from the row direction, the output matrix calculated by the MAC array is stored in the output buffer output_buffer in the data layout mode of "bank-first by column".
[0069] For the specific operation process, please refer to Figure 2 , the first bank in the weight buffer weight_buffer stores the 1st row, (C + 1)th row, ..., (nC + 1)th row of the left matrix, the second bank stores the 2nd row, (C + 2)th row, ..., (nC + 2)th row of the left matrix, and so on. For details, please refer to Figure 3 , if C = 16 (consistent with the number of banks), the data of each bank is broadcast to a column of the MAC array. Similarly, the first bank in the input buffer input_buffer stores the 1st column, (R + 1)th column, ..., (nR + 1)th column of the right matrix, the second bank stores the 2nd column, (R + 2)th column, ..., (nR + 2)th column of the right matrix, and so on. For details, please refer to Figure 4 , if R = 16 (consistent with the number of banks), the data of each bank is broadcast to a row of the MAC array. As Figure 2As shown in the figure, the calculation results of the first column of processing units PE in the MAC array are directly stored in the first bank of the output buffer output_buffer after splicing. Then, the first bank will store all the data in the 1st column, (R + 1)th column, …, (nR + 1)th column of the output matrix, that is, the results of multiplying all the data in the R banks of the input buffer input_buffer in the row direction by the data in the first bank of the weight buffer weight_buffer in the column direction. And so on, finally, the output matrix is stored in the output buffer output_buffer in the data layout mode of "column bank first".
[0070] In the technical solution of this application, on the basis of the above disclosure of a device for realizing fast calculation of continuous matrix multiplication, a method for realizing fast calculation of continuous matrix multiplication is also disclosed, including:
[0071] Based on the characteristics of the two data layout modes of "row bank first" and "column bank first", by adjusting the direction of data sent into the MAC array, the output matrix is controlled to switch between the two data layout modes of "row bank first" and "column bank first", so as to realize fast calculation of continuous matrix multiplication and finally realize model acceleration inference.
[0072] The input buffer input_buffer, the weight buffer weight_buffer, and the output buffer output_buffer all contain multiple banks. The data in each bank of the input buffer input_buffer is broadcast to a row of the MAC array, the data in each bank of the weight buffer weight_buffer is broadcast to a column of the MAC array, and each processing unit PE in the MAC array performs multiply-accumulate calculation on the input data in the row direction and the input data in the column direction. The calculation results of one row or one column of processing units PE are spliced and output to the corresponding bank of the output buffer output_buffer.
[0073] In the technical solution of this application, as Figure 3 (assuming the number of banks is 16, P is taken as 16, and the schematic diagram of storing a 64 * 64 matrix of fp16 in the data layout mode of "row bank first", where one fp16 data occupies 2 bytes) shown, the data layout mode of "row bank first" is as follows:
[0074] The data in each row of the left matrix is folded by P bytes and placed in one bank. After one row of data is placed, the next row of data starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of each row of data, if it does not reach the bank boundary, 0 is filled until the bank boundary is reached;
[0075] As Figure 4 (assuming the number of banks is 16, P is 16, a schematic diagram of storing a 64*64 matrix of fp16 in the data layout of "column-major by bank", where one fp16 data occupies 2 bytes), the data layout of "column-major by bank" is as follows:
[0076] Each column of data in the right matrix is folded by P bytes and placed in one bank. After one column of data is placed, the next column of data starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of each column of data, if it does not reach the bank boundary, 0 is filled until the bank boundary.
[0077] As Figure 1 and Figure 5 shown, when the output matrix is used as the left matrix in the next matrix multiplication calculation, the left matrix in this matrix multiplication calculation is stored in the input buffer input_buffer in the data layout of "row-major by bank", and the input buffer input_buffer provides data to the MAC array in the row direction. At the same time, the right matrix in this matrix multiplication calculation is stored in the weight buffer weight_buffer in the data layout of "column-major by bank", and the weight buffer weight_buffer provides data to the MAC array in the column direction, so as to control the output matrix calculated by the MAC array to be stored in the output buffer output_buffer in the data layout of "row-major by bank";
[0078] As Figure 2 and Figure 6 shown, when the output matrix is used as the right matrix in the next matrix multiplication calculation, the left matrix in this matrix multiplication calculation is stored in the weight buffer weight_buffer in the data layout of "row-major by bank", and the weight buffer weight_buffer provides data to the MAC array in the column direction. At the same time, the right matrix in this matrix multiplication calculation is stored in the input buffer input_buffer in the data layout of "column-major by bank", and the input buffer input_buffer provides data to the MAC array in the row direction, so as to control the output matrix calculated by the MAC array to be stored in the output buffer output_buffer in the data layout of "column-major by bank".
[0079] The technical solution of this application mentions that based on the characteristics of the two data placement methods of "rows prioritizing banks" and "columns prioritizing banks", by adjusting the direction of data fed into the MAC array, the output matrix can be controlled to switch between the two data placement methods of "rows prioritizing banks" and "columns prioritizing banks", thereby achieving fast calculation of consecutive matrix multiplications and ultimately realizing accelerated inference of the model.
[0080] It can be seen that when encountering consecutive matrix multiplication calculations during the processing of the feature map, only by reasonably allocating the data placement method and adjusting the direction of data fed into the MAC array, the output matrix can be controlled to switch between the two data placement methods of "rows prioritizing banks" and "columns prioritizing banks", without the need to convert the data format or perform additional hardware design.
[0081] As Figure 7 shown, when it is necessary to perform matrix multiplication calculation on the result of matrix multiplication of matrices ⑤ and ⑥ and matrix ④ again, it can be controlled that the results of matrix multiplication of matrices ① and ②, and matrices ③ and ④ are stored in the data placement methods of "rows prioritizing banks" and "columns prioritizing banks" respectively, and then matrix multiplication calculation can be directly performed to obtain matrix ⑦ without converting the data format. Similarly, when performing matrix multiplication calculation on matrices ⑦ and ④, since matrix ④ has already been stored in the data placement method of "columns prioritizing banks", specifying that matrix ⑦ is stored in the data placement method of "rows prioritizing banks" can perform fast calculation of consecutive matrix multiplications to obtain matrix ⑧. For matrix ⑧, it can also be specified to be stored in the data placement method of "rows prioritizing banks" or "columns prioritizing banks" to facilitate the next matrix multiplication calculation.
[0082] The technical solution of this application performs instruction simulation on the output modes of two matrix multiplication calculations based on the deep learning acceleration core NNA, and confirms the correctness of the output results and the output modes.
[0083] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.
Claims
1. An apparatus for realizing fast calculation of continuous matrix multiplication, characterized in that: It includes a deep learning acceleration core NNA and a first-level cache L1_buffer, and the first-level cache L1_buffer includes an input cache input_buffer, a weight cache weight_buffer, and an output cache output_buffer; The deep learning acceleration core NNA includes a MAC array for executing Matrix instructions, and the MAC array includes multiple processing elements PE for performing calculations; The input cache input_buffer stores the left matrix and the right matrix in the data layout of "row-major per bank" and "column-major per bank", and provides data to the MAC array in the row direction; The weight cache weight_buffer stores the right matrix and the left matrix in the data layout of "column-major per bank" and "row-major per bank", and provides data to the MAC array in the column direction; The output cache output_buffer stores the output matrix calculated by the MAC array in the data layout of "row-major per bank" or "column-major per bank".
2. The device for quickly calculating continuous matrix multiplication according to claim 1, wherein: When the input cache input_buffer stores the left matrix in the data layout of "row-major per bank" and provides data to the MAC array in the row direction, and the weight cache weight_buffer stores the right matrix in the data layout of "column-major per bank" and provides data to the MAC array in the column direction, the output cache output_buffer will store the output matrix calculated by the MAC array in the data layout of "row-major per bank"; When the weight cache weight_buffer stores the left matrix in the data layout of "row-major per bank" and provides data to the MAC array in the column direction, and the input cache input_buffer stores the right matrix in the data layout of "column-major per bank" and provides data to the MAC array in the row direction, the output cache output_buffer will store the output matrix calculated by the MAC array in the data layout of "column-major per bank".
3. The device for quickly calculating continuous matrix multiplication according to claim 2, wherein: The input cache input_buffer, the weight cache weight_buffer, and the output cache output_buffer all include multiple banks. The data of each bank in the input cache input_buffer is broadcast to a row of the MAC array, the data of each bank in the weight cache weight_buffer is broadcast to a column of the MAC array, and each processing element PE in the MAC array performs multiply-accumulate calculations on the input data in the row direction and the input data in the column direction. The calculation results of one row or one column of the processing elements PE are concatenated and output to the corresponding bank of the output cache output_buffer.
4. The device for realizing fast calculation of continuous matrix multiplication according to claim 3, characterized in that: The data layout of "row-major per bank" is as follows: The data of each row of the left matrix is folded by P bytes and placed in a bank. After a row of data is placed, the next row of data starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of each row of data, if it does not reach the bank boundary, 0 is filled until the bank boundary is reached; The data placement method of "column bank - first" is as follows: The data of each column of the right matrix is folded by P bytes and placed in a bank. After a column of data is placed, the next column of data starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of each column of data, if it does not reach the bank boundary, 0 is filled until the bank boundary is reached.
5. The apparatus for rapidly calculating continuous matrix multiplication according to claim 4, characterized in that: In the MAC array, the number of rows of the processing element PE is R, and the number of columns is C. The input data in the row direction and the input data in the column direction of each processing element PE are both P * 8 bits. Each processing element PE outputs an 8 - bit calculation result per cycle. Each cycle, a row of processing elements PE outputs C * 8 bits of calculation results to the corresponding bank of the output buffer output_buffer, or each cycle, a column of processing elements PE outputs R * 8 bits of calculation results to the corresponding bank of the output buffer output_buffer; The number of banks of the input buffer input_buffer is the same as the number of rows R of the processing element PE in the MAC array. The bit - width of each bank in the input buffer input_buffer is P * 8, and the total bit - width is R * P * 8; The number of banks of the weight buffer weight_buffer is the same as the number of columns C of the processing element PE in the MAC array. The bit - width of each bank in the weight buffer weight_buffer is P * 8, and the total bit - width is C * P * 8; When the output buffer output_buffer stores the output matrix calculated by the MAC array in the data placement method of "row bank - first", the number of banks of the output buffer output_buffer is the same as the number of rows R of the processing element PE in the MAC array. The bit - width of each bank in the output buffer output_buffer is C * 8, and the total bit - width is R * C * 8; When the output buffer output_buffer stores the output matrix calculated by the MAC array in the data placement method of "column bank - first", the number of banks of the output buffer output_buffer is the same as the number of columns C of the processing element PE in the MAC array. The bit - width of each bank in the output buffer output_buffer is R * 8, and the total bit - width is C * R * 8.
6. A method for implementing fast calculation of continuous matrix multiplication, applied to the device for implementing fast calculation of continuous matrix multiplication described in claim 1, characterized in that: Based on the characteristics of the two data placement methods of "row-major by bank" and "column-major by bank", by adjusting the direction of data fed into the MAC array, the output matrix is controlled to switch between the two data placement methods of "row-major by bank" and "column-major by bank", thereby realizing the fast calculation of continuous matrix multiplication and finally realizing the accelerated inference of the model.
7. The method for quickly calculating continuous matrix multiplication according to claim 6, characterized in that: When the output matrix is used as the left matrix in the next matrix multiplication calculation, the left matrix in this matrix multiplication calculation is stored in the input buffer input_buffer in the data placement method of "row-major by bank", and the input buffer input_buffer provides data to the MAC array in the row direction. At the same time, the right matrix in this matrix multiplication calculation is stored in the weight buffer weight_buffer in the data placement method of "column-major by bank", and the weight buffer weight_buffer provides data to the MAC array in the column direction, so as to control the output matrix calculated by the MAC array to be stored in the output buffer output_buffer in the data placement method of "row-major by bank"; When the output matrix is used as the right matrix in the next matrix multiplication calculation, the left matrix in this matrix multiplication calculation is stored in the weight buffer weight_buffer in the data placement method of "row-major by bank", and the weight buffer weight_buffer provides data to the MAC array in the column direction. At the same time, the right matrix in this matrix multiplication calculation is stored in the input buffer input_buffer in the data placement method of "column-major by bank", and the input buffer input_buffer provides data to the MAC array in the row direction, so as to control the output matrix calculated by the MAC array to be stored in the output buffer output_buffer in the data placement method of "column-major by bank".
8. The method for quickly calculating continuous matrix multiplication according to claim 7, characterized in that: The input buffer input_buffer, the weight buffer weight_buffer, and the output buffer output_buffer all contain multiple banks. The data of each bank in the input buffer input_buffer is broadcast to a row of the MAC array, and the data of each bank in the weight buffer weight_buffer is broadcast to a column of the MAC array. Each processing element PE in the MAC array performs the multiply-accumulate calculation of the input data in the row direction and the input data in the column direction, and the calculation results of one row or one column of the processing elements PE are spliced and output to the corresponding bank of the output buffer output_buffer.
9. The method for quickly calculating consecutive matrix multiplication according to claim 8, characterized in that: The data placement method of "row-major by bank" is as follows: Each row of data of the left matrix is folded by P bytes and placed in one bank. After one row of data is placed, the next row of data starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of each row of data, if it does not reach the bank boundary, 0 is filled until the bank boundary. The data placement method of "column in bank priority" is as follows: Each column of data in the right matrix is folded by P bytes and placed in a bank. After a column of data is placed, the next column of data starts to be placed from the starting boundary of the next adjacent bank until it returns to the first bank. During the placement of each column of data, if it does not reach the bank boundary, 0 is filled until the bank boundary is reached.