Method for improving large matrix multiplication calculation efficiency by using on-chip second-level cache
By using on-chip secondary cache cache matrix data in the deep learning acceleration kernel, the number of matrix transmissions is reduced, and the problem of low computational efficiency of large matrix multiplication is solved, which significantly improves computing efficiency and system performance.
Patent Information
- Application Number
- CN202411772881.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-04
- Publication Date
- 2025-03-25
AI Technical Summary
When performing large matrix multiplication calculations, the existing technology has low computational efficiency due to excessive matrix transmission times, which has become a bottleneck in system performance.
By using the on-chip secondary cache, partial data of one matrix is cached, and partial data of another matrix is loaded into the first cache each time, and matrix multiplication is performed with all data of the first matrix in the second cache, reducing the number of transmissions of another matrix between the deep learning acceleration core and external memory.
It significantly improves the computational efficiency of large matrix multiplication in deep learning acceleration kernels, and improves system performance by reducing the number of matrix transmissions.
Smart Images
Figure CN119691341A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to integrated circuit design, and in particular to a method for improving large matrix multiplication calculation efficiency by utilizing an on-chip secondary cache. Background Art
[0002] Matrix multiplication is the most commonly used computing mode in the field of scientific computing. In the field of deep learning, matrix multiplication is also one of the most commonly used operators. Let the dimensions of the two matrices be U*V and V*W respectively, and the dimension of the result matrix is U*W, then the amount of calculation for matrix multiplication is U*V*W.
[0003] In the emerging transformer model, when calculating the attention mechanism, the Query (Q) tensor and the Key (K) tensor perform matrix multiplication to calculate the attention between each token. This is a typical matrix multiplication calculation, and the matrix dimension of the matrix multiplication calculation is proportional to the length of the input sequence. For the transformer model, the Q matrix and the K matrix are U*V and V*U matrices respectively. The calculation result is a U*U square matrix, where U is the length of the input sequence and V is the internal dimension of the attention mechanism. The amount of calculation of the matrix multiplication of the Q matrix and the K matrix is U*U*V, and the calculation complexity is proportional to the square of the input sequence length U. As the length of the sequence processed by the transformer model continues to increase, the input sequence length of a typical transformer model can even reach or exceed 2048 tokens. The time complexity of the matrix multiplication of calculating attention will become a bottleneck for transformer model reasoning.
[0004] For large matrix multiplication, since the matrix is very large, it is impossible to load it all into the L1 cache L1_buffer at once. A common method is to slice the matrix and calculate it in blocks. To perform matrix multiplication calculations on the deep learning core, a direct method is to load several rows of the left matrix and several columns of the right matrix each time, and after performing the matrix multiplication calculation, the final result of a small area in the result matrix is obtained, and then several rows of the left matrix or several columns of the right matrix are switched until the entire final result of the result matrix is obtained, such as Figure 3 shown.
[0005] Assume that the left matrix (denoted as A) needs to be sliced M times and the right matrix (denoted as B) needs to be sliced N times. Each time the left matrix is sliced in units of whole rows, that is, several whole rows are loaded each time; the right matrix is sliced in units of whole columns, that is, several whole columns are loaded each time. This slicing method can ensure that the two slices calculate the final result of the result matrix (denoted as C), so that the calculation results can be directly exported to the deep learning acceleration core. This slicing method avoids the generation of partial results of the result matrix. If the two slices generate partial results of the result matrix, the partial results still need to be calculated in the subsequent calculation process, and need to be repeatedly transferred between the deep learning acceleration core and the external memory, which will burden the computing performance. The size of the slice is related to the size of the matrix, the capacity of the first-level cache L1_buffer, and the size of the calculation array. One way to perform this slicing calculation is expressed in pseudo code as follows:
[0006]
[0007] In the above pseudocode, A(i) represents the data of the i-th slice of the left matrix, B(j) represents the data of the j-th slice of the right matrix, C(i,j) represents the result of the multiplication of A(i) and B(j) in the result matrix, which should be stored in the corresponding position in the result matrix, Load_slice represents loading a slice into the first-level cache L1_buffer; Matmul represents performing matrix multiplication calculations on two slices in the first-level cache L1_buffer; store_slice represents exporting the result of the multiplication of two slices from the deep learning acceleration core to the external memory.
[0008] In this data flow mode, a slice of the left matrix is loaded each time, and then all slices of the right matrix are looped, and each loop calculates a block in the result matrix. The data flow mode represented by the above pseudo code can complete the large matrix multiplication calculation. Analyzing the above pseudo code, we can see that the left matrix is loaded once, and the right matrix is loaded M times, and the result matrix is written out once. When M is very large, the repeated loading of the right matrix will become a bottleneck of system performance.
[0009] Of course, you can also change the data flow method, as shown in the following pseudo code:
[0010]
[0011] In this data flow mode, a slice of the right matrix is loaded each time, and then all slices of the left matrix are looped, and each loop calculates a block in the result matrix. Analyzing the above pseudo code, we can see that the left matrix is loaded N times, the right matrix is loaded once, and the result matrix is written out once. When processing large matrices, N may be very large, and the data transmission time of the left matrix will become the bottleneck of system performance.
[0012] It can be seen from this that no matter which data flow method is used, for large matrix multiplication calculations, it is inevitable that the data of a matrix will be transmitted repeatedly many times, becoming a bottleneck of system performance. Summary of the invention
[0013] 1. Technical issues to be solved
[0014] In view of the above-mentioned shortcomings of the prior art, the present invention provides a method for improving the efficiency of large matrix multiplication calculations by utilizing on-chip secondary cache, which can effectively overcome the defect of low efficiency of large matrix multiplication calculations caused by excessive matrix transmission times in the prior art.
[0015] (II) Technical solution
[0016] To achieve the above objectives, the present invention is implemented through the following technical solutions:
[0017] A method for improving the efficiency of large matrix multiplication calculation by using on-chip L2 cache comprises the following steps:
[0018] S1, load K slices that have not been calculated in a matrix into the L2 cache L2_buffer;
[0019] S2, load a slice that has not been calculated in another matrix into the first-level cache L1_buffer;
[0020] S3, sequentially load one slice of the K slices in the second-level cache L2_buffer into the first-level cache L1_buffer;
[0021] S4, perform matrix multiplication calculation on two slices in the first-level cache L1_buffer, and export the calculation result to the deep learning acceleration core;
[0022] S5. If there is an uncalculated slice in the L2_buffer, return to S3. If there is no uncalculated slice in the L2_buffer and there is an uncalculated slice in the other matrix, return to S2. If there is no uncalculated slice in the L2_buffer and there is no uncalculated slice in the other matrix, enter S6.
[0023] S6. If there are uncalculated slices in a matrix, return to S1, otherwise end;
[0024] Among them, one matrix is the left matrix A or the right matrix B, and the other matrix is the right matrix B or the left matrix A.
[0025] Preferably, the deep learning acceleration core includes a first-level cache L1_buffer, a second-level cache L2_buffer and a computing unit;
[0026] The first-level cache L1_buffer is a temporary cache area before data is sent to the computing unit, and has a small capacity;
[0027] The second-level cache L2_buffer is a large-capacity data cache area inside the deep learning acceleration core. It can temporarily cache some data that needs to be used repeatedly. Its capacity is larger than that of the first-level cache L1_buffer.
[0028] Computation unit, a functional unit that performs matrix multiplication calculations.
[0029] Preferably, for the data transmission channel of the deep learning acceleration core, the first-level cache L1_buffer and the second-level cache L2_buffer can both obtain data from outside the deep learning acceleration core, and the computing unit can only obtain data from the first-level cache L1_buffer;
[0030] Among them, in some deep learning acceleration cores, the first-level cache L1_buffer can only obtain data from the second-level cache L2_buffer, and the first-level cache L1_buffer can obtain data from the external memory by transferring part of the second-level cache L2_buffer area.
[0031] Preferably, for the data transmission speed of the deep learning acceleration core, the data transmission speed between the first-level cache L1_buffer, the second-level cache L2_buffer and the external memory of the deep learning acceleration core is much smaller than the data transmission speed between the first-level cache L1_buffer and the second-level cache L2_buffer, and the data transmission speed difference between the two is more than 10 times.
[0032] (III) Beneficial effects
[0033] Compared with the prior art, the present invention provides a method for improving the efficiency of large matrix multiplication calculation by using on-chip secondary cache. When performing large matrix multiplication calculation, the on-chip secondary cache is used to cache part of the data of a matrix, and part of the data of another matrix is loaded into the primary cache each time, and matrix multiplication calculation is performed with all the data of the first matrix in the secondary cache. The data transmission speed between the primary cache and the secondary cache inside the deep learning acceleration core is much faster than the data transmission speed between the deep learning acceleration core and the external memory, and the number of transmissions of another matrix between the deep learning acceleration core and the external memory is reduced. After experiments, the present method can significantly improve the computational efficiency of large matrix multiplication in the deep learning acceleration core. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0035] Figure 1 A schematic diagram of the storage hierarchy of the deep learning acceleration core in the present invention;
[0036] Figure 2 A schematic diagram of the present invention using a secondary cache to perform large matrix multiplication calculations;
[0037] Figure 3 A schematic diagram of the existing block calculation for large matrix multiplication. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0039] A method to improve the efficiency of large matrix multiplication calculations by using on-chip L2 cache, such as Figure 2 As shown, S1, load K slices that have not been calculated in a matrix into the second-level cache L2_buffer;
[0040] S2, load a slice that has not been calculated in another matrix into the first-level cache L1_buffer;
[0041] S3, sequentially load one slice of the K slices in the second-level cache L2_buffer into the first-level cache L1_buffer;
[0042] S4, perform matrix multiplication calculation on two slices in the first-level cache L1_buffer, and export the calculation result to the deep learning acceleration core;
[0043] S5. If there is an uncalculated slice in the L2_buffer, return to S3. If there is no uncalculated slice in the L2_buffer and there is an uncalculated slice in the other matrix, return to S2. If there is no uncalculated slice in the L2_buffer and there is no uncalculated slice in the other matrix, enter S6.
[0044] S6. If there are uncalculated slices in a matrix, return to S1, otherwise end;
[0045] Among them, one matrix is the left matrix A or the right matrix B, and the other matrix is the right matrix B or the left matrix A.
[0046] like Figure 1 As shown, the deep learning acceleration core includes a first-level cache L1_buffer, a second-level cache L2_buffer and a computing unit;
[0047] The first-level cache L1_buffer is a temporary cache area before data is sent to the computing unit, and has a small capacity;
[0048] The second-level cache L2_buffer is a large-capacity data cache area inside the deep learning acceleration core. It can temporarily cache some data that needs to be used repeatedly. Its capacity is larger than that of the first-level cache L1_buffer.
[0049] Computation unit, a functional unit that performs matrix multiplication calculations.
[0050] 1) For the data transmission channel of the deep learning acceleration core, both the first-level cache L1_buffer and the second-level cache L2_buffer can obtain data from outside the deep learning acceleration core, and the computing unit can only obtain data from the first-level cache L1_buffer;
[0051] Among them, in some deep learning acceleration cores, the first-level cache L1_buffer can only obtain data from the second-level cache L2_buffer (this does not hinder the effectiveness of the technical solution of the present application), and the first-level cache L1_buffer can obtain data from the external memory by transferring part of the area of the second-level cache L2_buffer.
[0052] 2) Regarding the data transmission speed of the deep learning acceleration core, the first-level cache L1_buffer, the second-level cache L2_buffer and the external memory of the deep learning acceleration core (such as Figure 1 The data transmission speed between the DDR in the cache is much slower than that between the L1 cache L1_buffer and the L2 cache L2_buffer. The data transmission speed difference between the two is more than 10 times (i.e. Figure 1 The B bit width is much larger than the D bit width).
[0053] The method proposed in this application assumes that the L2_buffer inside the deep learning acceleration core can store K slices of the left matrix A or the slices of the right matrix B. For the case where the L2_buffer stores K slices of the left matrix A, the pseudo code is expressed as follows:
[0054]
[0055]
[0056] In the above pseudo code, Load_slice_to_l2_buffer means loading K slices into the L2 cache L2_buffer, and Load_slice_from_l2_buffer means loading 1 slice from the L2 cache L2_buffer into the L1 cache L1_buffer.
[0057] Compared with the existing method of performing block calculations on large matrix multiplication, the above process is different in that K slices of the left matrix A are loaded into the L2 cache L2_buffer at one time, and then each slice of the left matrix A is no longer loaded from the external memory of the deep learning acceleration core, but is directly loaded from the L2 cache L2_buffer, and each slice of the right matrix B is retained in the L1 cache L1_buffer, and matrix multiplication calculations are performed with the K slices of the left matrix A in the L2 cache L2_buffer to obtain K calculation results, and then the next slice of the right matrix B is switched. Since the data transmission speed between the L1 cache L1_buffer and the L2 cache L2_buffer is very fast, the data transmission time between the L1 cache L1_buffer and the L2 cache L2_buffer can even be ignored.
[0058] In this way, after a slice of the right matrix B is loaded from the external memory to the first-level cache L1_buffer, it can perform matrix multiplication calculation with K slices of the left matrix A, such as Figure 2 As shown in the figure. Since each slice of the right matrix B is loaded from the external memory to the deep learning acceleration core and then calculated with the K slices of the left matrix A, the total number of transmissions of the right matrix B is reduced from M times to M / K times (assuming that the left matrix A needs to be sliced M times and the right matrix B needs to be sliced N times). When the capacity of the second-level cache L2_buffer is large, K is relatively large, so the above process can significantly reduce the number of transmissions of the right matrix B.
[0059] For the case where the L2_buffer stores K slices of the right matrix B, the pseudo code is as follows:
[0060]
[0061] Similarly, the above process uses the L2_buffer inside the deep learning acceleration core to store K slices of the right matrix B, and then loops through all slices of the left matrix A, which can reduce the number of transmissions of the left matrix A from N times to N / K times (assuming that the left matrix A needs to be sliced M times and the right matrix B needs to be sliced N times). When the capacity of the L2_buffer is large, K is relatively large, so the above process can significantly reduce the number of transmissions of the left matrix A.
[0062] Taking a deep learning acceleration core as an example, the capacity of the L2_buffer in the deep learning acceleration core is 16 times that of the L1_buffer, that is, K is 16; the data transmission speed between the L1_buffer and the L2_buffer is 16 times that between the L1_buffer and the external memory. For the multiplication calculation of two 8192*8192 large matrices, the optimization effect of the technical solution of this application is as follows:
[0063] When the L2_buffer is not used for optimization, according to the existing block calculation method for large matrix multiplication, the multiplication calculation of two 8192*8192 large matrices requires 2432436790 cycles after simulation;
[0064] After L2_buffer optimization, the multiplication of two 8192*8192 large matrices requires 416473419 cycles after simulation, and the computing efficiency is improved by 5.84 times.
[0065] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for improving the computational efficiency of large matrix multiplication using an on-chip secondary cache, characterized in that: The following steps are involved: S1, load K slices that have not been calculated in a matrix into the L2 cache L2_buffer; S2, load a slice that has not been calculated in another matrix into the first-level cache L1_buffer; S3, sequentially load one slice of the K slices in the second-level cache L2_buffer into the first-level cache L1_buffer; S4, perform matrix multiplication calculation on two slices in the first-level cache L1_buffer, and export the calculation result to the deep learning acceleration core; S5. If there is an uncalculated slice in the L2_buffer, return to S3. If there is no uncalculated slice in the L2_buffer and there is an uncalculated slice in the other matrix, return to S2. If there is no uncalculated slice in the L2_buffer and there is no uncalculated slice in the other matrix, enter S6. S6. If there are uncalculated slices in a matrix, return to S1, otherwise end; Among them, one matrix is the left matrix A or the right matrix B, and the other matrix is the right matrix B or the left matrix A.
2. The method for improving the computational efficiency of large matrix multiplication by using on-chip L2 cache according to claim 1, characterized in that: The deep learning acceleration core includes a first-level cache L1_buffer, a second-level cache L2_buffer and a computing unit; The first-level cache L1_buffer is a temporary cache area before data is sent to the computing unit, and has a small capacity; The second-level cache L2_buffer is a large-capacity data cache area inside the deep learning acceleration core. It can temporarily cache some data that needs to be used repeatedly. Its capacity is larger than that of the first-level cache L1_buffer. Computation unit, a functional unit that performs matrix multiplication calculations.
3. The method for improving the computational efficiency of large matrix multiplication by using on-chip L2 cache according to claim 2, characterized in that: For the data transmission channel of the deep learning acceleration core, the first-level cache L1_buffer and the second-level cache L2_buffer can both obtain data from outside the deep learning acceleration core, and the computing unit can only obtain data from the first-level cache L1_buffer; Among them, in some deep learning acceleration cores, the first-level cache L1_buffer can only obtain data from the second-level cache L2_buffer, and the first-level cache L1_buffer can obtain data from the external memory by transferring part of the second-level cache L2_buffer area.
4. The method for improving the computational efficiency of large matrix multiplication by using on-chip L2 cache according to claim 2, characterized in that: As for the data transmission speed of the deep learning acceleration core, the data transmission speed between the first-level cache L1_buffer, the second-level cache L2_buffer and the external memory of the deep learning acceleration core is much smaller than the data transmission speed between the first-level cache L1_buffer and the second-level cache L2_buffer, and the data transmission speed difference between the two is more than 10 times.