Input data sharing and cache optimization method and system in matrix multiplication calculation and application

Abstract

The invention discloses an input data sharing and cache optimization method in matrix multiplication calculation. The method comprises the steps of 1, segmenting and distributing an input data matrix and a weight matrix according to the number N of calculation cores on a chip; 2, sequentially connecting the plurality of calculation cores end to end to form a data transmission annular structure; step 3, calculating the distributed matrix multiplication by each calculation core, and transmitting the current input sub-matrix of the calculation core to the next calculation core; step 4, performing matrix multiplication operation on the transmitted input sub-matrix and the weight sub-matrix in the next calculation kernel; and 5, iterating transmission and calculation of the input sub-matrixes, and carrying out N rounds of matrix multiplication of the input sub-matrixes and the weight sub-matrixes to complete the whole operation process. The invention further discloses a system for implementing the method, and the system has wide application value.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of chip design, and relates to an input data sharing and cache optimization method, system and application in matrix multiplication calculation. BACKGROUND

[0002] At present, the application of large models based on the Transformer architecture is thriving, and the demand for fast operation of large-scale matrices is becoming increasingly strong. Compared with traditional CPUs, ASIC chips can deploy more computing units for parallel operation, and the computing power advantage is obvious. The high concurrency of hardware supports the deployment of multiple computing cores, and a large-scale matrix is divided into multiple blocks for simultaneous calculation on average. Taking the Huawei Da Vinci architecture as an example, multi-level cache (L0 / L1 / L2) is used to reduce external storage access, and data reuse (such as fixed weight, fixed output, etc. data flow mode) is supported. The L0 cache is closely arranged next to the Cube core (responsible for matrix operation) and stores input data and weights. The L1 / L2 cache is used as a shared cache to store intermediate results and frequently accessed data. Each input data matrix needs to be multiplied by a fixed weight matrix, and when the weight is fixed, the L0 cache stores the input data that the computing unit needs to frequently read.

[0003] The storage system has a great influence on the performance, power consumption and area of the AI large model chip, and needs to be considered in the design of the AI chip architecture. The AI chip is also often limited by the storage system in terms of computing performance. In a conventional chip architecture, the input data matrix and the weight matrix are stored in a DRAM or other external storage chip, and the computing core needs to read and write data from the DRAM each time, and also needs to cache data through multiple levels of cache. Therefore, it is hoped that the current bottleneck of the AI chip in processing matrix multiplication calculation can be reduced as much as possible without reducing the computing performance, and the use of on-chip storage resources can be reduced, and the data interaction frequency between the cache and the external storage chip can be reduced. SUMMARY

[0004] In order to solve the problems in the prior art, the purpose of the present application is to provide an input data sharing and cache optimization method, system and application in matrix multiplication calculation, which realizes matrix data sharing between computing cores through hardware design and planning, evenly distributes data to multiple computing cores, effectively reduces on-chip RAM / Cache storage resources, reduces data interaction between the cache and the storage DRAM through data transmission between computing cores, and supports AI inference and / or training.

[0005] The present application provides an input data sharing and cache optimization method in matrix multiplication calculation, which is performed on the computing core of a chip; comprising:

[0006] Step one, dividing and distributing the input data matrix and the weight matrix according to the number N of computing cores on the chip;

[0007] Step two, sequentially connect the plurality of computing cores end to end to form a data transmission ring structure;

[0008] Step three, each computing core calculates the assigned matrix multiplication and passes the current input sub-matrix of the computing core to the next computing core;

[0009] Step four, the passed input sub-matrix performs matrix multiplication operation with the weight sub-matrix in the next computing core;

[0010] Step five, iterate the above input sub-matrix passing and calculation, a total of N rounds of matrix multiplication operation of input sub-matrix and weight sub-matrix are performed to complete the entire operation process.

[0011] In step one, the dimension of the input data matrix is WxC, and the dimension of the weight matrix is CxK;

[0012] The input data matrix and the weight matrix are distributed to N computing cores; the input data matrix is only segmented in dimension C, and the dimension of the segmented input sub-matrix is WxC / N; the weight matrix is segmented in dimension C and dimension K, and the dimension of the segmented weight sub-matrix is C / NxK / N;

[0013] Wherein, W represents the number of data samples passed through the model in one iteration (or one training step), and in the inference phase, such as in the autoregressive calculation in the decoding phase, the value of W is usually 1, corresponding to processing only one token each time; W can be flexibly adjusted according to the application scenario, and is usually much smaller than the dimensions C and K of the model, so in the present application, the W dimension is usually not segmented for calculation, and the C and K dimensions are mainly divided and processed in parallel.

[0014] In one specific embodiment of the present application, a multi-user mode is also supported, but the number of users accessed at the same time supported by a single chip does not exceed 64.

[0015] In step two, all the computing cores together form a ring structure, or the computing cores are divided into multiple groups, each group forming a ring structure; for each ring structure formed by the computing cores, the number of computing cores contained is ≥3;

[0016] When multiple ring structures are formed, if a computing core in any one or more ring structures less than the total number of ring structures fails, the ring structure containing the failed computing core is shielded, and the computing task is given to the ring structure without failure.

[0017] In step three, take one computing core in the ring structure as the starting point of counting, denoted as m_core_0, and the last computing core in the data transmission direction is denoted as m_core_i; wherein, i=N-1;

[0018] The first round of calculation is the matrix multiplication operation corresponding to each calculation core (W x C / N) m_core_0 · (C / N x K / N) m_core_0 , (W x C / N) m_core_1 · (C / N x K / N) m_core_1 , …, (W x C / N) m_core_i · (C / N x K / N) m_core_i ; then the input sub-matrix of the current calculation core is passed, the input sub-matrix (W x C / N) m_core_0 of the first calculation core m_core_0 is passed to the second calculation core m_core_1, the input sub-matrix (W x C / N) m_core_1 of the second calculation core m_core_1 is passed to the third calculation core m_core_2, and so on, the input sub-matrix (W x C / N) m_core_i of the last calculation core m_core_i is passed to the first calculation core m_core_0.

[0019] In one specific embodiment, the first round of calculation is that each calculation core calculates the matrix multiplication operation initially assigned to it, respectively represented as (1 x 64) m_core_0 · (64 x 64) m_core_0 , (1 x 64) m_core_1 · (64 x 64) m_core_1 , …, (1 x 64) m_core_i · (64 x 64) m_core_i ; the current input sub-matrix of the calculation core is passed to the next calculation core, i.e. the input sub-matrix (1 x 64) m_core_0 of the first calculation core m_core_0 is passed to the second calculation core m_core_1, the input sub-matrix (1 x 64) m_core_1 of the second calculation core m_core_1 is passed to the third calculation core m_core_2, and so on, the input sub-matrix (1 x 64) m_core_i of the last calculation core m_core_i is passed to the first calculation core m_core_0.

[0020] In one specific embodiment, the matrix multiplication operation in each calculation core and the backward passing of the input sub-matrix can be considered and performed in parallel.

[0021] In step four, the input sub-matrix passed by the previous calculation core and the weight sub-matrix of the current calculation core itself are subjected to matrix multiplication calculation, represented as (W x C / N) m_core_i · (C / N x K / N) m_core_0 , (W x C / N) m_core_0 · (C / N x K / N)m_core_1 ,..., (W x C / N) m_core_(i-1) • (C / N x K / N) m_core_i ;

[0022] In one embodiment, the input sub-matrix (1 x 64) m_core_i • (64 x 64) m_core_0 , (1 x 64) m_core_0 • (64 x 64) m_core_1 ,..., (1 x 64) m_core_(i-1) • (64 x 64) m_core_i .

[0023] In step five, when the transmission of a round of input sub-matrix is completed, during the calculation process, each calculation core can continue to transmit the input sub-matrix used in the current calculation to the next calculation core, that is, the input sub-matrix (W x C / N) m_core_0 is transmitted to the third calculation core m_core_2, the input sub-matrix (W x C / N) m_core_1 of the second calculation core m_core_1 is transmitted to the fourth calculation core m_core_3, and so on, the input sub-matrix (W x C / N) m_core_i of the last calculation core m_core_i is transmitted to the second calculation core m_core_1.

[0024] In one embodiment, the input sub-matrix (1 x 64) m_core_0 is transmitted to the third calculation core m_core_2, the input sub-matrix (1 x 64) m_core_1 of the second calculation core m_core_1 is transmitted to the fourth calculation core m_core_3, and so on, the input sub-matrix (1 x 64) m_core_i of the last calculation core m_core_i is transmitted to the second calculation core m_core_1.

[0025] Similarly, the input sub-matrix transmitted from the previous calculation core and the weight sub-matrix of the current calculation core are subjected to matrix multiplication calculation.

[0026] Similarly, the input sub-matrix transmitted from the previous calculation core and the weight sub-matrix of the current calculation core are subjected to matrix multiplication calculation. m_core_0 is transmitted to the last calculation core m_core_i, the input sub-matrix (W x C / N) m_core_1 of the second calculation core m_core_1 is transmitted to the first calculation core m_core_0, and so on, the input sub-matrix (W x C / N) m_core_i of the last calculation core m_core_i is transmitted to the second-to-last calculation core m_core_(i-1). The matrix multiplication operation of the input sub-matrix and the weight matrix is performed in each calculation core.

[0027] In one embodiment, the input sub-matrix (1x64) m_core_0 The input sub-matrix (1x64) of the second computing core m_core_1 is passed to the last computing core m_core_i m_core_1 The input sub-matrix (1x64) of the first computing core m_core_0 is passed to the last computing core m_core_i, and so on m_core_i The input sub-matrix (1x64) of the second last computing core m_core_(i-1) is passed to the last computing core m_core_i, and the matrix multiplication operation of the input sub-matrix and the weight matrix is performed in each computing core.

[0028] The entire operation is completed.

[0029] In the process of steps three to five above, the weight sub-matrix of each computing core does not participate in the transmission between the computing cores in the ring structure, and is written into the weight buffer of each computing core through the external DRAM before the start of each round of calculation.

[0030] In one embodiment, the same weight sub-matrix is used for the same computing core for different calculation rounds.

[0031] In one embodiment, during the calculation of the current round of matrix multiplication, the output sub-matrix required for the next round of calculation is simultaneously transmitted, and the weight sub-matrix required for the next round of calculation is preloaded and stored in the corresponding buffer.

[0032] In one embodiment, the inter-core distance between different computing cores is adjusted to minimize the transmission distance between each two computing cores.

[0033] The present application also provides an input data sharing and cache optimization system for implementing the above-mentioned matrix multiplication calculation, comprising one or more computing cores, the computing cores being arranged on a chip and being special matrix operation units designed for large model matrix multiplication optimization, and the computing cores being electrically connected through on-chip interconnection; each computing core comprising:

[0034] An input cache module for storing the input sub-matrix locally in the computing core;

[0035] A weight cache module for storing the weight sub-matrix corresponding to the input sub-matrix;

[0036] A computing module for performing multiplication and addition operation on the input sub-matrix and the weight sub-matrix;

[0037] A data transmission module for transmitting the input sub-matrix in the current computing core to the next computing core and receiving the input sub-matrix from the previous computing core;

[0038] The plurality of computing cores are constructed as a ring structure through on-chip interconnection, and support sequential transmission and parallel calculation of the input sub-matrix.

[0039] The data transmission module is configured to support N-1 times of data rotation transmission, and ensures that all computing cores can obtain all input sub-matrices to complete the complete matrix multiplication operation.

[0040] The application also provides the above method or the application of the above system in the design of a chip computing architecture supporting large language model reasoning or training.

[0041] The application also provides a hardware system for implementing the above method, which comprises a memory and a processor.

[0042] The application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the above method.

[0043] The application has the following beneficial effects: the application constructs an input data sharing mechanism among heterogeneous computing cores, and adopts a ring structure for efficient transmission of input matrices, so that input activation data can be reused among the plurality of computing cores after being loaded into a local cache once before the start of calculation, thereby greatly reducing the frequency of input data access to DRAM. On this basis, in combination with the strategy of loading weight sub-matrix locally according to the computing core, each computing core does not need to repeatedly load input matrix data and only needs to read weight data during the entire matrix multiplication and addition operation, thereby significantly reducing the DRAM bandwidth pressure. Compared with the traditional scheme, the application can reduce the on-chip cache usage by about 90%, reduce the data transmission bandwidth pressure by 20%, support full pipelining scheduling in the overall calculation process, effectively reduce the chip area and power consumption without reducing the calculation performance, and improve the overall efficiency and resource utilization of large model matrix calculation. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0045] Figure 1 is a ring structure data transmission schematic diagram of 16 computing cores in a specific implementation process of the application.

[0046] Figure 2is a schematic diagram of the position of the chip internal computing core in one embodiment of the application.

[0047] Figure 3 is a schematic diagram of the matrix multiplication calculation process of the eight computing cores in one embodiment of the application.

[0048] Figure 4 is a schematic diagram of the interconnection relationship and position among the 32 computing cores in one embodiment of the application.

[0049] Figure 5 is a schematic diagram of the multiplication of the input matrix (IA) and the weight matrix (Weight) to obtain the result matrix (OA) in one embodiment of the application.

[0050] Figure 6 is a schematic diagram of the calculation process in one embodiment of the application. DETAILED DESCRIPTION

[0051] The application will be further described in conjunction with the following specific embodiments and drawings. The process, conditions, experimental methods, etc. for implementing the application are generally known and commonly understood, and the application does not have special restrictions.

[0052] A chip implementing a large model based on the Transformer architecture needs to process large-scale matrix multiplication operations (GEMM), i.e., input data matrix multiplied by weight matrix. To improve computing power and matrix multiplication calculation efficiency, the chip is designed to have multiple cores for parallel processing. The matrix (WxC)·(CxK) is divided equally into N matrix computing cores. Each matrix computing core needs to process (WxC)·(CxK / N) matrix operations, and the matrix size is still large, requiring more cache resources; therefore, the dimension C can also be considered for division.

[0053] The matrix multiplication operation is shown in Figure 3 The dimensions of the matrix are also shown in the figure.

[0054] Specifically, large-scale matrix multiplication computations (GEMM) are processed in parallel by multiple matrix operation cores. Taking the operation (1×4096)·(4096×4096) as an example, the K dimensions are evenly divided among 64 computation cores, and each computation core needs to process (1×4096)·(4096×64) matrix operations. However, the matrix size is still large, and the cache resources required to store the matrix are considerable. Of course, a multi-level cache design can be adopted, continuously reading input data and weight data from DRAM and writing them to the cache during the computation process, which can appropriately reduce the cache capacity. However, this brings a new problem: whether the input data and weight data share DRAM bandwidth (weights and input data are placed in the same DRAM) or share bus transmission bandwidth (weights and input data are placed in different DRAMs), a decrease in transmission efficiency is inevitable.

[0055] If multiple computational cores can share the input data matrix, a single core only needs to process (W×C / N) of the input matrix at a time, and a small amount of cache resources can meet the storage requirements. N computational cores are connected sequentially in a ring, such as... Figure 1 As shown, each computational core completes the matrix operation and passes the input matrix data of size (W×C / N) to the next computational core to continue the computation. The input data matrix is ​​passed between computational cores until N-1 rounds of computation are completed, at which point the (W×C)·(C×K) matrix operation ends.

[0056] Specifically, if the computational cores can share the input data matrix cache, according to the C-dimensional partitioning method, a single computational core only needs to process (1×64)·(64×64) matrix operations per operation, and a small amount of cache resources can meet the storage requirements of the input data. In addition, during the computation process, only the weight data needs to be updated, and the weight data can exclusively enjoy the DRAM and transmission bus bandwidth, which greatly improves the computational efficiency.

[0057] The input matrix size is W·C, where W represents the length of the user input samples (Batch Size refers to the number of data samples passed through the model in one iteration). A smaller value like W does not require splitting. In different large model stages, the C dimension can be either the hidden layer dimension or the intermediate layer dimension. Because the values ​​of hidden_size (Dmodel) or intermediate_size (Dff) are relatively large, the Dmodel / Dff values ​​vary between different large models, possibly taking values ​​such as 4096 / 5120 / 14336 / 16384, etc. The weight matrix dimension (Dmodel × FF) is relatively large, requiring a larger cache capacity for one-time calculation, therefore C / K dimension splitting is necessary.

[0058] In one specific embodiment, the matrix dimensions of different sub-layers calculated by the LLM large model in different stages (decode / prefill) are different. Taking the LLM-3-8B model as an example, when the MLP layer of the decode or prefill stage calculates the gate / up FFN, the C dimension represents Dmodel, and the K dimension represents Dff; when the down FFN is calculated, the C dimension represents Dff, and the K dimension represents Dmodel. When the Attention layer of the decode stage calculates the self-Attention matrix multiplication, the C dimension represents Dhead or L, and the K dimension represents L or Dhead (L: context length); when the Attention layer of the prefill stage calculates the self-Attention matrix multiplication, the C dimension represents Dhead or S, and the K dimension represents S or Dhead (S: input sentence length); therefore, the values of W / C / K dimensions corresponding to different matrix multiplication operations are different, and the chip needs to process different specifications of matrix multiplication. For the specific embodiments mentioned in the present application, for the LLM3-8B model, the C / K dimensions of the weight (4096x4096) matrix represent Dmodel.

[0059] Since the currently commonly used LLM large model is a multi-head mechanism, Dhead represents the vector dimension size used when each independent attention head is calculated. Dmodel=num_heads*Dhead, that is, Dmodel (total hidden layer dimension) is evenly distributed to num_heads attention heads, and the dimension of each attention head is Dhead.

[0060] In the specific implementation process of the present application, all the computing cores are collectively formed into a ring structure, or the computing cores are divided into multiple groups, and each group forms a ring structure; if a certain computing core fails, it will affect the ring structure where the computing core is located, causing the final output result of the ring to be abnormal. At this time, the problem ring can be shielded, and other rings can be used for calculation.

[0061] In the present application, the ring interconnection structure is applied, the data transmission efficiency is the highest, and the cache optimization effect is the best. Improving the robustness can reduce the number of computing cores of the ring interconnection, so that a certain core problem will only affect the ring structure where it is located, and will not cause the entire system function to be abnormal. However, fewer computing cores forming a ring has a limited effect on reducing cache usage, and the size of each ring and the number of ring structures arranged should be selected according to the actual situation.

[0062] In one specific implementation, 64 computing cores are connected in turn in pairs to form a ring in the order of arrangement, each computing core respectively completes 64 times of (1 x 64) · (64 x 64) size matrix multiplication and addition operation, and transmits the (1 x 64) input matrix data to the next computing core to continue multiplication and addition calculation with the (64 x 64) weight matrix. A single computing core completes 64 times of calculation because the weight matrix (4096 x 64) is divided into 64 (64 x 64) in the C dimension. That is, in the first round of calculation, matrix_core_0 performs (1 x 64) m_core_0 · (64 x 64) m_core_0 multiplication and addition calculation, matrix_core_1 performs (1 x 64) m_core_1 · (64 x 64) m_core_1 ……, matrix_core_63 performs (1 x 64) m_core_63 · (64 x 64) m_core_63 multiplication and addition calculation, the subscript of the matrix represents where the data starts to be stored in the matrix_core; in the second round of calculation, matrix_core_0 transmits (1 x 64) m_core_0 to matrix_core_1, matrix_core_0 receives (1 x 64) m_core_63 from matrix_core_63, calculates (1 x 64) m_core_63 · (64 x 64) m_core_0 ; matrix_core_1 continues to calculate (1 x 64) m_core_0 · (64 x 64) m_core_1 , and each matrix_operation_core calculates (1 x 64) · (64 x 64), …, and so on. The single block 1 / 64 size input matrix is transmitted down among the matrix_operation_cores. The transmission process of the input data matrix among the matrix_operation_cores needs to complete 63 rounds of transmission (transmission rounds = matrix_operation_core number - 1, 64 matrix_operation_cores complete 63 rounds of transmission), and a total of 64 rounds of calculation (matrix_operation_core number is the transmission round + 1), and the operation ends.

[0063] In addition, 64 computing cores, each of which needs to refresh the weight matrix during the calculation process, and read out new weight values each time. Because the input data matrix is stored in the cache of each computing core before the calculation starts, only the weight matrix needs to be read from the external DRAM during the calculation process, which can fully utilize the DRAM storage chip bandwidth, and the weight data reading can be performed simultaneously with the matrix calculation process, which is easy to realize the matrix multiplication operation pipeline design, and greatly shortens the calculation time.

[0064] As shown in Figure 6 When the method of the application is implemented, first, the instruction set is loaded to the computing core, then the DRAM loads the initial IA (input matrix) data to the cache in the computing core, and after entering the calculation stage, the cache only needs to request the weight data from the DRAM. While the cache receives the weight data, the IA data is exchanged between the computing cores. The several stages of calculation are arranged in time sequence.

[0065] For the selection of the number of computing cores, in the specific implementation process of the application, the number of heterogeneous cores can be selected according to the actual chip area / performance. Considering the chip performance, it is suggested to build a minimum ring with 16 heterogeneous cores to share data, as shown in Figure 2 At the same time, it is also necessary to ensure that the transmission distance between every two computing cores is the shortest. On this basis, it can be extended to n rings, n is a positive integer power of 2 (for example, 1 / 2 / 4 / 8), and n is an integer greater than or equal to 0. Multiple rings (each with 16 heterogeneous cores) can be calculated in parallel, or can be interconnected to form a large ring. As shown in Figure 4 For the interconnection relationship and position of the 32 computing cores, the connection between tc0 and tc1 of ring0 is disconnected, tc0 is connected with tc1 of another ring ring1, and tc0 of ring1 is connected with tc1 of ring0, so that two rings ring0 and ring1 form a large ring composed of 32 heterogeneous cores. Similarly, it can be extended to the ring of 64 heterogeneous cores mentioned in the application. The connection mode between the heterogeneous cores can be configured. Small specification matrix operation can only use a certain ring (16 heterogeneous cores) to calculate, for example, the weight weight matrix (5120x5120), which can be divided into (4096x4096), (4096x1024), (1024x4096), (1024x1024) blocks, and multiple or one ring operation can be flexibly scheduled, which can effectively reduce the chip power consumption.

[0066] Taking the calculation of a (16×4096)·(4096×4096) matrix multiplication as an example, with data format fp16, without reducing computational performance, if the chip uses 64 computational cores, each core processing a (16×4096)·(4096×64) matrix multiplication would require 640Kbytes of on-chip RAM storage. Even if the weight matrix is ​​read from DRAM and written to the RAM cache, a single computational core would still need to store a (16×4096)·(64×64) matrix, requiring 136Kbytes of RAM storage. With this invention, the input matrix is ​​only (16×64) in size (1 / 64 of the original matrix), and the same design of reading the weight matrix from DRAM and writing it to the RAM cache is used. Each computational core calculates (16×64)·(64×64) at a time, requiring only 10Kbytes of on-chip RAM cache resources, reducing cache resource usage by approximately 90%, and significantly reducing chip area and cost.

[0067] Compared to the L0 / L1 / L2 cache design used in Ascend, the method of this invention has the advantage of reducing the number of times the input matrix data (IA) accesses DRAM. The input matrix data only needs to be read into the cache of each computation core once before the computation begins; during the computation, the computation core does not need to access DRAM to obtain the input matrix data. The process of large model matrix multiplication and addition operations (GEMM) in this invention is as follows: Figure 6 Each computing core has a fixed transmission bandwidth. If the chip uses N computing cores, each core processes (16×4096)·(4096×64) matrix multiplications. Each computing core requests 1A (128Kbyte) and 512KByte of data from DRAM each time. The "inter-core input sharing + ring transmission" mechanism used in this invention does not require requesting 1A data from DRAM during computation; therefore, it only needs to request 512KByte of weight from DRAM. With a fixed total on-chip transmission bandwidth, a 20% reduction in data volume means a 20% reduction in transmission time.

[0068] The related chip designed by the method of the application supports deploying the LLM3 large model. The commonly used weight matrix in the LLM3 large model has a dimension of (4096x4096), and the data format is an fp16 floating point number. To implement the LLM3 large model, the chip must have the ability to process a (4096x4096) matrix or even a larger matrix such as (4096x14336) or (5120x5120). Assuming that the input activation matrix has a dimension of (16x4096), the chip computing unit needs to process A·B=(W×C)·(C×K)=(W×K), that is, to process the calculation of (16x4096)·(4096x4096)=(16x4096). On-chip deployment of 64 computing cores is required, and each computing core needs to process a matrix calculation of (16x4096)·(4096x64) specifications. The input activation matrix cache and the weight matrix cache are designed inside the computing core. Data sharing between computing cores is adopted, and each computing core only needs to cache an input activation matrix of 1 / 64 size. A single computing core processes (16x64)·(64x64) at a time, and the data required for one-time calculation needs to be stored in the cache, that is, the input activation matrix 16x64x2Byte=2KByte, the weight matrix 64x64x2Byte=4KByte, and a total of 6KByte.

[0069] The cache size mentioned in the above specific examples is to adapt to the matrix dimension mentioned in the specific embodiment of the application, which is typical in the LLM3 model and can meet most matrix operations in 32 layers. The LLM3 Dmodel is 4096. In addition, considering the chip-level constraints, the minimum configuration recommended for the application is as follows.

[0070] The core idea of the application is to realize cache data sharing between heterogeneous cores, reduce data transmission time between internal cache and external DRAM, fully release the chip bus transmission bandwidth, and improve the computing efficiency of the chip. The cache configuration size is determined according to the parameters of different models, the performance and area of the chip planning. Large models such as LLM and DeepSeek are constantly developing, and the parameters will also be adjusted. Different specifications of chips have different cache areas and computing performance, and it is impossible to give a unified formula or recommended setting. The ring composed of heterogeneous cores is recommended to be in units of 16 cores, and is extended to 2^N (N is a positive integer greater than or equal to 0) rings. In actual use, an odd number of rings is generally not used, because when an odd number of rings is used, one ring containing 16 computing cores can only work as an independent ring, and cannot be connected to other rings to form a larger ring structure, which makes the entire computing system not flexible enough in use and affects the performance.

[0071] Before the calculation starts, the calculation core first requests input activation matrix data from the DDR, and the weight matrix data is much larger than the activation matrix, and the weight data can be written into the weight buffer of the calculation and the internal weight buffer during the matrix multiplication calculation. In this way, the weight data transmission time can be synchronized with the calculation, the full pipelining of the calculation can be realized, and the large-scale matrix calculation time is shortened.

[0072] The application can reduce the use of on-chip RAM as much as possible under the premise of not reducing performance and fully utilizing data transmission bandwidth. If the calculation performance is not to be reduced, taking the LLM3 large model mentioned above as an example, each calculation core needs to cache the complete input activation matrix (16*4096), that is, 16*4096*2Byte = 128Kbyte, which is more than 20 times the storage resource usage of the application.

[0073] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes. The solutions in the embodiments of the present application can be implemented in various computer languages, such as object-oriented programming languages Java and interpreted scripting language JavaScript.

[0074] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0075] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing devices to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0076] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable devices provide the function for implementing the processes specified in the flowcharts Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0077] Although the preferred embodiments of the application have been described, those skilled in the art will be able to make additional modifications and variations to these embodiments without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims be construed to cover all such modifications and variations as fall within the scope of the application.

[0078] Obviously, various modifications and changes can be made to the present application by those skilled in the art without departing from the spirit and scope of the present application. Thus, it is intended that the present application encompass such modifications and changes as fall within the scope of the claims and their equivalents.

[0079] The scope of protection of the present application is not limited to the above-mentioned embodiments. Changes and modifications within the spirit and scope of the present application may be conceived by those skilled in the art, and such changes and modifications are also intended to fall within the scope of the present application as defined by the following claims.

Claims

1. A method for input data sharing and cache optimization in matrix multiplication computation, characterized in that, The method comprises: Step one, dividing and allocating the input data matrix and the weight matrix according to the number N of computing cores on the chip; Step two, connecting the plurality of computing cores in sequence to form a data transmission ring structure; Step three, each computing core calculates the matrix multiplication allocated thereto and transmits the input sub-matrix of the computing core to the next computing core; Step four, the transmitted input sub-matrix is multiplied with the weight sub-matrix in the next computing core; Step five, iteratively transmitting and calculating the input sub-matrix, and performing N rounds of matrix multiplication of the input sub-matrix and the weight sub-matrix to complete the entire calculation process.

2. The method of claim 1, wherein, In step one, the dimension of the input data matrix is WxC, and the dimension of the weight matrix is CxK; The input data matrix and the weight matrix are allocated to N computing cores; the input data matrix is only divided in the dimension C, and the dimension of the divided input sub-matrix is WxC / N; the weight matrix is divided in the dimensions C and K, and the dimension of the divided weight sub-matrix is C / NxK / N; Wherein, the dimension W represents the number of data samples transmitted through the model in one iteration or one training step; the dimensions C and K represent the dimensions of different layers of the model.

3. The method of claim 1, wherein, In step two, all the computing cores together form a ring structure, or the computing cores are divided into multiple groups, and each group forms a ring structure; And / or, When multiple ring structures are formed, if any computing core in less than the total number of ring structures fails, the ring structure containing the failed computing core is shielded, and the calculation task is performed by the ring structure without failure.

4. The method of claim 1, wherein, In step three, take one computing core in the ring structure as the starting point of counting, denoted as m_core_0, and the last computing core in the data transmission direction is denoted as m_core_i; wherein, i=N-1; Each computing core calculates the matrix multiplication initially allocated thereto and transmits the input sub-matrix of the computing core to the next computing core.

5. The method of claim 1, wherein, In step five, when one round of input sub-matrix transmission is completed, during the calculation process, each computing core continues to transmit the input sub-matrix used in the current calculation to the next computing core for re-calculation, and so on, until the input sub-matrix is transmitted to the last computing core closest to the starting position thereof, and the entire calculation process is completed.

6. The method of claim 1, wherein, The weight sub-matrix of each computing core does not participate in the transmission between the computing cores in the ring structure, and is written into the weight buffer of each computing core by the external DRAM before each round of calculation starts; And / or, For different calculation rounds, the same weight sub-matrix is used for the same computing core; And / or, During the calculation of the current round of matrix multiplication, the output sub-matrix required for the next round is transmitted, and the weight sub-matrix required for the next round is preloaded and stored in the corresponding buffer; And / or, The inter-core distance between different computing cores is adjusted to make the transmission distance between each two computing cores the shortest.

7. A system for enabling input data sharing and cache optimization in matrix multiplication computations, characterized by, The computing core comprises: An input buffer module for storing the input sub-matrix locally; a weight cache module, configured to store a weight sub-matrix corresponding to the input sub-matrix; a calculation module, configured to perform a multiply-add operation on the input sub-matrix and the weight sub-matrix; a data transmission module, configured to transmit the input sub-matrix in the current calculation core to the next calculation core and receive the input sub-matrix from the previous calculation core; wherein the plurality of calculation cores are constructed in a ring structure through on-chip interconnection, supporting sequential delivery and parallel calculation of the input sub-matrix; and / or, the data transmission module is configured to support N-1 times of data rotation transmission, ensuring that all calculation cores can obtain all input sub-matrices to complete the complete matrix multiplication operation.

8. Use of the method according to any one of claims 1-6 or the system according to claim 7 in the design of a chip computing architecture supporting large language model inference or training.

9. A hardware system for implementing the method according to any one of claims 1-6, characterized in that, The hardware system comprises a memory and a processor, and the memory stores a computer program, which, when executed by the processor, implements the method according to any one of claims 1-6.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the method according to any one of claims 1-6.

Images