Method, system and application of input data sharing and cache optimization in matrix multiplication calculation

Abstract

The application discloses a kind of input data sharing and cache optimization method in matrix multiplication calculation, the method includes: step one, input data matrix and weight matrix are divided and distributed according to the number N of computing core on chip;Step two, a plurality of the computing core is sequentially connected to form data transmission ring structure;Step three, each computing core is calculated to the matrix multiplication distributed to it, and the input submatrix of current computing core is passed to the next computing core;Step four, the input submatrix of transmission is operated with the weight submatrix in the next computing core;Step five, the transmission and calculation of the above input submatrix are iterated, and the matrix multiplication operation of input submatrix and weight submatrix is carried out for N rounds, to complete the whole operation process.The application also discloses a system for implementing the above method, which has wide application value.

Description

Technical Field

[0001] This invention belongs to the field of chip design technology and relates to a method, system and application for input data sharing and caching optimization in matrix multiplication calculation. Background Technology

[0002] The application of large-scale models based on the Transformer architecture is booming, and the demand for fast computation of large-scale matrices is becoming increasingly strong. Compared with traditional CPUs, ASIC chips can deploy more computing units for parallel computation, resulting in a significant advantage in computing power. The high concurrency of the hardware supports the deployment of multiple computing cores, allowing a large-scale matrix to be divided into multiple blocks for simultaneous computation. Taking Huawei's Da Vinci architecture as an example, it employs multi-level caching (L0 / L1 / L2) to reduce external storage access and supports data reuse (such as data flow patterns with fixed weights and fixed outputs). The L0 cache is closely arranged next to the Cube core (responsible for matrix operations) and stores input data and weights. The L1 / L2 caches serve as shared caches, storing intermediate results and frequently accessed data. Each input data matrix needs to be multiplied with a fixed weight matrix. When the weights are fixed, the L0 cache stores the input data that the computing units need to read frequently.

[0003] Storage systems significantly impact the performance, power consumption, and area of ​​large-scale AI model chips, making them a crucial consideration in AI chip architecture design. AI chips often suffer from performance limitations due to their storage systems. Conventional chip architectures store both input data matrices and weight matrices in external storage chips such as DRAM. Each computational core needs to read and write data from DRAM, requiring multiple levels of cache to buffer the data. Therefore, the goal is to address the bottleneck of matrix multiplication calculations in current AI chips by minimizing on-chip storage resource usage and reducing the frequency of data interaction between caches and external storage chips without compromising computational performance. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a method, system, and application for input data sharing and cache optimization in matrix multiplication calculations. Through hardware design and planning, matrix data sharing between computing cores is achieved, and data is evenly distributed across multiple computing cores, effectively reducing on-chip RAM / Cache storage resources. Data transmission between computing cores also reduces data interaction between cache and DRAM, supporting AI inference and / or training.

[0005] This invention provides a method for input data sharing and caching optimization in matrix multiplication calculations, wherein the matrix multiplication calculations are performed on the computing core of a chip; including:

[0006] Step 1: Divide and allocate the input data matrix and weight matrix according to the number of computing cores N on the chip;

[0007] Step 2: Connect the multiple computing cores end to end in sequence to form a data transmission ring structure;

[0008] Step 3: Each computational core performs the assigned matrix multiplication and passes the current input submatrix of the computational core to the next computational core;

[0009] Step 4: Perform matrix multiplication between the input submatrix and the weight submatrix in the next computation kernel;

[0010] Step 5: Iterate through the passing and calculation of the input submatrix, performing N rounds of matrix multiplication of the input submatrix and weight submatrix to complete the entire calculation process.

[0011] In step one, the dimension of the input data matrix is ​​W×C, and the dimension of the weight matrix is ​​C×K;

[0012] The input data matrix and the weight matrix are assigned to N computational kernels; the input data matrix is ​​partitioned only along dimension C, and the dimension of the partitioned input submatrix is ​​W×C / N; the weight matrix is ​​partitioned along dimensions C and K, and the dimension of the partitioned weight submatrix is ​​C / N×K / N.

[0013] Where W represents the number of data samples passed through the model in one iteration (or one training step). In the inference phase, such as in the autoregressive calculation in the decoding phase, the value of W is usually 1, corresponding to processing only one token at a time. W can be flexibly adjusted according to the application scenario and is usually much smaller than the model's dimensions C and K. Therefore, in this invention, the W dimension is usually not split and calculated, but the C and K dimensions are mainly divided and processed in parallel.

[0014] In one specific embodiment of the present invention, a multi-user mode is also supported, but the number of users accessing the device at the same time on a single chip does not exceed 64.

[0015] In step two, all the computing cores together form a ring structure, or the computing cores are divided into multiple groups, each group forming a ring structure; for each ring structure formed by computing cores, the number of computing cores contained therein is ≥3;

[0016] When multiple ring structures are formed, if a computing core in any one or more ring structures less than the total number of ring structures fails, the ring structure containing the failed computing core is shielded, and the computing task is handed over to the ring structure that has not failed.

[0017] In step three, an arbitrary computational kernel in the ring structure is taken as the starting point for counting, denoted as m_core_0, and the last computational kernel along the data transmission direction is denoted as m_core_i; where i = N-1;

[0018] The matrix multiplication operation corresponding to each computational core in the first round is (W×C / N). m_core_0 ·(C / N×K / N) m_core_0 (W×C / N) m_core_1 ·(C / N×K / N) m_core_1 ... (W×C / N) m_core_i ·(C / N×K / N) m_core_i Then, the input submatrix of the current computation kernel is passed, specifically the input submatrix (W×C / N) of the first computation kernel m_core_0. m_core_0 The input submatrix (W×C / N) of the second computational kernel m_core_1 is passed to the second computational kernel m_core_1. m_core_1 The input submatrix (W×C / N) is passed to the third computational kernel m_core_2, and so on, until the last computational kernel m_core_i. m_core_i Passed to the first computation core m_core_0;

[0019] In one specific implementation, in the first round of computation, each computational core calculates the initially assigned matrix multiplication operation, represented as (1×64). m_core_0 (64×64) m_core_0 (1×64) m_core_1 (64×64) m_core_1 ... (1×64) m_core_i (64×64) m_core_i The current input submatrix of the computation kernel is passed to the next computation kernel, which is the input submatrix (1×64) of the first computation kernel m_core_0. m_core_0 The input submatrix (1×64) of the second computational kernel m_core_1 is passed to it. m_core_1 The input submatrix (1×64) is passed to the third computational kernel m_core_2, and so on, until the last computational kernel m_core_i. m_core_i Passed to the first computation core m_core_0.

[0020] In one specific implementation, matrix multiplication operations performed in each computational core can be considered to be performed in parallel with the backpropagation of the input submatrix.

[0021] In step four, the input submatrix passed from the previous computational kernel is multiplied by the weight submatrix of the current computational kernel, and the result is represented as (W×C / N). m_core_i ·(C / N×K / N) m_core_0 (W×C / N) m_core_0 ·(C / N×K / N)m_core_1 ... (W×C / N) m_core_(i-1) ·(C / N×K / N) m_core_i ;

[0022] In one specific implementation, they are represented as (1×64). m_core_i (64×64) m_core_0 (1×64) m_core_0 (64×64) m_core_1 ... (1×64) m_core_(i-1) (64×64) m_core_i .

[0023] In step five, once the transfer of one round of input submatrix is ​​complete, during the calculation process, each computational core can continue to transfer the input submatrix currently used for calculation to the next computational core, i.e., the input submatrix (W×C / N). m_core_0 The input submatrix (W×C / N) is passed to the third computational kernel m_core_2 and the second computational kernel m_core_1. m_core_1 The input submatrix (W×C / N) is passed to the fourth computational kernel m_core_3, and so on, until the last computational kernel m_core_i. m_core_i Passed to the second computational core m_core_1;

[0024] In one specific implementation, the input submatrix is ​​(1×64). m_core_0 The input submatrix (1×64) is passed to the third computational kernel m_core_2 and the second computational kernel m_core_1. m_core_1 This is passed to the fourth computational core m_core_3, and so on, until the last computational core m_core_i receives its input submatrix (1×64). m_core_i Passed to the second computational core m_core_1;

[0025] Similarly, matrix multiplication is performed between the input submatrix passed from the previous computational kernel and the weight submatrix of the current computational kernel itself;

[0026] This process continues until the input submatrix (W×C / N) is complete. m_core_0 The input submatrix (W×C / N) is passed to the last computation kernel m_core_i and the second computation kernel m_core_1. m_core_1 The input submatrix (W×C / N) is passed to the first computation kernel m_core_0, and so on, up to the last computation kernel m_core_i. m_core_i The input submatrix and weight matrix are passed to the penultimate computation kernel m_core_(i-1), where matrix multiplication is performed on each computation kernel.

[0027] In one specific implementation, the input submatrix is ​​(1×64). m_core_0 The input submatrix (1×64) is passed to the last computational kernel m_core_i and the second computational kernel m_core_1. m_core_1 The input submatrix (1×64) is passed to the first computation kernel m_core_0, and so on, up to the last computation kernel m_core_i. m_core_i The input submatrix and weight matrix are passed to the penultimate computation kernel m_core_(i-1), where matrix multiplication is performed on each computation kernel.

[0028] The entire calculation process is complete.

[0029] During steps three through five above, the weight submatrices of each computational core do not participate in the transfer between computational cores in the ring structure. Instead, they are written into the weight cache of each computational core before the start of each round of computation via external DRAM.

[0030] In a specific implementation, the same weight submatrix is ​​used for the same computation kernel across different computation rounds.

[0031] In a specific implementation, during the matrix multiplication calculation in this round, the output submatrix required for the next round is passed and the weight submatrix required for the next round is preloaded and stored in the corresponding cache.

[0032] In a specific implementation process, the inter-core distance between different computing cores will be adjusted to minimize the transmission distance between any two computing cores.

[0033] This invention also provides a system for optimizing input data sharing and caching in the above-mentioned matrix multiplication calculation, comprising one or more computing cores. The computing cores are mounted on a chip and are dedicated matrix operation units designed for large-model matrix multiplication optimization. Multiple computing cores are electrically connected through on-chip interconnects. Each computing core includes:

[0034] An input cache module is used to store the input submatrix local to the computing core;

[0035] The weight caching module is used to store the weight submatrix corresponding to the input submatrix;

[0036] The calculation module is used to perform multiplication and addition operations on the input submatrix and the weight submatrix;

[0037] The data transmission module is used to transfer the input submatrix in the current computing core to the next computing core and to receive the input submatrix from the previous computing core.

[0038] The multiple computing cores are interconnected on-chip to form a ring structure, supporting the sequential transfer and parallel computation of the input submatrix.

[0039] The data transmission module is configured to support N-1 data round-robin transmissions, ensuring that all computing cores can obtain all input submatrices to complete the full matrix multiplication operation.

[0040] The present invention also provides the application of the above-described method or system in chip computing architecture design that supports large language model inference or training.

[0041] The present invention also provides a hardware system for implementing the above method, the hardware system comprising: a memory and a processor; the memory storing a computer program, which, when executed by the processor, implements the above method.

[0042] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0043] The beneficial effects of this invention include: By constructing an input data sharing mechanism among heterogeneous computing cores and employing a ring structure for efficient transmission of the input matrix, the input activation data is loaded into the local cache once before computation begins and can then be reused among multiple computing cores, thereby significantly reducing the frequency of input data access to DRAM. Furthermore, by combining this with a strategy of locally loading weight submatrices by computing core, each computing core does not need to repeatedly load input matrix data during the entire matrix multiplication and addition operation, but only needs to read weight data, thus significantly reducing DRAM bandwidth pressure. Compared with traditional solutions, this invention can reduce on-chip cache usage by approximately 90%, reduce data transmission bandwidth pressure by 20%, and support fully pipelined scheduling for the overall computation process. Without compromising computational performance, it effectively reduces chip area and power consumption, and improves the overall efficiency and resource utilization of large model matrix computation. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram of data transmission in a ring structure composed of 16 computing cores in a specific embodiment of the present invention.

[0046] Figure 2This is a schematic diagram showing the location of the internal computing core of the chip in a specific embodiment of the present invention.

[0047] Figure 3 This is a schematic diagram of the matrix multiplication calculation process performed by eight computing cores in a specific embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram showing the interconnection relationship and location of 32 computing cores in a specific embodiment of the present invention.

[0049] Figure 5 This is a schematic diagram of the result matrix (OA) obtained by multiplying the input matrix (IA) and the weight matrix (Weight) in a specific embodiment of the present invention.

[0050] Figure 6 This is a schematic diagram of the calculation process in a specific embodiment of the present invention. Detailed Implementation

[0051] The present invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0052] Implementing large models based on the Transformer architecture requires handling large-scale matrix multiplication operations (GEMM), where the input data matrix is ​​multiplied by the weight matrix. To improve computing power and efficiency, the chip is designed for multi-core parallel processing. The matrix (W×C)·(C×K) is evenly divided among N matrix computation cores. Each core needs to handle (W×C)·(C×K / N) matrix operations, which still results in a large matrix size and requires significant cache resources. Therefore, it is advisable to further divide the dimension C.

[0053] Matrix multiplication operations are as follows Figure 3 As shown in the figure, the dimensions of the matrix are also represented.

[0054] Specifically, large-scale matrix multiplication computations (GEMM) are processed in parallel by multiple matrix operation cores. Taking the operation (1×4096)·(4096×4096) as an example, the K dimensions are evenly divided among 64 computation cores, and each computation core needs to process (1×4096)·(4096×64) matrix operations. However, the matrix size is still large, and the cache resources required to store the matrix are considerable. Of course, a multi-level cache design can be adopted, continuously reading input data and weight data from DRAM and writing them to the cache during the computation process, which can appropriately reduce the cache capacity. However, this brings a new problem: whether the input data and weight data share DRAM bandwidth (weights and input data are placed in the same DRAM) or share bus transmission bandwidth (weights and input data are placed in different DRAMs), a decrease in transmission efficiency is inevitable.

[0055] If multiple computational cores can share the input data matrix, a single core only needs to process (W×C / N) of the input matrix at a time, and a small amount of cache resources can meet the storage requirements. N computational cores are connected sequentially in a ring, such as... Figure 1 As shown, each computational core completes the matrix operation and passes the input matrix data of size (W×C / N) to the next computational core to continue the computation. The input data matrix is ​​passed between computational cores until N-1 rounds of computation are completed, at which point the (W×C)·(C×K) matrix operation ends.

[0056] Specifically, if the computational cores can share the input data matrix cache, according to the C-dimensional partitioning method, a single computational core only needs to process (1×64)·(64×64) matrix operations per operation, and a small amount of cache resources can meet the storage requirements of the input data. In addition, during the computation process, only the weight data needs to be updated, and the weight data can exclusively enjoy the DRAM and transmission bus bandwidth, which greatly improves the computational efficiency.

[0057] The input matrix size is W·C, where W represents the length of the user input samples (Batch Size refers to the number of data samples passed through the model in one iteration). A smaller value like W does not require splitting. In different large model stages, the C dimension can be either the hidden layer dimension or the intermediate layer dimension. Because the values ​​of hidden_size (Dmodel) or intermediate_size (Dff) are relatively large, the Dmodel / Dff values ​​vary between different large models, possibly taking values ​​such as 4096 / 5120 / 14336 / 16384, etc. The weight matrix dimension (Dmodel × FF) is relatively large, requiring a larger cache capacity for one-time calculation, therefore C / K dimension splitting is necessary.

[0058] In one specific implementation, the matrix dimensions computed by different sub-layers in different stages (decode / prefill) of the LLM large model differ, taking the LLM-3-8B model as an example. When the MLP layer computes the gate / up FFN in the Decode or prefill stage, the C dimension represents Dmodel, and the K dimension represents Dff; when computing the down FFN, the C dimension represents Dff, and the K dimension represents Dmodel. In the Decode stage, when the Attention layer computes the self-Attention matrix multiplication, the C dimension represents Dhead or L, and the K dimension represents L or Dhead (L: context length); in the Prefill stage, when the Attention layer computes the self-Attention matrix multiplication, the C dimension represents Dhead or S, and the K dimension represents S or Dhead (S: input sentence length). Therefore, the W / C / K dimension values ​​corresponding to different matrix multiplication operations are different, and the chip needs to handle matrix multiplications of different specifications. For the specific embodiment mentioned in this invention, for the LLM3-8B model, the C / K dimensions of the weight (4096×4096) matrix both represent Dmodel.

[0059] Since the commonly used LLM large models are multi-head mechanisms, Dhead represents the vector dimension size used by each independent attention head for computation. Dmodel = num_heads * Dhead, meaning that Dmodel (total hidden layer dimension) is evenly distributed among num_heads attention heads, and the dimension of each attention head is Dhead.

[0060] In the specific implementation of this invention, all the computing cores together form a ring structure, or the computing cores are divided into multiple groups, each group forming a ring structure. If a computing core malfunctions, it will affect the ring structure in which that computing core is located, causing the final output of the ring to be abnormal. In this case, the problematic ring can be shielded, and other rings can be used for calculation.

[0061] This invention employs a ring interconnect structure, which offers the highest data transmission efficiency and the best cache optimization effect. Improving robustness can reduce the number of computational cores in the ring interconnect, so that if one core fails, only the ring it belongs to will be affected, preventing the entire system from malfunctioning. However, using fewer computational cores in a ring has limited effect on reducing cache usage; therefore, the size of each ring and the number of ring structures must be selected based on the specific circumstances.

[0062] In one implementation, 64 computational cores are connected in pairs to form a ring. Each core performs 64 matrix multiplication-addition operations of size (1×64)·(64×64), and then passes the (1×64) input matrix data to the next core for further multiplication-addition with the (64×64) weight matrix. A single core performs 64 calculations because the weight matrix (4096×64) is divided into 64 (64×64) parts along dimension C. That is, in the first round of computation, matrix_core_0 performs (1×64) operations. m_core_0 (64×64) m_core_0 Multiply-accumulate operations are performed in matrix_core_1 (1×64). m_core_1 (64×64) m_core_1 ..., matrix_core_63 performs (1×64) m_core_63 The first round of calculation involves multiplying and adding (64×64)m_core_63, where the matrix index indicates which matrix_core the data is initially stored in; the second round of calculation involves matrix_core_0 storing (1×64)... m_core_0 The data is passed to matrix_core_1, and matrix_core_0 receives (1×64) data from matrix_core_63. m_core_63 Calculate (1×64) m_core_63 (64x64) m_core_0 matrix_core_1 continues to calculate (1×64). m_core_0 (64×64) m_core_1 Similarly, each matrix_operation_core calculates (1×64)·(64×64), ..., and so on, with a single 1 / 64-sized input matrix being passed between matrix_operation_cores. The process of passing the input data matrix between matrix_operation_cores requires 63 rounds of transmission (transmission rounds = number of matrix_operation_cores - 1; 64 matrix_operation_cores complete 63 rounds of transmission), for a total of 64 rounds of calculation (number of matrix_operation_cores = number of transmission rounds + 1), after which the operation ends.

[0063] In addition, there are 64 computation cores. The weight matrix needs to be continuously refreshed during the computation process, and new weight values ​​need to be read for each computation. Since the input data matrix is ​​stored in the cache of each computation core before the computation begins, the weight matrix only needs to be read from the external DRAM during the computation process, which can make full use of the DRAM memory chip bandwidth. In addition, reading the weight data can be performed simultaneously with the matrix computation process, making it easy to implement a pipelined design for matrix multiplication operations and greatly shortening the computation time.

[0064] like Figure 6 As shown, in implementing the method of this invention, the instruction set is first loaded into the computing core, and then the DRAM loads the initial IA (input matrix) data into the cache within the computing core. After entering the computing stage, the cache only needs to request weight data from the DRAM. While the cache receives the weight data, the computing cores exchange IA data. The several stages of the computing are arranged in chronological order.

[0065] Regarding the selection of the number of computing cores, in the specific implementation of this invention, the number of heterogeneous cores can be selected based on the actual chip area / performance. Considering chip performance, it is recommended to use 16 heterogeneous cores as the minimum required to build a ring and share data, such as... Figure 2 As shown, it is also necessary to ensure that the transmission distance between any two computational cores is minimized. Based on this, it can be extended to n rings, where n is a positive integer power of 2 (e.g., 1 / 2 / 4 / 8), and n is an integer greater than or equal to 0. Multiple rings (each with 16 heterogeneous cores) can be computed in parallel or interconnected to form a larger ring. For example... Figure 4 To define the interconnections and locations between the 32 compute cores, the connection between TC0 and TC1 in ring 0 is disconnected. TC0 is then connected to TC1 in another ring, ring 1, and TC0 in ring 1 is connected to TC1 in ring 0. This forms a large ring consisting of 32 heterogeneous cores, linking rings 0 and 1. This can also be extended to a ring with 64 heterogeneous cores as mentioned in this invention. The connection method between heterogeneous cores is configurable. Small-scale matrix operations can be performed using only one ring (16 heterogeneous cores). For example, a weight matrix (5120×5120) can be divided into multiple blocks (4096×4096), (4096×1024), (1024×4096), and (1024×1024), allowing for flexible scheduling of multiple or single-ring operations, effectively reducing chip power consumption.

[0066] Taking the calculation of a (16×4096)·(4096×4096) matrix multiplication as an example, with data format fp16, without reducing computational performance, if the chip uses 64 computational cores, each core processing a (16×4096)·(4096×64) matrix multiplication would require 640Kbytes of on-chip RAM storage. Even if the weight matrix is ​​read from DRAM and written to the RAM cache, a single computational core would still need to store a (16×4096)·(64×64) matrix, requiring 136Kbytes of RAM storage. With this invention, the input matrix is ​​only (16×64) in size (1 / 64 of the original matrix), and the same design of reading the weight matrix from DRAM and writing it to the RAM cache is used. Each computational core calculates (16×64)·(64×64) at a time, requiring only 10Kbytes of on-chip RAM cache resources, reducing cache resource usage by approximately 90%, and significantly reducing chip area and cost.

[0067] Compared to the L0 / L1 / L2 cache design used in Ascend, the method of this invention has the advantage of reducing the number of times the input matrix data (IA) accesses DRAM. The input matrix data only needs to be read into the cache of each computation core once before the computation begins; during the computation, the computation core does not need to access DRAM to obtain the input matrix data. The process of large model matrix multiplication and addition operations (GEMM) in this invention is as follows: Figure 6 Each computing core has a fixed transmission bandwidth. If the chip uses N computing cores, each core processes (16×4096)·(4096×64) matrix multiplications. Each computing core requests 1A (128Kbyte) and 512KByte of data from DRAM each time. The "inter-core input sharing + ring transmission" mechanism used in this invention does not require requesting 1A data from DRAM during computation; therefore, it only needs to request 512KByte of weight from DRAM. With a fixed total on-chip transmission bandwidth, a 20% reduction in data volume means a 20% reduction in transmission time.

[0068] The chip designed using the method of this invention supports the deployment of large LLM3 models. A commonly used dimension in LLM3 models is a (4096×4096) weight matrix, with data in fp16 floating-point format. For a chip to implement a large LLM3 model, it must be capable of processing matrices of (4096×4096) or even larger dimensions, such as (4096×14336) or (5120×5120). The current chip uses a (4096×4096) matrix as its unit; assuming the input activation matrix dimension is (16×4096), the chip's computational unit needs to process A·B=(W×C)·(C×K)=(W×K), which is equivalent to processing (16×4096)·(4096×4096)=(16×4096). The chip deploys 64 computational cores, each capable of processing matrix calculations of (16×4096)·(4096×64). The computational core is internally designed with input activation matrix cache and weight matrix cache. Using this invention, data sharing between computational cores means each core only needs to cache 1 / 64 of the input activation matrix. A single computational core processes (16×64)·(64×64) in a single operation, requiring the cache to hold the data needed for one computation: 16×64×2 bytes = 2KB for the input activation matrix and 64×64×2 bytes = 4KB for the weight matrix, totaling 6KB.

[0069] The cache size mentioned in the specific examples above is to accommodate the matrix dimension mentioned in the specific embodiments of this invention. This matrix is ​​typical in the LLM3 model and can satisfy most matrix operations in 32 layers. The LLM3 Dmodel is 4096. In addition, considering chip-level constraints, this is the minimum configuration recommended by this invention.

[0070] The core idea of ​​this invention is to achieve cache data sharing among heterogeneous cores, reduce data transfer time between internal cache and external DRAM, and fully release chip bus bandwidth to improve chip computing efficiency. The cache configuration size depends on the parameters of different models, chip performance planning, and area. Large models such as LLM and DeepSeek are constantly evolving, and their parameters will be adjusted accordingly. Different chip specifications also have different requirements for cache area / computational performance, making it impossible to provide a unified formula or recommended settings. It is suggested that the rings composed of heterogeneous cores be based on 16 cores, and then expanded to 2^N (N is a positive integer greater than or equal to 0) rings. In practical use, an odd number of rings is generally not used because when an odd number of rings are used, one ring containing 16 computing cores can only work as an independent ring and cannot connect with other rings to form a larger ring structure. This makes the entire computing system less flexible in use and affects performance.

[0071] Before computation begins, the computation core requests the activation matrix data from the DDR. The weight matrix data is much larger than the activation matrix, and the weight data can be written from DRAM to the computation and internal weight cache during matrix multiplication. In this way, the weight data transfer time can be synchronized with the computation, enabling fully pipelining of the computation and shortening the computation time for large-scale matrix operations.

[0072] This invention can minimize on-chip RAM usage without compromising performance and while fully utilizing data transmission bandwidth. To maintain computational performance, taking the LLM3 large model mentioned above as an example, each computational core needs to cache the complete input activation matrix (16×4096), which is 16×4096×2 bytes = 128 KB, exceeding the storage resource usage of this invention by more than 20 times.

[0073] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0074] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0075] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0077] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0078] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0079] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A method for optimizing input data sharing and caching in matrix multiplication calculations, characterized in that, The method includes: Step 1: Divide and allocate the input data matrix and weight matrix according to the number of computing cores N on the chip; In step one, the dimension of the input data matrix is ​​W×C, and the dimension of the weight matrix is ​​C×K; The input data matrix and the weight matrix are assigned to N computational kernels; the input data matrix is ​​partitioned only along dimension C, and the dimension of the partitioned input submatrix is ​​W×C / N; the weight matrix is ​​partitioned along dimensions C and K, and the dimension of the partitioned weight submatrix is ​​C / N×K / N. Wherein, dimension W represents the number of data samples passed through the model in one iteration or training step; dimension C and dimension K represent the dimensions of different layers of the model; Step 2: Connect the multiple computing cores end to end in sequence to form a data transmission ring structure; Step 3: Before the calculation begins, the input matrix data is read into the cache of each computing core all at once. During the calculation, the computing core does not need to access DRAM to obtain the input matrix data. The computing core locally loads the weight submatrix. Each computing core performs the matrix multiplication assigned to it and passes the current input submatrix of the computing core to the next computing core. Step 4: Perform matrix multiplication between the input submatrix and the weight submatrix in the next computation kernel; Step 5: Iterate through the passing and calculation of the input submatrix, performing N rounds of matrix multiplication of the input submatrix and weight submatrix to complete the entire calculation process.

2. The method as described in claim 1, characterized in that, In step two, all the computing cores together form a ring structure, or the computing cores are divided into multiple groups, each group forming a ring structure. And / or, When multiple ring structures are formed, if a computing core in any one or more ring structures less than the total number of ring structures fails, the ring structure containing the failed computing core is shielded, and the computing task is handed over to the ring structure that has not failed.

3. The method as described in claim 1, characterized in that, In step three, any one of the computational kernels in the ring structure is taken as the starting point for counting, denoted as m_core_0, and the last computational kernel along the data transmission direction is denoted as m_core_i; where i=N-1; Each computational core performs the initially assigned matrix multiplication operation and passes the current input submatrix of the computational core to the next computational core.

4. The method as described in claim 1, characterized in that, In step five, once the input submatrix has been passed through in one round, during the calculation process, each calculation core will continue to pass the input submatrix used in the current calculation to the next calculation core for recalculation, and so on, until the input submatrix is ​​passed to the last calculation core closest to its starting position for calculation, thus completing the entire operation process.

5. The method as described in claim 1, characterized in that, The weight submatrix of each computing core does not participate in the transfer between computing cores in the ring structure, but is written into the weight cache of each computing core before the start of each round of computation via external DRAM; And / or, For different computation rounds, the same weight submatrix is ​​used for the same computation kernel; And / or, During the matrix multiplication process, the output submatrix required for the next round is passed and the weight submatrix required for the next round is preloaded and stored in the corresponding cache. And / or, The inter-core distance between different computing cores is adjusted to minimize the transmission distance between any two computing cores.

6. A system for implementing the input data sharing and caching optimization method in matrix multiplication calculation as described in any one of claims 1-5, characterized in that, It includes multiple computing cores, each of which includes: An input cache module is used to store the input submatrix local to the computing core; The weight caching module is used to store the weight submatrix corresponding to the input submatrix; The calculation module is used to perform multiplication and addition operations on the input submatrix and the weight submatrix; The data transmission module is used to transfer the input submatrix in the current computing core to the next computing core and to receive the input submatrix from the previous computing core. The multiple computing cores are interconnected on-chip to form a ring structure, which supports the sequential transfer and parallel computation of the input submatrix; And / or, The data transmission module is configured to support N-1 data round-robin transmissions, ensuring that all computing cores can obtain all input submatrices to complete the full matrix multiplication operation.

7. The method as described in any one of claims 1-5, or the system as described in claim 6, in the design of a chip computing architecture that supports large language model inference or training.

8. A hardware system for implementing the method as described in any one of claims 1-5, characterized in that, The hardware system includes a memory and a processor; the memory stores a computer program that, when executed by the processor, implements the method as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images