Large Language Model Accelerator

Abstract

The present invention relates to intelligent integrated circuits, intelligent chips, and AI chips in artificial intelligence hardware platforms; deep neural networks, multi-layer neural networks, convolutional neural networks and other fields in general artificial intelligence technologies, and in particular to a large language model accelerator. The large language model operation node of the present invention includes: at least one computing unit and a first storage unit, both of which are located in the same design architecture, wherein: the computing unit is used for calculation; the first storage unit is used for storing static weight data during the large language model operation process; the second storage unit is arranged outside the design structure, and is used to store KV cache data during the large language model operation process. The present invention effectively avoids the high power consumption and high latency problems caused by the traditional external memory static weight data reading. Compared with the traditional HBM solution, the present invention has obvious advantages in performance, power consumption and cost.

Description

Technical Field

[0001] The present invention relates to intelligent integrated circuits, intelligent chips, and AI chips in artificial intelligence hardware platforms; deep neural networks, multi-layer neural networks, convolutional neural networks and other fields in general artificial intelligence technologies, and in particular to a large language model accelerator. Background Art

[0002] In recent years, large language models have made significant progress in the field of artificial intelligence and have been increasingly used in a wide range of applications, such as text generation, dialogue systems, translation, and document summarization. Among them, the reasoning process is one of the core links of large language models and plays a vital role in the operating efficiency of the model. Since large language models are usually trained based on a large amount of data and have an ultra-large-scale network structure, the model consumes a lot of computing resources and memory during the reasoning process.

[0003] Currently, large language models have been widely used in various fields, but as the scale of the model increases, the speed and efficiency of reasoning have become an urgent problem to be solved. Existing reasoning acceleration solutions mainly use high-performance GPU / TPU, etc. Compared with CPUs, these solutions can achieve a certain degree of acceleration while maintaining versatility, but there are still problems of high latency, high power consumption and low hardware utilization in the reasoning process. On the other hand, in order to achieve high-performance reasoning, the use of advanced processes greatly increases the cost of reasoning; and as the size of the model continues to grow, the overhead of interconnection and communication will also increase significantly. At present, there are many challenges in the reasoning process of large language models:

[0004] (1) The growing number of user KVcaches

[0005] The amount of KV Cache data dynamically generated during the inference process will increase with the increase of sequence length and number of users. The storage and reading and writing of this data will increase power consumption and latency.

[0006] (2) High power consumption and high latency in reading weight data

[0007] Large language models have billions to tens of billions of parameters, which causes traditional DRAM to have problems such as high power consumption and high latency during the weight reading stage.

[0008] (3) Operation speed due to huge amount of calculation

[0009] A large number of matrix multiplication operations are required in the parameter calculation process of large language models. In the decoding stage, every time a token is generated, all the parameters of the neural network need to be recalculated, which requires a huge amount of calculation. Summary of the invention

[0010] 1. Technical issues to be solved

[0011] The present invention is expected to at least partially solve one of the above technical problems.

[0012] 2. Technical Solution

[0013] A first aspect of the present invention provides a large language model operation node. The large language model operation node includes: at least one computing unit and a first storage unit, both of which are located in the same design architecture, wherein: the computing unit is used to perform calculations; the first storage unit is used to store static weight data during the large language model operation process; and the second storage unit is arranged outside the design structure and is used to store KV cache data during the large language model operation process.

[0014] The second aspect of the present invention provides a large language model layer cluster. The large language model layer cluster includes: N operation nodes, N≥1, and the operation nodes are the large language model operation nodes as above.

[0015] The third aspect of the present invention provides a large language model accelerator. The large language model accelerator includes: M layer clusters, M is the number of layers of the large language model, M≥2; each layer cluster includes: 1 or more computing nodes; the computing node includes: at least 1 computing unit and at least 1 storage unit, wherein: the computing unit is used for computing; the storage unit is used for storing data required for large language model computing; wherein the layer clusters in the M layer clusters correspond to the layer order of the large language model, and the data required for large language model computing is stored in the storage unit of the computing node of the corresponding layer cluster.

[0016] 3. Beneficial Effects

[0017] It can be seen from the above technical solution that the present invention has at least one of the following beneficial effects compared with the prior art:

[0018] (1) Reduce power consumption and latency

[0019] The present invention effectively avoids the high power consumption and high latency problems caused by traditional external memory static weight data reading. Compared with the traditional HBM solution, the present invention has obvious advantages in performance, power consumption and cost.

[0020] (2) Expandable and highly flexible computing power

[0021] In the present invention, a single computing node has a first and a second computing unit specifically for matrices and vectors, and the cascaded architecture can meet the computing power requirements of models of different sizes. The second computing unit is a programmable unit that can flexibly meet different large language model architectures and adapt to different activation functions, normalization and other requirements.

[0022] (3) Support for very long sequences

[0023] In the present invention, by caching the user KV Cache in the second storage unit, the hot data storage requirements of ultra-long sequence lengths can be met.

[0024] (4) Weak interconnection between layer clusters

[0025] The present invention proposes the concept of layer cluster, that is, a chip or a group of chips is used to form a basic unit to process a single layer, so as to divide the entire large language model into layers. Since there is no interaction of static weight data or KV cache data between adjacent layer clusters, only the transmission of token vectors, the layer clusters are weakly interconnected.

[0026] This distributed implementation method can effectively split the original large chip into multiple small chip groups to complete the same operation. This can not only effectively improve the communication and computing efficiency of the system, but also reduce the total cost of the chip. Using multiple low-cost chips to complete the same calculation of a large-cost chip can gain advantages in power consumption, price, and performance.

[0027] (5) Weak interconnection between computing nodes in the same layer cluster

[0028] In the present invention, in the same layer of cluster, there is no interaction of static weight data or KVcache data between computing nodes, so the computing nodes are weakly interconnected, which greatly improves the computing efficiency of the computing nodes and reduces the pressure on the communication bandwidth.

[0029] (6) Stackable architecture

[0030] In the present invention, the large language model accelerator architecture and the layer cluster architecture are both stackable architectures, that is, a combination of hardware architectures suitable for large language model calculations can be selected according to the depth and scale of the large language model. Furthermore, it is quite flexible and convenient to increase or decrease layer clusters and computing nodes according to the actual operation of the large language model.

[0031] (7) Economical and Efficient Interconnection

[0032] In the present invention, prefill and decode are separated to reduce the data transmission requirements between clusters, thereby reducing the dependence on high-cost interconnection technology.

[0033] (8) High concurrency design

[0034] In the present invention, batch processing and pipeline parallelism are used to simultaneously process requests from hundreds of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Schematic diagram of the structure of a large language model computing node according to an embodiment of the present invention.

[0036] Figure 2 Schematic diagram of the structure of a large language model accelerator according to an embodiment of the present invention.

[0037] Figure 3 This is a design diagram for data storage and decoder calculation of a large language model accelerator according to an embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram of a static weight matrix in a large language model accelerator according to an embodiment of the present invention being divided into “rows” and stored in N1 static storage units.

[0039] Figure 5 This is a schematic diagram of a static weight matrix in a large language model accelerator according to an embodiment of the present invention being divided into “columns” and stored in N1 static storage units. DETAILED DESCRIPTION

[0040] Based on the transformer type structural algorithm, the present invention proposes the concepts of classified storage, hierarchical setting, and decentralized computing, and designs unique computing nodes, layer clusters, and accelerators for large language models. The purpose is to achieve reasoning acceleration, that is, to reduce the consumption of time, bandwidth, and computing resources in the reasoning process without losing model performance.

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific implementation methods and with reference to the accompanying drawings.

[0042] According to a first aspect of the present invention, a large language model operation node is provided. Figure 1 FIG. 1 is a schematic diagram of the structure of a large language model operation node according to an embodiment of the present invention. Figure 1 As shown, the large language model operation node of this embodiment includes:

[0043] Two computing units and a first storage unit are located in the same chip, wherein:

[0044] A first storage unit, used to store static weight data during the large language model operation process;

[0045] The first computing unit is mainly used to process matrix computing tasks in the large language model computing node;

[0046] The second computing unit is mainly used to process vector computing tasks in large language model operation nodes.

[0047] The second storage unit is arranged outside the chip and is mainly used to store KVcache data in the process of large language model operation.

[0048] Regarding the large language model operation node in this embodiment, the following aspects need to be highlighted:

[0049] 1. Calculation unit of classification design

[0050] In large language model operations, matrix operations and vector operations are the two most basic operations. However, different chip structures may be more efficient for one type of operation and less efficient for another type of operation.

[0051] Different existing technologies set matrix operations and vector calculations in the same calculation unit for processing. This embodiment sets up two independent calculation units - a first calculation unit and a second calculation unit. For matrix operations in large language model operations, they are assigned to the first calculation unit that is better at matrix operations for processing. For vector operations in large language model operations, they are assigned to the second calculation unit that is better at vector operations for processing. This gives full play to the advantages of different design structures and improves computing efficiency. Specifically,

[0052] (a) First computing unit

[0053] The first computing unit is mainly designed for matrix calculations, also known as the matrix computing unit, and is mainly responsible for matrix-related operations, such as matrix-vector multiplication, matrix-matrix multiplication, etc.

[0054] (b) Second computing unit

[0055] The second computing unit is mainly designed for vector computing, also known as the vector computing unit, and is mainly responsible for vector-related operations. Through the combination of some basic operators (addition, subtraction, multiplication, division, power calculation, square root lookup table, etc.), users can flexibly implement any vector operation, such as: various calculations between vectors and vectors, vectors and scalars; addition, multiplication, division, exponential calculations of corresponding positions of vectors; various activation functions, normalization functions, etc.

[0056] Those skilled in the art should understand that the above are only preferred embodiments of the present invention. In other embodiments of the present invention, matrix operations and vector calculations may also be performed by the same computing unit, or more than two computing units may be designed, all of which can implement the present invention and are also within the protection scope of the present invention.

[0057] 2. Classified storage, two separate storage units

[0058] In the large language model operation process, the data types involved include: static data and dynamic data. Among them:

[0059] ① In static data, static weight data is shared by all users, and its characteristics are: frequent calls, but small data volume and relatively fixed.

[0060] ② Dynamic data includes: KV Cache data. KV Cache data is bound to a specific user, that is, different users correspond to different KV cache data. Its characteristics are: the call frequency is relatively low, but the data volume increases with the increase in the number of user questions and the number of users.

[0061] It should be noted that in the process of providing feedback for the current user's real-time input, only the current user's KV-Cache is involved in the calculation, while other users' KV-Cache data is not involved in the calculation.

[0062] In addition, dynamic data also includes: intermediate calculation results related to a specific calculation process; and communication cache data between layer clusters.

[0063] In view of the data types in the large language model calculation process, the present invention has specially set up the storage unit. In this embodiment, the first storage unit is located inside the chip, that is, as an on-chip storage unit; the second storage unit is located outside the chip, that is, as an off-chip storage unit.

[0064] The first storage unit: as an on-chip memory, it is mainly used to store static data, such as static weight data of large language models, and a small amount of dynamic data, such as intermediate calculation results and communication cache data. Compared with the second storage unit, the first storage unit can provide greater bandwidth, lower latency and lower read and write power consumption. By completely residing the static weight data in the first storage unit, the present invention avoids the high power consumption and high latency problems caused by frequent data reading from the second storage unit.

[0065] The second storage unit: as the main user of off-chip memory, it stores KV cache data. The characteristic of the dynamic data KV Cache generated when the large language model is running is that its size increases with the growth of sequence length, so storing it in the on-chip first memory will severely limit the maximum supported sequence length. The present invention uses the second storage unit, that is, off-chip storage to store KV Cache data, and uses the high-density and high-capacity characteristics of off-chip storage to support longer sequence lengths. Furthermore, in order to prevent off-chip storage reading from becoming a performance bottleneck, a single computing node will be configured with multiple external memory channels to increase throughput, and technologies such as dynamic compression will be used to further improve efficiency.

[0066] In order to adapt to the corresponding operation process, call frequency, data volume, read and write update, etc., the difference between the first storage unit and the second storage unit is mainly reflected in two aspects:

[0067] ①Interface bandwidth;

[0068] In the present invention, since the static weight data is called more frequently, the first storage unit storing the static weight data and the computing unit are arranged on the same chip, so that a larger interface bandwidth can be provided between the two. Specifically, in this embodiment, it satisfies: K1≥4K2; wherein K1 is the interface bandwidth between the first storage unit and the computing unit; K2 is the interface bandwidth between the second storage unit and the computing unit.

[0069] Through the above configuration, the present invention effectively avoids the high power consumption and high latency problems caused by traditional external memory weight data reading. Compared with the traditional HBM solution, the present invention has obvious advantages in performance, power consumption and cost.

[0070] Those skilled in the art should understand that the above interface bandwidth settings are only examples. In actual scenarios, the interface bandwidth can be set as needed, and the present invention can also be implemented and is also within the protection scope of the present invention.

[0071] ② Storage capacity

[0072] In the present invention, since the user's KV cache data is large in size and will become larger and larger with the increase of the number of inference layers, the KV Cache data is stored outside the chip, thereby providing a larger storage capacity and more flexible capacity expansion capability.

[0073] In this embodiment, T2≥4T1 is satisfied, where T1 is the storage capacity of the first storage unit, and T2 is the storage capacity of the second storage unit. In addition, the storage capacity of the first storage unit is a fixed setting, and the storage capacity of the second storage unit is an expandable setting.

[0074] In order to meet the above requirements, in this embodiment, the first storage unit is SRAM; the second storage unit is DRAM, and the DRAM and the chip exchange data through a DDR interface.

[0075] The present invention stores user KV Cache data in a second storage unit, which can meet the storage requirements of hot data with ultra-long sequence lengths. At the same time, when the KV cache data continues to increase, the capacity of the second storage unit can be increased at a low cost without replacing the chip.

[0076] Those skilled in the art should understand that the above storage capacity and memory type are only examples. In actual scenarios, the storage capacity and storage type can be set as needed. In particular, the second storage unit can also be RRAM or MRAM. These variations can also implement the present invention and are also within the scope of protection of the present invention.

[0077] 3. Design architecture

[0078] In this embodiment, the computing unit (including: a first computing unit and a second computing unit) and the first storage unit are arranged on the same chip, and the second storage unit is outside the chip.

[0079] In fact, the purpose of placing the computing unit and the first storage unit on the same chip is to ensure the interface bandwidth between the two. This is because compared with the large amount of KV cache data, the static weight data is called very frequently, and sufficient interface bandwidth between the computing unit and the first storage unit can ensure the efficiency of operations in the large language model.

[0080] Those skilled in the art should understand that designing the computing unit and the first storage unit on the same chip is only one way to ensure the interface bandwidth between the two. Other similar design architectures can also be used to ensure the interface bandwidth between the two, for example:

[0081] ① The design architecture is: die; at least one computing unit and the first storage unit are located in the same die; the second storage unit is outside the die;

[0082] ② The same design architecture is: board; at least one computing unit and the first storage unit are located in the same board; the second storage unit is outside the board;

[0083] ③ The design architecture is: packaging of a chip where at least one computing unit is located and a chip where the first storage unit is located; and the second storage unit is outside the packaging.

[0084] ④ The design architecture is: at least one die where the computing unit is located and the die where the first storage unit is located are packaged; the second storage unit is outside the package;

[0085] ⑤ The design architecture is: a functional structure block integrating at least one computing unit and a first storage unit; the second storage unit is outside the functional structure block.

[0086] The above five variations can also ensure the interface bandwidth between the computing unit and the first storage unit, can also implement the present invention, and are also within the protection scope of the present invention.

[0087] ④Interface of computing node

[0088] In this embodiment, the computing node also includes: an external cluster interface, which is used for the computing node to communicate with the upstream processing unit; an internal cluster interface, which is used for the computing node to communicate with other large language model computing nodes in the large language model layer cluster; wherein V1≥4V2, wherein V1 is the interface bandwidth of the internal cluster interface; and V2 is the interface bandwidth of the external cluster interface. In this embodiment, the computing node includes two internal cluster interfaces - a first internal cluster interface and a second internal cluster interface, which are used to implement a front-to-back series setting in the layer cluster.

[0089] In this embodiment, since the static weight data and KV cache data required for the calculation are all located in the layer cluster, only the vector to be calculated (input token vector) comes from outside the layer cluster, and the data amount of the vector to be calculated is limited, therefore, setting the bandwidth of the interface within the cluster to 4 times the bandwidth of the interface outside the cluster can make the best possible use of interface resources and improve computing efficiency.

[0090] Figure 2 Schematic diagram of the structure of a large language model accelerator according to an embodiment of the present invention. Figure 3 This is a design diagram of data storage and decoding calculation of a large language model accelerator according to an embodiment of the present invention. Figure 2 and Figure 3 For the large language accelerator shown, one horizontal layer of computing nodes belongs to the same layer cluster. In the present invention, the static weight data of the large language model will all be placed in the first storage unit, that is, the on-chip memory. However, the on-chip memory capacity of a single computing node is limited, so it is necessary to allocate the static weight data so that it can be stored in the on-chip memory of multiple computing nodes. The first level of allocation is the layer cluster, and one layer cluster is responsible for one layer of the large language model, and there is no interaction of static weight data between layer clusters. Therefore, all static weight data of a specific layer of the large language model are stored in the corresponding layer cluster, specifically, in the first storage unit of multiple computing nodes preset in the layer cluster.

[0091] Based on the above, the second aspect of the present invention provides a large language model layer cluster. In an exemplary embodiment of the present invention, a large language model layer cluster (Layer Cluster) is provided. Figure 2 and Figure 3 As shown, the large language model layer cluster of this embodiment includes: N operation nodes, N≥1, and the operation nodes are the large language model operation nodes of the above embodiment.

[0092] Regarding the large language model layer cluster in this embodiment, the following aspects need special explanation:

[0093] 1. Data stored in the layer cluster

[0094] As described above, in this embodiment, the data stored in multiple computing nodes in a layer cluster are static weight data of the corresponding layer in the large language model and KV cache data of the same layer. There is no weight interaction between layer clusters.

[0095] 2. The number of computing nodes in the layer cluster

[0096] In this embodiment, the number of computing nodes in a single layer cluster is mainly determined by the size of the static weight data of the single-layer large language model and the capacity of the first memory. In addition, as the amount of static weight data increases, the number of computing nodes in the hierarchical group can be expanded. Since the second storage unit is an off-chip memory, its storage capacity is set to be expandable, and the capacity of the second storage unit can be expanded to adapt to the increase in the amount of KV cache data, therefore, the amount of KV cache data does not directly determine the number of computing nodes in the layer cluster.

[0097] 3. Second storage units of different computing nodes in the layer cluster

[0098] In the same layer of cluster, the second storage units of different computing nodes are: physically independent; or logically independent. Both of these methods can implement the present invention.

[0099] 4. Interfaces and connections between computing nodes within the cluster

[0100] like Figure 2 As shown, in this embodiment, in the same layer cluster, N computing nodes are arranged in series in the large language model layer cluster through the intra-cluster interface. In addition, the intra-cluster interface is: PCIE interface or serial port. The external cluster interface is PCIE interface or serial port.

[0101] Based on the above, the third aspect of the present invention provides a large language model accelerator. In an exemplary embodiment of the present invention, a large language model accelerator is provided. Figure 2 and Figure 3 As shown, the large language model accelerator of this embodiment includes: M layer clusters, which are large language model layer clusters as in the above embodiment, M is the number of layers of the large language model, M≥2. Each layer cluster includes: 1 or more computing nodes. The computing node includes: at least 1 computing unit and at least 1 storage unit, wherein: the computing unit is used for computing; the storage unit is used for storing data required for large language model computing. Among them, the layer clusters in the M layer clusters correspond to the layer order of the large language model, and the data required for the large language model computing is stored in the storage unit of the computing node of the corresponding layer cluster.

[0102] Specifically, the large language model accelerator of the present invention is composed of multiple layer clusters (Layer Cluster), each layer cluster contains multiple computing nodes, and the computing nodes in the layer cluster are sequentially connected in series one by one through the corresponding interfaces, and the layer clusters are also sequentially connected in series. In addition, the accelerator will work in conjunction with the upstream processing unit, and each computing node will have a separate interface to communicate with the upstream processing unit. The number of layer clusters and the number of computing nodes can be expanded with the size of the large language model.

[0103] In this embodiment, the upstream processing unit is connected to M layer clusters and is used for the allocation of resources of each layer cluster, tokenization and prefill operations of user input data, and de-tokenization operations of the top layer cluster.

[0104] It should be noted that in Figure 2 and Figure 3 The upstream processing unit in the embodiment is not included in the large language model accelerator of this embodiment. However, in other embodiments of the present invention, the upstream processing unit may also be included in the large language model accelerator, which is also within the protection scope of the present invention.

[0105] In this embodiment, the first level of allocation is the layer cluster, and one layer cluster is responsible for one layer of the large language model, and there is no interaction of static weight data between layers. Therefore, a layer cluster will store all static weight data of its corresponding network layer in the first memory of all its internal computing nodes. The number of computing nodes in a single layer cluster is determined by the amount of static weight data of the single-layer network and the capacity of the first memory. Based on the above, it can be understood that the number of computing nodes in different layer clusters can be the same or different. In addition, in some embodiments, a layer cluster may include only one computing node.

[0106] In this embodiment, the layer clusters are interconnected by chip2chip, but the present invention is not limited to this. In addition to the above-mentioned low-cost chip2chip interconnection solution, if the data transmission volume between chips exceeds the chip2chip capability, it can also be converted to a die2die interconnection form to provide a higher interconnection bandwidth, that is, multiple chips are packaged together in the form of core particles. Alternatively, all dies can be placed on the same wafer and interconnected inside the wafer through a metal layer. These solutions can also implement the present invention and are also within the scope of protection of the present invention.

[0107] Based on the above, the following aspects need to be emphasized again:

[0108] ① Weak interconnection between layer clusters

[0109] Large language models are formed by stacking multiple layers, and there are only very few token interactions between layers. Based on this, the present invention proposes the concept of layer cluster, that is, using a chip or a group of chips to form a basic unit to process a single layer, so as to divide the entire large language model into layers. Since there is no interaction of static weight data or KV cache data between adjacent layer clusters, only the transmission of token vectors, the layer clusters are weakly interconnected.

[0110] This distributed implementation method can effectively split the original large chip into multiple small chip groups to complete the same operation. This can not only effectively improve the communication and computing efficiency of the system, but also reduce the total cost of the chip. Using multiple low-cost chips to complete the same calculation of a large-cost chip can gain advantages in power consumption, price, and performance.

[0111] ② In the same layer cluster, the weak connection between computing nodes

[0112] In the same layer of cluster, there is no interaction of static weight data or KV cache data between computing nodes. Therefore, computing nodes are weakly interconnected, which greatly improves the computing efficiency of computing nodes and reduces the pressure on communication bandwidth.

[0113] ③Stackable architecture

[0114] In the decoder-only structure, the amount of data transferred between layers is small. This feature allows the structure between layers to be stacked layer by layer. The output of the upper layer is used as the input of the lower layer by transferring feature maps between layers. Compared with the memory usage of weights, KV cache data, and token data, the transferred feature maps are smaller, so the bandwidth requirements are relatively loose. Except for the different weights between layers, the calculation structure between layers is completely consistent; their calculation architecture is the same, and the only difference is the data involved in the calculation. Therefore, the stackable architecture design is feasible.

[0115] In the present invention, the large language model accelerator architecture and the layer cluster architecture are both stackable architectures, that is, a combination of hardware architectures suitable for large language model calculations can be selected according to the depth and scale of the large language model. Furthermore, it is quite flexible and convenient to increase or decrease layer clusters and computing nodes according to the actual operation of the large language model.

[0116] ④Reduction of interconnection costs

[0117] Thanks to the separation of prefill and decoding and layer clustering, the present invention has very low requirements for chip interconnection. For most high-performance neural network accelerators, interconnection is usually a large part of the cost. In the present invention, since the layer clusters are weakly interconnected and the computing nodes are weakly interconnected, the interconnection cost can be greatly reduced.

[0118] The following uses the m-th layer cluster as an example to illustrate the operation process in the large language model accelerator. Those skilled in the art should understand that m = 1, 2, ..., M. The m-th layer cluster includes: N operation nodes. The N operation nodes include:

[0119] ① N1 static computing nodes, wherein the first storage unit of the N1 static computing nodes is used to store static weight data of the mth layer in the large language model, 1≤N1≤N;

[0120] ②N2 dynamic computing nodes, the second storage units of the N2 dynamic computing nodes are used to store the KV cache data of the mth layer in the large language model, 1≤N2≤N.

[0121] It should be noted that static computing nodes and dynamic computing nodes are not mutually exclusive. For one of the N computing nodes, it can be in one of the following roles: a pure static computing node; a pure dynamic computing node; a dual role of a static computing node and a dynamic computing node; neither a static computing node nor a dynamic computing node.

[0122] For static weight data and KV cache data, it is simplest if they are stored on only one computing node. However, due to the amount of data and storage capacity, it is difficult to put all the data on one computing node, and it is likely to be placed on multiple computing nodes, which involves the problem of how to store data on different computing nodes and perform calculations on different computing nodes. In the present invention, the storage / calculation of static weight data and KV cache data has similarities and special features, which are described in detail below.

[0123] Taking static weight data as an example, it is shared by all users and is reflected in the weight matrix of S×T, S≥32; T≥32. In general, S≥128, T≥128. Static weight data is stored on N1 nodes, N1≥2. There are two ways to store static weight data on N1 computing nodes: split by "row"; split by "column". The following are explained respectively.

[0124] (1) Split by “row”

[0125] In this segmentation method, the weight matrix is ​​segmented by "rows", and the first storage unit of the nth static operation node among N1 static operation nodes stores the weight row slices of the n1th to n2th rows of the weight matrix, n = 1, 2, ..., N1.

[0126] In this embodiment, the weight matrices of different computing nodes are evenly divided by "rows", and the S rows of the weight matrix are evenly stored in the N1 static computing nodes. However, the present invention is not limited to this, and the number of rows of the static weight matrix stored in different computing nodes may also be different.

[0127] Figure 4 FIG. 1 is a schematic diagram of a static weight matrix in a large language model accelerator according to an embodiment of the present invention, which is divided into N1 static storage units by “rows”. Figure 4 As shown in the figure, assuming the matrix size is 65536×65536, each computing node stores a row slice of size 4096×65536, and there are a total of 16 computing nodes in the current layer cluster to store all the current weights. When the static weight matrix is ​​divided by row, the computing unit in each computing node can directly call the static weight data of the computing node to participate in the calculation, without having to call the static weight data from other layer clusters and other computing nodes, which reduces the requirements for transmission bandwidth and improves the calculation efficiency.

[0128] Furthermore, adapted to the above weight row slices, the current operation node does not need to obtain all the vectors to be calculated, but only needs to obtain the vector slice corresponding to the weight row slice. Specifically, the input vector will also be divided into multiple slices, and each slice is sent to the operation node where the corresponding weight slice is located to complete the calculation. After all static operation nodes are calculated, an accumulation operation unit is required to accumulate the calculation results of each static operation node to obtain the result vector of this matrix-vector multiplication operation.

[0129] Specifically, for the nth computing node in the mth layer cluster, when the static weight matrix is ​​split by "rows" and the weight row slices are stored in the first storage unit, the process of the computing unit performing the matrix-vector multiplication calculation is as follows:

[0130] ① Read row slices from the storage unit of the computing node;

[0131] ② Receive the vector slice corresponding to the row slice in the vector to be calculated;

[0132] ③ Perform matrix-vector multiplication calculation by weight row slices and corresponding vector slices;

[0133] ④ The current layer cluster also includes: an accumulation operation node, which is used to accumulate the calculation results of each operation node in the m-th layer cluster to obtain a result vector of the matrix-vector multiplication calculation;

[0134] Among them, the operation node is a static operation node, the storage unit is a first storage unit, the row slice is a weight row slice, m=1,2,...,M; n=1,2,...,N1.

[0135] The functional structure that splits the vector to be calculated into "vector slices" and sends the "vector slices" to the corresponding operation node is called "upstream operation node". The upstream operation node can be located in: the current layer cluster; or the layer cluster of the previous layer; or the upstream processing unit shared by M large language model layer clusters.

[0136] In a preferred embodiment of the present invention, for the first layer cluster, its upstream computing node is the upstream processing unit; for other layer clusters, its upstream computing node is another computing node located in the current layer cluster other than the static computing node.

[0137] Those skilled in the art should understand that the advantage of storing the weight matrix divided by "rows" in different computing nodes is that the communication overhead is smaller and each node only needs to accept a part of the input vector for calculation.

[0138] (2) Split by “column”

[0139] In this segmentation method, the weight matrix is ​​segmented by "columns", and the first storage unit of the nth static operation node among the N1 static operation nodes stores the weight column slices of the n3th to n4th columns of the weight matrix, n = 1, 2, ..., N1.

[0140] In this embodiment, the weight matrices of different computing nodes are evenly divided by "columns", and the T columns of the weight matrix are evenly stored in the N1 static computing nodes. However, the present invention is not limited to this, and the number of columns of the static weight matrix stored in different computing nodes may also be different.

[0141] Figure 5 This is a schematic diagram of the static weight matrix in the large language model accelerator of an embodiment of the present invention being divided by "columns" and stored in N1 static storage units. Similarly, assuming that the matrix size is 65536×65536, the weight matrix is ​​divided by "columns", and the column slice allocated to each chip contains multiple complete columns (in the above example, each computing node stores 65536×4096). Similarly, when the static weight matrix is ​​divided by "columns", the computing unit in each computing node can directly call the static weight data of the computing node to participate in the calculation, without having to call the static weight data from other layer clusters and other computing nodes, thereby reducing the requirements for transmission bandwidth and improving computing efficiency.

[0142] Furthermore, unlike the case of splitting by "rows", if splitting by "columns", it is necessary to read the complete vector to be calculated, and after each operation node completes the operation, the calculation results need to be spliced ​​to obtain the result vector of the large language model operation at this layer.

[0143] Specifically, for the nth computing node in the mth layer cluster, when the static weight matrix is ​​split by "columns" and the weight row slices are stored in the first storage unit, the process of the computing unit performing matrix vector calculation is as follows:

[0144] ① Read column slices from the storage unit of the computing node;

[0145] ② Receive the vector to be calculated;

[0146] ③ Calculate by column slices and the vector to be calculated;

[0147] ④ The m-th layer cluster further includes: a splicing operation node, which is used to splice the calculation results of each operation node in the m-th layer cluster to obtain a result vector of the matrix-vector multiplication calculation;

[0148] Among them, the operation node is a static operation node, the storage unit is a first storage unit, the column slice is a weight column slice, n=1, 2, ..., N1; or, the operation node is a dynamic operation node, the storage unit is a second storage unit, the column slice is a KVcache column slice, n=1, 2, ..., N2.

[0149] Those skilled in the art should understand that the advantage of storing the weight matrix divided by “columns” in different computing nodes is that the output data does not need to be accumulated, thus simplifying result processing.

[0150] The above is an explanation of the basic operations of storage and matrix-vector multiplication using static weight data as an example. In fact, the processing method of KV cache data is similar to this processing method. KV cache data includes: K matrix, V matrix, which can also be split, stored and participate in matrix-vector multiplication operations according to "rows" or "columns", but the operation nodes are replaced by dynamic operation nodes, the storage unit is the second storage unit, and the row slice is the KV cache row slice, n = 1, 2, ..., N2.

[0151] Different from the static weight matrix, the KV cache data is for different users, that is, different users correspond to different KV cache data. The present invention also needs to solve the problem of how to store the data of multiple users in the second storage units of multiple computing nodes.

[0152] It should be noted that the present invention does not place the KV cache data of the same user on one computing node, because this will cause the subsequent matrix-vector multiplication to be concentrated on the computing node, thereby slowing down the computing speed of the computing node, and the resources of other computing nodes are idle and wasted. The present invention also divides the KV cache data of the same user by "row" or "column", obtains N2 KV cache row slices or KV cache column slices, and distributes them to N2 dynamic computing nodes for storage. In this way, the matrix-vector multiplication operation for the KV cache data will be distributed to multiple dynamic computing nodes, thereby improving resource utilization.

[0153] In addition, the dynamic operation node can also process the matrix-vector multiplication calculation of multiple users in a batch processing manner. Specifically: the storage space of the second storage unit is divided into u segments, each segment is used to store the KVcache of a single user; when receiving a batch processing request containing vectors to be calculated or vector slices of vectors to be calculated of multiple users, the dynamic operation node calls the KV cache data of the corresponding user stored in the corresponding segment of the second storage unit, and performs the matrix-vector multiplication calculation of each user sequentially or in parallel. In the matrix-vector multiplication calculation, the matrix is ​​the user's KV cache data; the vector is the user's: vector to be calculated or vector slice of the vector to be calculated.

[0154] Based on the matrix-vector multiplication calculation of the static weight data and KV cache data in the above layer cluster, the layer cluster in the large language model accelerator can implement the first operation and the second operation:

[0155] 1. First operation

[0156] The first calculation is the calculation involving static weight data. It should be noted that static weight data is relatively stable and is not updated frequently. However, for KV cache data, new KV cache data will be generated every time the user inputs information. The newly generated KV cache data and the previous KV cache data together form new KV cache data.

[0157] The output of the first operation also needs to generate new KV cache data and update the user's KV cache data. As mentioned above, the KV cache data includes: a K cache matrix and a V cache matrix.

[0158] In the first operation, it includes: performing a matrix-vector multiplication calculation. In the matrix-vector multiplication calculation:

[0159] ① The computing nodes involved in the computing are: N1 static computing nodes, and the storage unit is the first storage unit;

[0160] ② The matrix to be calculated is a static weight matrix, and the row slices are weight row slices; or, the column slices are weight column slices;

[0161] ③The vector to be calculated is: the user's input token vector;

[0162] Wherein, the input token vector can be normalized in advance when necessary. For the first-layer cluster, the token vector is obtained by the upstream processing unit. For other-layer clusters, the token vector is obtained by the previous-layer cluster.

[0163] ④The result vector includes: the user's k vector, v vector, and q vector;

[0164] During the update process of KV cache data, the k vector is rotated and position encoded to obtain a new k vector; the new k vector and the v vector are stored in the second storage unit of the corresponding computing node in the m-th layer cluster according to the existing KV cache storage method, and the new k vector and the user's existing K cache matrix together constitute the updated K cache matrix, and the v vector and the user's existing V cache matrix together constitute the updated V cache matrix.

[0165] In addition, the q vector in the result vector will be used as data for matrix-vector multiplication calculation in the second operation, which will be described in detail below.

[0166] Those skilled in the art should understand that the above only describes the contents in the first operation that are related to the inventive concept of the present invention. For other contents in the first operation, such as activation function, normalization function, etc., reference can be made to the relevant description of the prior art and will not be repeated here.

[0167] 2. Second operation

[0168] The second operation involves KV cache data, which can also be called self-attention calculation, mainly the matrix-vector multiplication calculation and activation function of KV cache data and q, k, v vectors.

[0169] In the second operation, the operation nodes involved in the operation are: N2 dynamic operation nodes, the storage unit is the second storage unit, n = 1, 2, ..., N2. As described above, the layer cluster performs the first calculation, and the result vector of the matrix-vector multiplication also includes: the user's q vector;

[0170] The layer cluster performs the second computation, which consists of three stages:

[0171] In the first stage, a matrix-vector multiplication calculation is performed, wherein the matrix involved in the operation is a K cache matrix, wherein the row slice is the row slice of the updated K cache matrix, or the column slice is the column slice of the updated K cache matrix; the vector to be calculated is a q vector; and the result vector obtained is a first result vector;

[0172] In the second stage, the first result vector is activated by the activation function to obtain the second vector to be calculated;

[0173] In the third stage, a matrix-vector multiplication calculation is performed, wherein the matrix involved in the operation is the updated V cache matrix, the row slice is the row slice of the updated V cache matrix, or the column slice is the column slice of the updated V cache matrix; the vector to be calculated is the new vector to be calculated; and the result vector obtained is the third result vector.

[0174] Similarly, those skilled in the art should understand that the above only describes the contents in the second operation related to the inventive concept of the present invention. For other contents in the second operation, reference can be made to the relevant description of the prior art and will not be repeated here.

[0175] 3. Output and input of layer clusters

[0176] In the layer cluster, the first calculation and the second calculation are operated alternately and reciprocally, that is, in one input / output of the layer cluster, the first calculation and the second calculation are not only calculated once, but can also be set by the system to be operated alternately and reciprocally for multiple times.

[0177] When the second calculation is performed after the first calculation, the vector to be calculated in the matrix-vector multiplication calculation of the first stage of the second calculation is the user's q vector output by the first calculation;

[0178] When the first calculation is performed after the second calculation, the vector to be calculated in the matrix-vector multiplication calculation of the first calculation is the third result vector output by the matrix-vector multiplication calculation of the third stage of the second calculation;

[0179] The layer cluster result vector output token output by the current layer cluster is: the result vector of the first calculation or the third result vector of the third stage of the second calculation. Preferably, the layer cluster result vector output by the current layer cluster is the result vector of the first calculation.

[0180] Among them, if the m-th layer cluster is not the last layer, the output token of the cluster result vector of this layer is sent to the m+1-th layer cluster as the vector to be calculated - input token; if the m-th layer large language model is the last layer, the output token of the cluster result vector of this layer is post-processed and de-tokenized and then output to the user.

[0181] 4. Operation process of large language model accelerator

[0182] The concepts of memory-intensive and computation-intensive, this invention focuses on memory-intensive chips, and computation-intensive is handed over to upstream computing units. Here, we divide the processing process of the large language model into two stages, called prefill and decoding stages. After the request is initiated, the prefill stage first vectorizes the user's input and generates the corresponding KV cache; in the decoding stage, the data will go through multiple decoding processes, and the prediction results inferred in each decoding process become the input of the next same decoding reasoning process, and so on.

[0183] In these two stages, the computationally intensive stage requires greater computing power, and the memory-intensive stage requires faster weight access and KV-Cache access. At the same time, the decoding stage is the main stage of large model reasoning. Therefore, the present invention adopts the prefill-decoding split strategy, focusing on accelerating the decoding stage, that is, generating output tokens, thus avoiding the decoding performance loss caused by compatibility with prefill and accelerating the decoding stage to the greatest extent.

[0184] 4.1 In the initialization phase,

[0185] The upstream processing unit transmits the corresponding static weight data to the M level clusters, and each level cluster stores the received static weight data in the first storage unit of the N1 computing nodes to which it belongs.

[0186] The upstream processing unit performs tokenization and prefill calculation on the prompt word input by the user, generates initial KV Cache data and the first output token, and transmits the corresponding initial KV cache data to the M level clusters. Each level cluster stores the received KV cache data in the second storage unit of the N2 computing nodes to which it belongs. The output token is input to the first level cluster for calculation.

[0187] 4.2 Intermediate Operations

[0188] The first output token generated by the upstream processing unit. The output Token is passed to the first layer cluster as an input vector to perform the first calculation and the second calculation. The first calculation or the second calculation is performed alternately and reciprocally. Finally, the layer cluster result vector output Token output by the first layer cluster is: the result vector of the first calculation or the second result vector of the third stage of the second calculation. The layer cluster result vector output Token is input into the second layer cluster as a new input token. The second layer cluster performs the first calculation and the second calculation with the input token as the vector to be calculated. It is passed layer by layer in this way until the last layer cluster outputs the output token vector. The details of the first calculation and the second calculation are not described in detail here.

[0189] 4.3 Result Output

[0190] After the layer cluster calculation of the last layer is completed, the output token vector is output. Post-processing (usually softmax) can be performed in the layer cluster calculation of the last layer or directly passed to the upstream processing unit for calculation. Then the upstream processing unit (such as GPU, etc.) is responsible for de-tokenization and output to the user.

[0191] The currently generated output token will be treated as a new input token and retransmitted to the first layer cluster for a new round of calculation until the output token generated by the last layer cluster is the terminator.

[0192] 5.KV Cache management mechanism and concurrency design

[0193] The present invention has two concurrency levels, the first is batch processing, that is, packaging multiple users' requests into a whole at the same time, and the second is pipeline parallelism.

[0194] 5.1 Batch Processing

[0195] Batch processing is the parallelism of the layer cluster time. For batch processing, in the same layer cluster, for the dynamic computing node, the storage space of its second storage unit is divided into n segments, each segment is used to store the KV Cache of a single user. Under this setting, the dynamic computing node performs the second calculation of the corresponding user in turn. The batch processing method has been described in detail above and will not be repeated here.

[0196] 5.2 Pipeline Parallelism

[0197] Pipeline parallelism is the parallelism between layer clusters.

[0198] For pipeline parallelism, the entire large language model accelerator can be abstracted into an ultra-long multi-stage pipeline. The upstream processing unit generates output tokens based on user input and sends them to the first-stage pipeline. After each layer of cluster processing, it outputs the output token to the next layer of clusters, and then receives new input tokens from the previous layer of clusters to start new processing.

[0199] Therefore, the number of user requests that can be processed simultaneously is the number of pipeline stages multiplied by the batch size.

[0200] 6. Other contents of the large language model accelerator

[0201] In this embodiment, the large language model is a Transformer model based on a decoder-only framework. However, those skilled in the art should understand that the above is only a preferred implementation of this embodiment. In other embodiments of the present invention, the framework of the large language model can also be encoder-only or encoder-decoder, and the present invention can be applied and is also within the protection scope of the present invention.

[0202] In this embodiment, the large language model accelerator is a large language model reasoning accelerator. However, those skilled in the art should understand that the above is only a preferred implementation of this embodiment. In other embodiments of the present invention, the large language model accelerator may also be based on Transformer, especially other networks of decoder-only architecture, or other neural network structures with large intra-layer computation and small inter-layer data transmission, such as various cascade networks based on matrix-vector multiplication. The present invention can be applied to all these scenarios, and they are also within the scope of protection of the present invention.

[0203] In this embodiment, the large language model belongs to one of the following fields: text generation, dialogue system, translation, document summarization. However, those skilled in the art should understand that the above is only a preferred implementation of this embodiment. In other embodiments of the present invention, the large language model can also implement a large language model for other scenarios, such as image generation, video generation, image recognition, image segmentation and other fields.

[0204] In summary, the high-performance large language model accelerator proposed in the present invention not only significantly improves the reasoning performance, but also has the cascadable characteristics, so that it can efficiently support large language models of different specifications. Specifically, the beneficial effects produced by the present invention are as follows:

[0205] (1) Reduce power consumption and latency

[0206] The present invention effectively avoids the high power consumption and high latency problems caused by traditional external memory static weight data reading. Compared with the traditional HBM solution, the present invention has obvious advantages in performance, power consumption and cost.

[0207] (2) Expandable and highly flexible computing power

[0208] In the present invention, a single computing node has a first and a second computing unit specifically for matrices and vectors, and the cascaded architecture can meet the computing power requirements of models of different sizes. The second computing unit is a programmable unit that can flexibly meet different large language model architectures and adapt to different activation functions, normalization and other requirements.

[0209] (3) Support for very long sequences

[0210] In the present invention, by caching the user KV Cache in the second storage unit, the hot data storage requirements of ultra-long sequence lengths can be met.

[0211] (4) Weak interconnection between layer clusters

[0212] The present invention proposes the concept of layer cluster, that is, a chip or a group of chips is used to form a basic unit to process a single layer, so as to divide the entire large language model into layers. Since there is no interaction of static weight data or KV cache data between adjacent layer clusters, only the transmission of token vectors, the layer clusters are weakly interconnected.

[0213] This distributed implementation method can effectively split the original large chip into multiple small chip groups to complete the same operation. This can not only effectively improve the communication and computing efficiency of the system, but also reduce the total cost of the chip. Using multiple low-cost chips to complete the same calculation of a large-cost chip can gain advantages in power consumption, price, and performance.

[0214] (5) Weak interconnection between computing nodes in the same layer cluster

[0215] In the present invention, in the same layer of cluster, there is no interaction of static weight data or KVcache data between computing nodes, so the computing nodes are weakly interconnected, which greatly improves the computing efficiency of the computing nodes and reduces the pressure on the communication bandwidth.

[0216] (6) Stackable architecture

[0217] In the present invention, the large language model accelerator architecture and the layer cluster architecture are both stackable architectures, that is, a combination of hardware architectures suitable for large language model calculations can be selected according to the depth and scale of the large language model. Furthermore, it is quite flexible and convenient to increase or decrease layer clusters and computing nodes according to the actual operation of the large language model.

[0218] (7) Economical and Efficient Interconnection

[0219] In the present invention, prefill and decode are separated to reduce the data transmission requirements between clusters, thereby reducing the dependence on high-cost interconnection technology.

[0220] (8) High concurrency design

[0221] In the present invention, batch processing and pipeline parallelism are used to simultaneously process requests from hundreds of users.

[0222] At this point, the various embodiments of the present invention have been introduced. According to the above description, those skilled in the art should have a clear understanding of the present invention.

[0223] The ordinal numbers used in the specification and claims, such as "first", "second", "third", "primary", "secondary", as well as Arabic numerals, letters, etc., to modify the corresponding elements (or steps), are intended only to clearly distinguish an element (or step) with a certain name from another element (or step) with the same name, and do not mean that the element (or step) has any ordinal number, nor do they represent the order of one element (or step) and another element (or step).

[0224] Meanwhile, unless steps are specifically described or must occur in sequence, the order of the steps is not limited to that listed above and may be changed or rearranged according to desired designs.

[0225] The present invention may also be implemented as a device or apparatus program (e.g., a computer program and a computer program product) for performing a portion or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0226] The present invention can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. The physical implementation of the hardware structure includes but is not limited to physical devices, which include but are not limited to transistors, memristors, DNA computers, single-chip microcomputers, microprocessors or digital signal processors (DSPs). In addition, the present invention is not directed to any specific programming language. It should be understood that the content of the present invention can be implemented using various programming languages, and the description of specific languages ​​herein is to disclose the best implementation of the present invention.

[0227] Those skilled in the art should understand that in the claims and description of the present invention, the word "comprising" does not exclude the existence of elements (or steps) not listed in the claims. The word "a" or "an" preceding an element (or step) does not exclude the existence of multiple such elements (or steps).

[0228] For certain implementations, if they are not the key content of the present invention and are well known to ordinary technicians in the relevant technical field, they are not described in detail in the drawings or text of the specification due to space limitations. In this case, reference can be made to relevant existing technologies for understanding.

[0229] Similarly, it should be understood that in order to simplify the present invention, in the above description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the claims, each inventive aspect lies in less than all the features of the previous single embodiment. Moreover, the embodiments can be mixed and matched with each other or with other embodiments based on design and reliability considerations, that is, the technical features in different embodiments can be freely combined to form more embodiments. Therefore, the claims following the specific embodiment are hereby explicitly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0230] The above specific embodiments provide a detailed description of the objectives, technical means and beneficial effects of the present invention. It should be understood that the purpose of the detailed description is to enable those skilled in the art to understand the present invention more clearly, and it is not used to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A large language model accelerator, characterized in that: include: M layer clusters, M is the number of layers in the large language model, M ≥ 2; Each layer cluster includes: 1 or more computing nodes; The computing node includes: at least one computing unit and at least one storage unit, wherein: The computing unit is used to perform computing; The storage unit is used to store data required for large language model operations; The layer clusters in the M layer clusters correspond to the layer order of the large language model, and the data required for the operation of the large language model is stored in the storage unit of the operation node of the corresponding layer cluster; The data required for the calculation of a specific layer of the large language model includes: static weight data, KV cache data; the N computing nodes in the m-th layer cluster include: N1 static computing nodes; the static computing nodes include: a first storage unit for storing static weight data; the static weight data of the m-th layer in the large language model resides in the first storage unit of the N1 static computing nodes, m=1,2,...,M, 1≤N1≤N; The N computing nodes in the m-th layer cluster include: N2 dynamic computing nodes; the dynamic computing unit includes: a second storage unit for storing KV cache data; wherein the KV cache data of the m-th layer in the large language model is stored in the second storage units of the N2 dynamic computing nodes, m=1,2,...,M, 1≤N2≤N; Among them, for the computing node in the m-th layer cluster, it is in one of the following roles: a simple static computing node; a simple dynamic computing node; a dual role of a static computing node and a dynamic computing node; neither a static computing node nor a dynamic computing node.

2. The large language model accelerator according to claim 1, characterized in that: At least one computing unit and a first storage unit, both of which are located in the same design architecture; The second storage unit is arranged outside the design architecture.

3. The large language model accelerator according to claim 2, characterized in that: Satisfy: K1 ≥ 4K2; Wherein, K1 is the interface bandwidth between the first storage unit and the computing unit; K2 is the interface bandwidth between the second storage unit and the computing unit.

4. The large language model accelerator according to claim 2, characterized in that: The design architecture is: a chip; the at least one computing unit and the first storage unit are located in the same chip, and the second storage unit is outside the chip; Or, the design architecture is: die; the at least one computing unit and the first storage unit are located in the same die; the second storage unit is outside the die; Or, the same design architecture is: a board; the at least one computing unit and the first storage unit are located in the same board; the second storage unit is outside the board; Alternatively, the design architecture is: a chip where the at least one computing unit is located and a chip where the first storage unit is located are packaged; and the second storage unit is outside the package; Or, the design architecture is: the die where the at least one computing unit is located and the die where the first storage unit is located are packaged; the second storage unit is outside the package; Alternatively, the design architecture is: a functional structure block integrating the at least one computing unit and the first storage unit; and the second storage unit is outside the functional structure block.

5. The large language model accelerator according to claim 2, characterized in that: Meet: T2 ≥ 4T1; Wherein, T1 is the storage capacity of the first storage unit; T2 is the storage capacity of the second storage unit.

6. The large language model accelerator according to claim 5, characterized in that: The first storage unit is: SRAM; And / or, the second storage unit is one or more of the following: DRAM, RRAM, MRAM; And / or, the storage capacity of the first storage unit is a fixed setting; and the storage capacity of the second storage unit is an expandable setting.

7. The large language model accelerator according to claim 4, It is characterized by: The first storage unit is: SRAM; the second storage unit is DRAM, and the DRAM and the chip exchange data through a DDR interface; And / or, the first storage unit is also used to store intermediate calculation results of the large language model inference process.

8. The large language model accelerator according to claim 2, characterized in that: The at least one computing unit comprises: A first computing unit, used to process matrix computing tasks in a large language model computing node; The second computing unit is used to process vector computing tasks in the large language model computing node.

9. The large language model accelerator according to claim 2, characterized in that: Also includes: The cluster external interface is used for the large language model computing node to communicate with the upstream processing unit; Intra-cluster interface, used for communication between the large language model computing node and other large language model computing nodes in the cluster; Among them, V1≥4V2, where V1 is the interface bandwidth of the interface within the cluster; V2 is the interface bandwidth of the interface outside the cluster.

10. The large language model accelerator according to claim 1, characterized in that: In the layer cluster: N computing nodes are set up in series in the large language model layer cluster through the intra-cluster interface; And / or, the second storage units of different computing nodes are: physically independently arranged; or, logically independently arranged; And / or, the number of computing nodes in the large language model layer cluster is a scalable setting.

11. The large language model accelerator according to claim 1, characterized in that: The static weight data of the mth layer in the large language model is a weight matrix of S×T, where S≥32; T≥32; N1≥2; The weight matrix is ​​divided into "rows", and the first storage unit of the nth static operation node among the N1 static operation nodes stores the weight row slices of the n1th to n2th rows of the weight matrix, n=1, 2, ..., N1, 1≤n1<n2≤S; Or, the weight matrix is ​​divided by "columns", and the first storage unit of the nth static operation node among the N1 static operation nodes stores the weight column slices of the n3th to n4th columns of the weight matrix, n=1,2,...,N1,1≤n3<n4≤T.

12. The large language model accelerator according to claim 11, characterized in that: The number of static computing nodes in different layers of clusters is the same or different; And / or, the weight matrix is ​​evenly divided by "rows", and the S rows of the weight matrix are evenly stored in the first storage units of N1 static computing nodes; or, the weight matrix is ​​evenly divided by "columns", and the T columns of the weight matrix are evenly stored in the first storage units of N1 static computing nodes.

13. The large language model accelerator according to claim 1, characterized in that: The KV cache data is a KV cache matrix of a user, and the KV cache matrix includes: a K cache matrix; and / or a V cache matrix; N2≥2, the user's KV cache matrix is ​​split by "row" or "column", and N2 KV cache row slices or KV cache column slices are obtained, which are distributed to N2 dynamic computing nodes for storage.

14. The large language model accelerator according to claim 13, characterized in that: The number of dynamic computing nodes in different layers of clusters is the same or different; And / or, the KV cache matrix: is evenly divided by "rows"; or, is evenly divided by "columns"; And / or, the number of users is greater than 1; different users correspond to different KV cache data; and / or, for different users, the KV cache matrix is ​​segmented in the same or different ways; And / or, for different users, the dynamic computing nodes storing KV cache row slices or KV cache column slices are the same or different.

15. The large language model accelerator according to claim 11 or 13, characterized in that: In the nth operation node, the computing unit is used to perform matrix-vector multiplication calculation, including: Read a row slice from the storage unit of the computing node; Receiving a vector slice corresponding to the row slice in the vector to be calculated; Calculating by the row slice and the corresponding vector slice; The m-th layer cluster further includes: an accumulation operation node, which is used to accumulate the calculation results of each operation node in the m-th layer cluster to obtain a result vector of the matrix-vector multiplication calculation; Among them, the operation node is a static operation node, the storage unit is a first storage unit, the row slice is a weight row slice, n=1, 2, ..., N1; or, the operation node is a dynamic operation node, the storage unit is a second storage unit, and the row slice is a KV cache row slice, n=1, 2, ..., N2.

16. The large language model accelerator according to claim 15, characterized in that: Also includes: The upstream computing node is used to split the vector to be calculated and send the vector slices corresponding to the specific row slices to the corresponding computing node; The upstream computing node is located in: the current layer cluster; or, the layer cluster of the previous layer; or, an upstream processing unit shared by M large language model layer clusters.

17. The large language model accelerator according to claim 11 or 13, characterized in that: In the nth operation node, the computing unit is used to perform matrix-vector multiplication calculation, including: Read the column slice from the storage unit of the computing node; Receive the vector to be calculated; Calculating using the column slice and the vector to be calculated; The m-th layer cluster further includes: a splicing operation node, which is used to splice the calculation results of each operation node in the m-th layer cluster to obtain a result vector of the matrix-vector multiplication calculation; Among them, the operation node is a static operation node, the storage unit is a first storage unit, the column slice is a weight column slice, n=1, 2, ..., N1; or, the operation node is a dynamic operation node, the storage unit is a second storage unit, and the column slice is a KV cache column slice, n=1, 2, ..., N2.

18. The large language model accelerator according to claim 13, characterized in that: The number of users is greater than 1; different users correspond to different KV cache data; For dynamic operation nodes, The storage space of the second storage unit is divided into u segments, each segment is used to store the KV cache of a single user; When receiving a batch processing request containing vectors to be calculated or vector slices of vectors to be calculated of multiple users, the dynamic operation node calls the KV cache data of the corresponding users stored in the corresponding segment of the second storage unit, and performs matrix-vector multiplication calculations for each user sequentially or in parallel, in which the matrix is ​​the user's KV cache data; the vector is the user's: vector to be calculated or vector slices of the vector to be calculated.

19. The large language model accelerator according to claim 1, characterized in that: The KV cache data includes: a K cache matrix and a V cache matrix; The layer cluster performs a first calculation, which includes: a matrix-vector multiplication calculation; in the matrix-vector multiplication calculation: the matrix to be calculated is static weight data; the vector to be calculated is: the user's input token vector; the result vector includes: the user's k vector and v vector; The k vector is rotationally encoded to obtain a new k vector; the new k vector and the v vector are stored in the second storage unit of the corresponding operation node in the m-th layer cluster according to the existing KVcache storage method, the new k vector and the user's existing K cache matrix together constitute an updated K cache matrix, and the v vector and the user's existing V cache matrix together constitute an updated V cache matrix.

20. The large language model accelerator according to claim 19, characterized in that: The layer cluster performs a first calculation, wherein the result vector of the matrix-vector multiplication calculation further includes: a q vector of the user; After the first calculation is completed, the layer cluster performs the second calculation, including: In the first stage, a matrix-vector multiplication calculation is performed. In the matrix-vector multiplication calculation, the matrix to be calculated is the user's KVcache matrix; the vector to be calculated is: the user's q vector, and the result vector is: the first result vector; In the second stage, the first result vector is subjected to an activation function to obtain a second vector to be calculated; In the third stage, a matrix-vector multiplication calculation is performed. In the matrix-vector multiplication calculation, the matrix to be calculated is the user's KVcache matrix; the vector to be calculated is the second vector to be calculated; and the result vector is: a third result vector.

21. The large language model accelerator according to claim 20, characterized in that: The first calculation and the second calculation are performed in an alternating manner according to a preset method, wherein: When the second calculation is performed after the first calculation, the vector to be calculated in the matrix-vector multiplication calculation of the first stage of the second calculation is the user's q vector output by the first calculation; When the first calculation is performed after the second calculation, the vector to be calculated in the matrix-vector multiplication calculation of the first calculation is the third result vector output by the matrix-vector multiplication calculation of the third stage of the second calculation; And / or, the layer cluster result vector output token output by the current layer cluster is: the result vector of the first calculation or the third result vector of the third stage of the second calculation; wherein, if the m-th layer cluster is not the last layer, the layer cluster result vector output token is sent to the m+1-th layer cluster as the vector input token to be calculated; if the m-th layer large language model is the last layer, the layer cluster result vector output token is post-processed and de-tokenized and then output to the user.

22. The large language model accelerator according to claim 21, characterized in that: The layer cluster of the M layer adopts pipeline parallel operation mode. Among them, the upstream processing unit generates an output token for the user input and sends it to the first-layer cluster; after each layer of cluster processes and outputs the output token to the next layer of cluster, it receives a new input token from the previous layer of cluster to start new processing.

23. The large language model accelerator according to any one of claims 1 to 14, 19 to 22, characterized in that: The large language model is a Transformer model based on a Decoder-only framework; and / or, the number of computing nodes in clusters at different levels is the same or different; And / or, the large language model accelerator is a large language model reasoning accelerator; and / or, the number of large language model layer clusters in the large language model accelerator is a scalable setting; and / or, the number of large language model operation nodes in the large language model layer cluster is scalable; And / or, the layer clusters are interconnected with each other in one or more of the following ways: chip2chip interconnection; die2die interconnection; board-level interconnection; interconnection inside the wafer through a metal layer.

24. The large language model accelerator according to any one of claims 1 to 14, 19 to 22, characterized in that: The large language model belongs to one of the following fields: text generation, dialogue system, translation, document summarization, image generation, video generation, image recognition, image segmentation.

Images