Multi-chip storage and calculation integrated calculation device oriented to large language model

Abstract

The invention provides a large language model-oriented multi-core particle storage and calculation integrated calculation device, which is characterized by comprising an IO core particle and a plurality of PIM core particles, and is characterized in that the IO core particle comprises an embedding module used for generating an embedding vector and adding a position code; the IO side data communication module cuts the to-be-calculated vector, distributes the to-be-calculated vector to each PIM core particle, and receives a local calculation feedback vector sent by each PIM core particle; the synchronization module is used for synchronizing each local calculation feedback vector and executing residual addition and layer normalization calculation; the PIM core particle comprises a PIM side data communication module, a token memory and a token side data communication module; a global memory; the plurality of PIM-bank modules store network weights of large language models and are used for executing vector-matrix multiplication according to the local calculation vectors to generate corresponding local calculation feedback vectors; and a softmax module. In short, the method can improve the reasoning speed of the large language model.

Description

Technical Field

[0001] The present invention belongs to the field of large language model reasoning and in-memory processing integrated chips, and specifically relates to a multi-core storage and computing integrated computing device for large language models. Background Art

[0002] Due to the flexibility and scalability of integrated chips, a variety of integrated chip-based technologies have been proposed for high-performance computing, including different hardware architecture solutions and workload partitioning solutions.

[0003] TransPIM implements a PIM architecture based on high-bandwidth memory, or HBM. Each HBM stack has multiple 3D stacked dynamic random access memory, or DRAM core particles, located on top of the computing core particles, connected through silicon vias (TSVs) to provide higher bandwidth and lower access latency. TransPIM uses a token-based sharding bank approach to divide the calculations of different tokens in the static linear layer into different banks to avoid data movement within the layer, and the design of the data flow of tokens between banks reduces data movement during self-attention. However, in order to achieve parallel computing during the pre-filling of large language models, or LLMs, it replicates the same weight matrix in multiple HBM-banks, which makes it difficult to accommodate larger models.

[0004] COMB-MCM implements lossless digital PIM circuits that support unstructured adaptive power consumption regulation. It uses a scalable integrated chip architecture to achieve 2.5D fan-out packaging at 65nm. Each core has unit storage and computing resources, and the storage and computing capacity can be increased proportionally by configuring the number of cores. The integrated chip architecture used by COMB-MCM provides a new design idea for heterogeneous heterogeneous integration in a single package, but lacks specific optimization for the entire LLM process, and requires additional hardware to support LLM's nonlinear operators and data synchronization.

[0005] Simba models the cost and benefit of DNN reasoning on fine-grained integrated chips, and can easily expand any number of cores, support flexible mapping of DNN layers to distributed computing and storage units, and introduce tiling optimization. Gemini implements the "layer-pipeline" LP mapping scheme for DNN multi-core accelerator integrated chips, taking into account reasoning performance, energy consumption, and the manufacturing cost of communication between single cores and bare chips, namely D2D, accurately defines the optimization space of LP and uses the simulated annealing SA engine for exploration.

[0006] The above-mentioned existing workload partitioning schemes on integrated chips are mainly aimed at traditional DNN networks, lacking support for LLM special model architectures and support for different pre-filling and decoding processes. Workload partitioning still has great optimization potential. Summary of the invention

[0007] The present invention is made to solve the above-mentioned problems, and its purpose is to provide a multi-core storage and computing integrated computing device for large language models.

[0008] The present invention provides a multi-core storage and calculation integrated computing device for a large language model, which has the following characteristics, including: an IO core and multiple PIM cores, wherein the IO core includes: an embedding module, which is used to generate an embedding vector for a token input into the large language model and add a position code; an IO side data communication module, which cuts the vector to be calculated and distributes it to each PIM core, and receives the local calculation feedback vector sent by each PIM core; a synchronization module, which is used to synchronize each local calculation feedback vector, and perform residual addition and layer normalization calculation; a token memory, which is used to store the intermediate calculation results generated during the reasoning process of the large language model, and the PIM core includes: a PIM side data communication module, which is used to perform data communication with the IO core; a global memory, which stores the cut vector to be calculated as the local calculation vector; multiple PIM-bank modules, which store the network weights of the large language model, and are used to perform vector-matrix multiplication according to the local calculation vector to generate the corresponding local calculation feedback vector; a softmax module, which is used to perform softmax calculation.

[0009] In the multi-core storage and computing integrated computing device for large language models provided by the present invention, it can also have the following characteristics: wherein the PIM-bank module includes: a sub-memory storing part of the network weights; an IO unit communicating data with the global memory, the softmax module and the PIM-side data communication module; and a computing unit for performing vector-matrix multiplication.

[0010] In the multi-core storage and computing integrated computing device for a large language model provided by the present invention, it can also have the following characteristics: wherein, in a Transformer layer of the large language model, the process of performing self-attention calculation on the vector to be calculated to obtain the result includes the following steps: step S1, the IO side data communication module cuts the vector to be calculated and distributes it to each PIM core; step S2, in each PIM core, the PIM side data communication module receives the cut vector to be calculated as a local calculation vector, and stores the local calculation vector in the global memory; step S3, in each PIM core, the computing unit performs vector-matrix multiplication on the local calculation vector and the network weight to obtain the corresponding Q vector, K vector and V vector, and stores the Q vector in the global memory, and stores the K vector and V vector in the sub-memory; step S4, in each PIM core, the computing unit performs vector-matrix multiplication on the K vector and the Q vector to obtain the corresponding S vector, and transmits it to the softmax module through the IO unit; step S5, in each PIM core, the computing unit performs vector-matrix multiplication on the K vector and the Q vector to obtain the corresponding S vector, and transmits it to the softmax module through the IO unit; step S6, in each PIM core In the IM core, the softmax module performs softmax calculation on the S vector to obtain the corresponding P vector, and stores the P vector in the global memory; step S6, in each PIM core, the calculation unit performs vector-matrix multiplication on the P vector and the V vector to obtain the corresponding local A vector, and sends it to the IO core through the PIM side data communication module; step S7, the synchronization module synchronizes all local A vectors to generate A vectors; step S8, the IO side data communication module cuts the A vectors and distributes them to each PIM core; step S9, in each PIM core, the calculation unit performs vector-matrix multiplication on part of the A vector and the network weight to obtain a local self-attention post-projection output vector, and sends it to the IO core through the PIM side data communication module; step S10, the synchronization module synchronizes all local self-attention post-projection output vectors to generate a self-attention post-projection output vector; step S11, the synchronization module performs residual addition calculation and layer normalization calculation in sequence according to the vector to be calculated and the self-attention post-projection output vector to obtain the result.

[0011] In the multi-core storage and computing integrated computing device for a large language model provided by the present invention, it can also have the following characteristics: wherein, in a Transformer layer of the large language model, the process of performing feedforward calculation on the vector to be calculated to obtain the result includes the following steps: step T1, the IO side data communication module cuts the vector to be calculated and distributes it to each PIM core; step T2, in each PIM core, the computing unit performs vector-matrix multiplication according to the received vector to be calculated and the network weight to obtain a local dimensionality-raising projection output vector, and sends it to the IO core through the PIM side data communication module; step T3, the synchronization module performs vector-matrix multiplication on all local dimensionality-raising projections. The output vector is synchronously generated into an increased-dimensional projection output vector; step T4, the IO side data communication module cuts the increased-dimensional projection output vector and distributes it to each PIM core particle; step T5, in each PIM core particle, the calculation unit performs vector-matrix multiplication according to the received increased-dimensional projection output vector and the network weight to obtain a local reduced-dimensional projection output vector, and sends it to the IO core particle through the PIM side data communication module; step T6, the synchronization module synchronously generates a reduced-dimensional projection output vector for all local reduced-dimensional projection output vectors; step T7, the synchronization module performs residual addition calculation and layer normalization calculation in turn according to the vector to be calculated and the reduced-dimensional projection output vector to obtain the result.

[0012] The multi-core storage and computing integrated computing device for large language models provided by the present invention may also have the following features: in multi-head attention calculations, the IO side data communication module cuts the vector to be calculated according to the attention heads and distributes it to each PIM core, and each PIM core performs calculations on the corresponding attention head.

[0013] In the multi-core integrated computing device for large language models provided by the present invention, it can also have the following characteristics: wherein, for any PIM core, the IO side data communication module cuts the local calculation vector to be processed by the PIM core into multiple local calculation sub-vectors, and sends them to the PIM core in sequence. The PIM core calculates the local calculation sub-vectors received at each time step in combination with the corresponding network weights stored in the global memory, and summarizes the calculation results of the current time step and the calculation results of the previous time step and stores them in the global memory until all local calculation sub-vectors are received.

[0014] In the multi-core storage and computing integrated computing device for large language models provided by the present invention, it can also have the following characteristics: wherein, in the pre-filling stage of the large language model, all PIM-bank modules in each PIM core are divided into two groups, group 0 and group 1, group 0 stores part of the K vectors and V vectors, group 1 stores the remaining K vectors and V vectors, the K vectors stored in group 0 correspond to the V vectors stored in group 1, the V vectors stored in group 0 correspond to the K vectors stored in group 1, after all the K vectors in group 0 complete the self-attention calculation, the K vectors in group 1 are self-attention calculated, and all PIM-bank modules and softmax modules run in a pipeline manner.

[0015] In the multi-core storage-computing integrated computing device for large language models provided by the present invention, it can also have the following characteristics: wherein, in the decoding stage of the large language model, all PIM-bank modules in each PIM core are divided into two groups, group 0 and group 1, group 0 stores part of the K vector and V vector, group 1 stores the remaining K vector and V vector, the K vector stored in group 0 corresponds to the V vector stored in group 1, the V vector stored in group 0 corresponds to the K vector stored in group 1, group 0 and group 1 simultaneously perform vector-matrix multiplication on the stored K vector and the Q vector stored in the global memory to generate the corresponding S vector, and the softmax module generates the P vector corresponding to the S vector, after all S vectors are calculated, group 0 and group 1 perform subsequent self-attention calculations based on the generated P vector.

[0016] In the multi-core storage and computing integrated computing device for large language models provided by the present invention, it can also have the following characteristics: wherein, the embedding module includes: an embedding vector generation unit, which stores a lookup table and is used to generate an embedding vector corresponding to each token according to the lookup table; a coordinate rotation digital calculation unit, which is used to generate a position code corresponding to each token; an adding unit, which is used to combine the embedding vector and the position code corresponding to each token to generate a corresponding token vector, and the vector to be calculated includes the token vector.

[0017] In the multi-core storage and computing integrated computing device for large language models provided by the present invention, it can also have the following characteristics: wherein, the synchronization module includes: a synchronization unit, used to synchronize each local calculation feedback vector to generate a corresponding synchronization vector; a residual addition unit, used to perform residual addition of the synchronization vector and the vector to be calculated to obtain an addition vector; a layer normalization unit, used to perform layer normalization on the addition vector to obtain a normalized vector.

[0018] Functions and Effects of the Invention

[0019] According to the multi-core storage and calculation integrated computing device for large language models involved in the present invention, because the IO core adopts a multi-head multi-core collaborative partitioning scheme to cut the calculation vector and distribute it to each PIM core for calculation, and synchronizes all the local calculation feedback vectors obtained by calculation to generate synchronization vectors, and performs residual addition calculation and subsequent layer normalization calculation according to the synchronization vector, thereby avoiding inefficient data synchronization of PIM cores and redundant weight matrix replication in the ring broadcast mechanism. Therefore, the multi-core storage and calculation integrated computing device for large language models of the present invention can improve the reasoning speed of large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Schematic diagram of the principle of large language model reasoning in an embodiment of the present invention;

[0021] Figure 2 Schematic diagram of a multi-core storage and computing integrated computing device for a large language model in an embodiment of the present invention;

[0022] Figure 3 1 is a flow chart of performing self-attention calculation on a vector to be calculated to obtain a result in an embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of a flow chart of obtaining a result by performing feed-forward calculation on a vector to be calculated in an embodiment of the present invention;

[0024] Figure 5 is a schematic diagram of a multi-head multi-core collaborative partitioning solution in an embodiment of the present invention;

[0025] Figure 6 is a schematic diagram of the working principle of an implicit pipeline in an embodiment of the present invention;

[0026] Figure 7 is a schematic diagram of parallel prefilling-sequential decoding in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following embodiments and the accompanying drawings specifically illustrate the multi-core storage and computing integrated computing device for large language models of the present invention.

[0028] This embodiment provides a multi-core storage and computing integrated computing device for a large language model, which is used to load the large language model and perform reasoning.

[0029] Figure 1 It is a schematic diagram of the principle of large language model reasoning in an embodiment of the present invention.

[0030] like Figure 1As shown in the figure, the large language model includes an embedding module, N Transformer layers, and a language model head. The embedding module embeds words and encodes positions on the input, and then processes it through N Transformer layers and the language model head. The Transformer layer includes a self-attention layer and a feedforward layer.

[0031] Figure 2 It is a schematic diagram of the framework of a multi-core storage and computing integrated computing device for a large language model in an embodiment of the present invention.

[0032] like Figure 2 As shown, the multi-chip storage and computing integrated computing device 100 for a large language model includes an IO chip 10 and multiple PIM chips 20.

[0033] The IO chip 10 includes an embedding module 101 , an IO-side data communication module 102 , a synchronization module 103 and a token memory 104 .

[0034] The embedding module 101 is used to generate an embedding vector for the token input into the large language model and add a positional encoding.

[0035] The embedding module 101 includes an embedding vector generating unit 1011 , a coordinate rotation digital calculating unit 1012 , and an adding unit 1013 .

[0036] The embedding vector generating unit 1011 stores a lookup table, which is used to generate an embedding vector corresponding to each token according to the lookup table.

[0037] The coordinate rotation digital calculation unit 1012 is used to generate the position code corresponding to each token by approximating the sine and cosine functions.

[0038] The adding unit 1013 is used to combine the embedding vector and position code corresponding to each token to generate a corresponding token vector.

[0039] The IO-side data communication module 102 is used to perform data communication with each PIM chip 20 , external devices and external DRAM, and includes a C2C interface 1021 , a DDR interface 1022 and a plurality of D2D interfaces 1023 .

[0040] The C2C interface 1021 is used for data interaction with external devices.

[0041] The DDR interface 1022 is used for data exchange with external DRAM.

[0042] The IO side data communication module 102 cuts the vector to be calculated, and sends the cut vector to be calculated to each PIM core 20 through the D2D interface 1023 corresponding to each PIM core 20, and receives the local calculation feedback vector sent by each PIM core 20. In this embodiment, the transmission speed of the D2D interface 1023 is 256Gbps and complies with the UCIe interface standard.

[0043] In this embodiment, the IO side data communication module 102 realizes a wide range of IO functions through the above-mentioned interface, which can achieve greater parallelism, thereby realizing the scalability of the multi-chip storage and computing integrated computing device 100.

[0044] The synchronization module 103 is used to synchronize various local calculation feedback vectors and perform residual addition and layer normalization calculations.

[0045] The synchronization module 103 includes a synchronization unit 1031 , a residual adding unit 1032 and a layer normalization unit 1033 .

[0046] The synchronization unit 1031 is used to synchronize each local calculation feedback vector to generate a corresponding synchronization vector.

[0047] The residual adding unit 1032 is used to perform residual addition on the synchronization vector and the vector to be calculated to obtain an added vector.

[0048] The layer normalization unit 1033 is used to perform layer normalization on the added vector to obtain a normalized vector.

[0049] In this embodiment, before the self-attention post-projection, dimension-upward projection, and dimension-downward projection calculation, the synchronization unit 1031 needs to splice and synchronize the local calculation feedback vectors generated by each PIM core 20 that arrive at the IO core 10 at different times into a calculation feedback vector, and then the IO side data communication module 102 cuts the calculation feedback vector and distributes it to the corresponding PIM core 20, so as to continue the subsequent projection calculation. Compared with the existing ring broadcast mechanism, the IO core can minimize the bandwidth of data transmission between cores and the hardware resources of the PIM core 20 through the above operation mechanism.

[0050] The token memory 104 is used to store the intermediate calculation results generated during the large language model inference process. In this embodiment, the intermediate results include token vectors, vectors to be calculated for residual addition, and calculation feedback vectors.

[0051] The PIM core 20 includes a PIM-side data communication module 201 , a global memory 202 , a plurality of PIM-bank modules 203 and a softmax module 204 .

[0052] The PIM-side data communication module 201 is used to perform data communication with the IO chip 10. In this embodiment, the PIM-side data communication module 201 is a D2D interface corresponding to the IO chip 10.

[0053] The global memory 202 is used to store the clipped vectors to be calculated as local calculation vectors and Q vectors.

[0054] The plurality of PIM-bank modules 203 store network weights of a large language model and are used to perform vector-matrix multiplication according to the local calculation vector to generate a corresponding local calculation feedback vector.

[0055] The PIM-bank module 203 includes a sub-memory 2031 , an IO unit 2032 , and a calculation unit 2033 .

[0056] The sub-memory 2031 stores part of the network weights. In this embodiment, the sub-memory 2031 is a static random access memory SRAM array or a random access memory RAM type memory as an alternative, such as a memristive random access memory RRAM, or an embedded dynamic random access memory eDRAM, or a magnetoresistive random access memory MRAM.

[0057] In this embodiment, the trained network weights of the large language model are distributed and stored in the sub-memories 2031 of different PIM-bank modules 203 through cutting before starting the inference. The cutting method is cutting on the non-hidden dimension so that the output of each PIM-bank module 203 of each PIM core particle 20, that is, the local calculation feedback vector, can be directly spliced ​​to obtain a complete result.

[0058] The IO unit 2032 is used to perform data communication with the PIM-side data communication module 201 , the global memory 202 , the softmax module 204 and the calculation unit 2033 .

[0059] The computing unit 2033 is used to perform vector-matrix multiplication.

[0060] The softmax module 204 is used to perform softmax calculation.

[0061] In conjunction with the accompanying drawings, the process of implementing the embedded module and the Transformer layer in the multi-core storage and computing device 100 is described respectively below. The implementation process of the language model header is the same as the principle of the implementation process of the above-mentioned Transformer layer, and will not be repeated.

[0062] Figure 3 It is a flowchart of performing self-attention calculation on a calculation vector to obtain a result in an embodiment of the present invention.

[0063] like Figure 1 and Figure 3 As shown in the figure, in a Transformer layer of a large language model, the process of performing self-attention calculation on the calculation vector to obtain the result includes the following steps:

[0064] In step S1 , the IO-side data communication module 102 cuts the vector to be calculated and distributes it to each PIM chip 20 .

[0065] Step S2 : In each PIM core 20 , the PIM-side data communication module 201 receives the trimmed vector to be calculated as a local calculation vector, and stores the local calculation vector into the global memory 202 .

[0066] Step S3, in each PIM core 20, the computing unit 2033 performs vector-matrix multiplication according to the local computing vector read from the global memory 202 by the IO unit 2032 and the corresponding network weight stored in the sub-memory 2031, obtains the corresponding Q vector, K vector and V vector, and stores the Q vector in the global memory 202, and stores the K vector and V vector in the sub-memory 2031.

[0067] Step S4: In each PIM core 20, the calculation unit 2033 performs vector-matrix multiplication on the K vector and the Q vector to obtain the corresponding S vector, and transmits it to the softmax module 204 through the IO unit 2032. In this embodiment, S=K*Q.

[0068] Step S5: In each PIM core 20, the softmax module 204 performs softmax calculation on the S vector to obtain the corresponding P vector. In this embodiment, P = softmax (S).

[0069] Step S6: In each PIM chip 20, the calculation unit 2033 performs vector-matrix multiplication on the P vector and the V vector to obtain the corresponding local A vector, and sends it to the IO chip 10 through the PIM side data communication module 201. In this embodiment, A=P*V.

[0070] Step S7: the synchronization unit 1031 synchronizes all local A vectors to generate an A vector.

[0071] In step S8 , the IO-side data communication module 102 cuts the A vector and distributes it to each PIM chip 10 .

[0072] Step S9, in each PIM core particle 20, the calculation unit 2033 performs vector-matrix multiplication on the partial A vector and the network weight to obtain the local self-attention post-projection output vector, and sends it to the IO core particle 10 through the PIM side data communication module 201.

[0073] Step S10, the synchronization unit 1031 synchronizes all local self-attention post-projection output vectors to generate a self-attention post-projection output vector.

[0074] In step S11, the residual addition unit 1032 performs residual addition calculation according to the vector to be calculated and the self-attention post-projection output vector, and the layer normalization unit 1033 performs layer normalization calculation on the calculation result to obtain the result.

[0075] Figure 4 It is a schematic diagram of a flow chart of obtaining a result by performing feedforward calculation on a vector to be calculated in an embodiment of the present invention.

[0076] like Figure 1 and Figure 4 As shown in the figure, in a Transformer layer of a large language model, the process of performing feed-forward calculation on the vector to be calculated to obtain the result includes the following steps:

[0077] In step T1 , the IO-side data communication module 102 cuts the vector to be calculated and distributes it to each PIM chip 20 .

[0078] Step T2, in each PIM core 20, the calculation unit 2033 performs vector-matrix multiplication according to the received vector to be calculated and the corresponding network weight stored in the sub-memory 2031 to obtain a local dimensional projection output vector, and sends it to the IO core 10 through the PIM side data communication module 201.

[0079] Step T3: the synchronization unit 1031 synchronizes all local up-dimensional projection output vectors to generate an up-dimensional projection output vector.

[0080] In step T4 , the IO-side data communication module 102 cuts the dimension-upgraded projection output vector and distributes it to each PIM core 20 .

[0081] Step T5, in each PIM core 20, the calculation unit 2033 performs vector-matrix multiplication according to the received dimension-upgraded projection output vector and the corresponding network weight stored in the sub-memory 2031 to obtain a local dimension-downgraded projection output vector, and sends it to the IO core 10 through the PIM side data communication module 201.

[0082] Step T6: the synchronization unit 1031 synchronizes all local dimension reduction projection output vectors to generate a dimension reduction projection output vector.

[0083] In step T7, the residual addition unit 1032 performs residual addition calculation according to the vector to be calculated and the dimensionality reduction projection output vector, and the layer normalization unit 1033 performs layer normalization calculation on the calculation result to obtain the result.

[0084] Figure 5It is a schematic diagram of a multi-head and multi-core collaborative partitioning solution in an embodiment of the present invention.

[0085] like Figure 5 As shown, the multi-core storage and computing integrated computing device 100 of this embodiment manages memory occupancy and computing parallelism through a multi-head multi-core collaborative partitioning scheme. The scheme follows the token parallel data flow, and the workload is divided into each PIM core 20 according to the attention head. The scheme divides the projection layer, the self-attention layer, and the linear layer of the feedforward network according to the output channel. Each PIM core 20 independently processes the assigned workload segment and contributes a part of the entire output token. For example, in multi-head attention calculation, there are N attention heads, namely Head, then the IO side data communication module 102 cuts the vector to be calculated according to the attention head and distributes it to N PIM cores 20, each PIM core 20 performs the calculation of the corresponding attention head, and the N PIM cores 20 work together in the subsequent self-attention post-projection output calculation and the dimension-up projection output calculation and dimension-down projection output calculation.

[0086] In this embodiment, the multi-chip storage and computing integrated computing device 100 uses an implicit pipeline to improve operation delay in the multi-head multi-chip collaborative partitioning scheme.

[0087] Figure 6 It is a schematic diagram of the working principle of the implicit pipeline in an embodiment of the present invention.

[0088] like Figure 6 As shown, for any PIM core 20, the IO side data communication module 102 cuts the local calculation vector to be processed by the PIM core 20, i.e., tonken, into multiple local calculation sub-vectors, and sends them to the PIM core 20 in sequence. After receiving the local calculation sub-vector at each time step, the PIM core 20 immediately performs calculations in combination with the corresponding network weight, i.e., weight, stored in the global memory 202, and concatenates the calculation result of the current time step with the calculation result of the previous time step, i.e., summarizes them, and stores them in the global memory 202 until all the local calculation sub-vectors are received. Finally, the PIM core 20 sends the calculation result corresponding to the entire local calculation vector stored in the global memory 202 to the IO core 10, and waits for the IO core 10 to summarize the data sent by all PIM cores and perform the next step of calculation.

[0089] In the pre-filling phase, the large language model completes an encoding process based on all prompt inputs and generates the first new token. For example, the large language model generates "yes" based on multi-token reasoning of the "large language model". Then it enters the decoding phase, and based on the parameters and data of the existing model, it generates subsequent new tokens based on single-token reasoning based on previously generated tokens. For example, "more" is generated based on "yes", and then "strong" is generated based on "more", and so on, until the termination symbol, i.e., a period, is generated.

[0090] In this embodiment, the multi-chip storage and computing device 100 performs self-attention calculations through a parallel pre-filling-sequential decoding scheme.

[0091] Figure 7 is a schematic diagram of parallel prefilling-sequential decoding in an embodiment of the present invention.

[0092] like Figure 7 As shown, (a) is a schematic diagram of parallel pre-filling, and (b) is a schematic diagram of sequential decoding.

[0093] The parallel pre-filling is specifically as follows: in the pre-filling stage of the large language model, all PIM-bank modules 203 in each PIM core 20 are divided into two groups, group 0 and group 1. Group 0 is used to store part of the K vectors and V vectors, and group 1 is used to store the remaining K vectors and V vectors. The K vectors stored in group 0 correspond to the V vectors stored in group 1, and the V vectors stored in group 0 correspond to the K vectors stored in group 1. For example, if K0 to K63 are stored in group 0, then the corresponding V0 to V63 are stored in group 1, and if K64 to K127 are stored in group 1, then the corresponding V64 to V127 are stored in group 0. After all the K vectors in group 0 have completed the self-attention calculation, the K vectors in group 1 are self-attention calculated, that is, in this embodiment, all the K vectors in group 0 are first calculated until all the corresponding A vectors are generated, and then the K vectors in group 1 are calculated, so as to generate all the A vectors that need to be calculated in the PIM core.

[0094] Among them, all PIM-bank modules and softmax modules are run in a pipeline manner. That is, the vector-matrix calculation of Q vector and K vector, the softmax calculation of S vector, and the vector-matrix calculation of P vector and V vector are respectively used as pipeline level 0, pipeline level 1, and pipeline level 2. The previous pipeline level continuously performs corresponding calculations, and when there are computing resources in the next pipeline level, the calculation results of the previous pipeline level are input into the next pipeline level, thereby realizing the process of uninterrupted calculation at each level, that is, pipeline operation.

[0095] Sequential decoding is specifically as follows: in the decoding stage of the large language model, all PIM-bank modules 203 in each PIM core 20 are divided into two groups, group 0 and group 1. Group 0 stores part of the K vector and V vector, and group 1 stores the remaining K vector and V vector. The K vector stored in group 0 corresponds to the V vector stored in group 1, and the V vector stored in group 0 corresponds to the K vector stored in group 1. Group 0 and group 1 simultaneously perform vector-matrix multiplication on the stored K vector and the Q vector stored in the global memory 202 to generate the corresponding S vector, and the softmax module 204 generates the P vector corresponding to the S vector. After all S vectors are calculated, group 0 and group 1 perform subsequent self-attention calculations based on the generated P vector. That is, all PIM-bank modules 203 first perform vector-matrix calculations on Q vectors and K vectors, and after the softmax module 204 completes all P vector calculations, all PIM-bank modules 203 then perform vector-matrix calculations on P vectors and V vectors, so that the number of softmax hardware units performing operations in the softmax module in the decoding module stage is doubled compared to the pre-filling stage, thereby enabling the softmax module in the decoding module stage to operate at the highest utilization rate.

[0096] In this embodiment, the multi-chip storage and computing integrated computing device 100 is subjected to a performance comparison test with the existing Nvidia A100 GPU. The test shows that the multi-chip storage and computing integrated computing device 100 can achieve 2.20 to 11.75 times of reasoning acceleration and 225.55 to 305.41 times of energy efficiency compared with the A100 GPU. In this embodiment, the multi-chip storage and computing integrated computing device 100 is further compared with the existing LLM dedicated accelerator technology. Through the comparison of test data, the multi-chip storage and computing integrated computing device 100 achieves 1.31 to 6.75 times of acceleration and 2.14 times of energy efficiency compared with the LLM dedicated accelerator technology.

[0097] Functions and Effects of the Embodiments

[0098] According to the multi-core storage and computing integrated computing device for large language models involved in this embodiment, the IO core adopts a multi-head multi-core collaborative partitioning scheme to cut the vector to be calculated and distribute it to each PIM core for calculation, and synchronizes all the local calculation feedback vectors obtained by calculation to generate synchronization vectors, and performs residual addition calculation and subsequent layer normalization calculation according to the synchronization vector, thereby avoiding inefficient data synchronization of PIM cores and redundant weight matrix replication in the ring broadcast mechanism. In short, this method can improve the reasoning speed of large language models.

[0099] Furthermore, through implicit pipelines, PIM cores can perform local computations before synchronization is completed, thereby hiding part of the data transmission delay.

[0100] Furthermore, through a parallel pre-filling-sequential decoding scheme, the connections between the PIM-bank modules in the PIM core are adjusted, and the self-attention intermediate results in the pre-filling stage are processed immediately after they are generated, without the need to temporarily store the intermediate results; in the decoding stage, computing resources are allocated with the highest utilization rate, and the computational efficiency of the softmax module is increased to reduce the bottleneck delay of this layer.

[0101] Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A multi-core storage and computing integrated computing device for large language models, used to load large language models and perform reasoning, characterized in that: include: One IO chip and multiple PIM chips, Wherein, the IO core particle comprises: An embedding module, used to generate an embedding vector for a token input into the large language model and add a positional encoding; The IO side data communication module cuts the vectors to be calculated and distributes them to each of the PIM core particles, and receives the local calculation feedback vectors sent by each of the PIM core particles; A synchronization module, used to synchronize the local calculation feedback vectors and perform residual addition and layer normalization calculations; A token memory is used to store the intermediate calculation results generated during the large language model inference process. The PIM core particle comprises: A PIM-side data communication module, used for data communication with the IO chip; A global memory for storing the clipped vector to be calculated as a local calculation vector; A plurality of PIM-bank modules storing network weights of the large language model, used to perform vector-matrix multiplication according to the local calculation vector to generate the corresponding local calculation feedback vector; The softmax module is used to perform softmax calculations.

2. The multi-core storage and computing integrated computing device for large language models according to claim 1, Features: Wherein, the PIM-bank module includes: A sub-memory storing part of the network weights; An IO unit, performing data communication with the global memory, the softmax module and the PIM-side data communication module; A computing unit, configured to perform the vector-matrix multiplication.

3. The multi-core storage and computing integrated computing device for large language models according to claim 2, Features: Wherein, in a Transformer layer of the large language model, the process of performing self-attention calculation on the vector to be calculated to obtain a result includes the following steps: Step S1, the IO side data communication module cuts the vector to be calculated and distributes it to each of the PIM core particles; Step S2, in each of the PIM core particles, the PIM-side data communication module receives the cut vector to be calculated as a local calculation vector, and stores the local calculation vector in the global memory; Step S3, in each of the PIM core particles, the computing unit performs vector-matrix multiplication according to the local computing vector and the network weight to obtain a corresponding Q vector, K vector and V vector, and stores the Q vector in the global memory, and stores the K vector and the V vector in the sub-memory; Step S4, in each of the PIM core particles, the computing unit performs vector-matrix multiplication on the K vector and the Q vector to obtain a corresponding S vector, and transmits the S vector to the softmax module through the IO unit; Step S5, in each of the PIM core particles, the softmax module performs softmax calculation on the S vector to obtain a corresponding P vector; Step S6, in each of the PIM core particles, the calculation unit performs vector-matrix multiplication on the P vector and the V vector to obtain a corresponding local A vector, and sends the vector to the IO core particle through the PIM side data communication module; Step S7, the synchronization module synchronizes all the local A vectors to generate an A vector; Step S8, the IO side data communication module cuts the A vector and distributes it to each of the PIM core particles; Step S9, in each of the PIM core particles, the computing unit performs vector-matrix multiplication on part of the A vector and the network weight to obtain a local self-attention post-projection output vector, and sends it to the IO core particle through the PIM side data communication module; Step S10, the synchronization module synchronizes all the local self-attention post-projection output vectors to generate a self-attention post-projection output vector; Step S11, the synchronization module performs residual addition calculation and layer normalization calculation in sequence according to the vector to be calculated and the self-attention post-projection output vector to obtain the result.

4. The multi-core storage and computing integrated computing device for large language models according to claim 2, Features: The process of performing feedforward calculation on the vector to be calculated in a Transformer layer of the large language model to obtain a result includes the following steps: Step T1, the IO side data communication module cuts the vector to be calculated and distributes it to each of the PIM core particles; Step T2, in each of the PIM core particles, the calculation unit performs vector-matrix multiplication according to the received vector to be calculated and the network weight to obtain a local dimension-raising projection output vector, and sends the vector to the IO core particle through the PIM side data communication module; Step T3, the synchronization module synchronizes all the local up-dimensional projection output vectors to generate an up-dimensional projection output vector; Step T4, the IO side data communication module cuts the dimension-upgraded projection output vector and distributes it to each of the PIM core particles; Step T5, in each of the PIM core particles, the calculation unit performs vector-matrix multiplication according to the received dimension-upgrading projection output vector and the network weight to obtain a local dimension-downgrading projection output vector, and sends the local dimension-downgrading projection output vector to the IO core particle through the PIM side data communication module; Step T6, the synchronization module synchronizes all the local dimensionality reduction projection output vectors to generate a dimensionality reduction projection output vector; Step T7, the synchronization module performs residual addition calculation and layer normalization calculation in sequence according to the vector to be calculated and the dimensionality reduction projection output vector to obtain the result.

5. The multi-core storage and computing integrated computing device for large language models according to claim 1, characterized in that: in, In multi-head attention calculation, the IO-side data communication module cuts the vector to be calculated according to the attention head and distributes it to each of the PIM core particles, and each of the PIM core particles performs calculation of the corresponding attention head.

6. The multi-core storage and computing integrated computing device for large language models according to claim 1, characterized in that: in, For any of the PIM core particles, the IO side data communication module cuts the local calculation vector to be processed by the PIM core particle into multiple local calculation sub-vectors, and sends them to the PIM core particle in sequence. The PIM core calculates the local calculation sub-vector received at each time step in combination with the corresponding network weight stored in the global memory, and aggregates the calculation result of the current time step with the calculation result of the previous time step and stores them in the global memory until all the local calculation sub-vectors are received.

7. The multi-core storage and computing integrated computing device for large language models according to claim 1, characterized in that: in, In the pre-filling stage of the large language model, all the PIM-bank modules in each PIM core are divided into two groups: Group 0 and Group 1. Group 0 stores part of the K vector and V vector, and group 1 stores the remaining K vector and V vector. The K vector stored in group 0 corresponds to the V vector stored in group 1, and the V vector stored in group 0 corresponds to the K vector stored in group 1. After all the K vectors in group 0 have completed the self-attention calculation, the K vectors in group 1 are subjected to the self-attention calculation. All the PIM-bank modules and the softmax module are run in a pipeline manner.

8. The multi-core storage and computing integrated computing device for large language models according to claim 1, characterized in that: in, In the decoding stage of the large language model, all the PIM-bank modules in each PIM core are divided into two groups: Group 0 and Group 1. Group 0 stores part of the K vector and V vector, and group 1 stores the remaining K vector and V vector. The K vector stored in group 0 corresponds to the V vector stored in group 1, and the V vector stored in group 0 corresponds to the K vector stored in group 1. Group 0 and group 1 simultaneously perform vector-matrix multiplication on the stored K vector and the Q vector stored in the global memory to generate a corresponding S vector, and the softmax module generates a P vector corresponding to the S vector. After all the S vectors are calculated, group 0 and group 1 perform subsequent self-attention calculations based on the generated P vector.

9. The multi-core storage and computing integrated computing device for large language models according to claim 1, Features: Wherein, the embedded module includes: An embedding vector generating unit storing a lookup table, configured to generate the embedding vector corresponding to each of the tokens according to the lookup table; A coordinate rotation digital calculation unit, used to generate the position code corresponding to each of the tokens; An adding unit is used to combine the embedding vector corresponding to each of the tokens with the position code to generate a corresponding token vector, The vector to be calculated includes the token vector.

10. The multi-core storage and computing integrated computing device for large language models according to claim 1, Features: Wherein, the synchronization module includes: A synchronization unit, used for synchronizing each of the local calculation feedback vectors to generate a corresponding synchronization vector; A residual adding unit, used for performing residual addition on the synchronization vector and the vector to be calculated to obtain an added vector; The layer normalization unit is used to perform layer normalization on the added vector to obtain a normalized vector.

Images