LLM reasoning-oriented P-D semi-separated heterogeneous multi-core particle integrated architecture

Abstract

The invention provides an LLM inference-oriented P-D semi-separated heterogeneous multi-core-particle integrated architecture, which comprises NPU core particles, which are used for carrying out matrix operation on long sequence output by adopting a parallel processing unit according to the calculation intensive characteristic of a Prefill stage, and a P-D semi-separated heterogeneous multi-core-particle integrated architecture, which is used for carrying out matrix operation on long sequence output by adopting a parallel processing unit according to the calculation intensive characteristic of the Prefill stage, the DRAM-PIM core particle adopts a KV Cache stored on a chip according to the memory access intensive characteristic of the Decode stage, and is provided with a near memory calculation unit for calculating an attention mechanism; according to the 3D-RRAM-CIM core particle, aiming at the characteristic that a weight value is not changed and does not need to be refreshed in the LLM reasoning process, a 3D-RRAM-CIM core particle structure is adopted to store a model weight, and meanwhile weight activation is calculated and used for calculating KV and FFN of a Decode stage. Meanwhile, the invention provides a corresponding ping-pong pipeline design and hardware scheduling method. The overall throughput of the system is remarkably improved, and the reasoning delay is reduced.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit chip design technology, specifically to a PD semi-discrete heterogeneous multi-chip integrated architecture for LLM inference acceleration, and to a ping-pong pipeline design and hardware scheduling method supporting the PD semi-discrete architecture. Background Technology

[0002] In recent years, Large Language Models (LLMs), especially those based on the Transformer architecture, have achieved great success in fields such as natural language processing. However, their massive number of parameters and computational requirements pose serious challenges to inference deployment. The autoregressive inference process of LLMs comprises two computationally distinct stages: the Prefill stage, which processes input prompts, and the Decode stage, which generates tokens one by one. The Prefill stage involves large-scale parallel computation on long sequences, making it a typical computationally intensive task requiring high-throughput computing units. In contrast, the Decode stage processes only one or a small number of tokens at a time, but requires loading the entire history key-value cache (KVCache), making it a typical memory-intensive task with extremely high memory bandwidth requirements.

[0003] Existing technologies for accelerating LLM inference can be broadly categorized into two types. The first is a PD-fused inference architecture, where both the Prefill and Decode stages are deployed on the same hardware, such as using GPUs to handle all stages uniformly. Despite numerous dataflow optimization methods, this inference architecture still has shortcomings, primarily a severe mismatch between hardware resources and computational characteristics: when executing memory-intensive Decode tasks, the autoregressive processing of individual tokens leads to a significant amount of idle computational units; while attempting to combine the two tasks through continuous batching results in resource contention causing long Prefill tasks to block short Decode tasks, creating pipeline "bubbles" that significantly reduce system throughput and increase inference latency.

[0004] The second approach employs a completely separate PD inference architecture, which strictly divides the hardware cluster into a "dedicated Prefill cluster" and a "dedicated Decode cluster" at the physical or software level. While this solution resolves the hardware resource contention issue in PD-inference fusion, its static and fixed resource allocation introduces new problems. First, the fixed hardware resource allocation ratio cannot adapt to dynamic load changes caused by different request combinations (such as long prompts with short generation versus short prompts with long generation), resulting in resource waste and pipeline bubbles. This leads to frequent idle waiting in one cluster, significantly reducing the overall hardware utilization. Second, since both Prefill and Decode involve reading and writing to the KV Cache, each hardware cluster handling Prefill and Decode needs to store its own KV Cache, leading to KV Cache storage redundancy. Furthermore, the PD separation also presents data communication issues between the Prefill and Decode clusters, increasing system inference latency.

[0005] Furthermore, the hardware platforms currently widely used for LLM inference deployments are hardware clusters based on xPUs such as GPUs and NPUs. These traditional xPU homogeneous hardware clusters suffer from limitations in memory space and computing power during LLM inference tasks, making it difficult to meet the ever-increasing computational and storage demands of LLM. Simultaneously, single hardware clusters face challenges such as low hardware utilization during the decoding phase and high communication overhead between multiple devices when dealing with the heterogeneity of memory access during prefill and decode computations in LLM inference tasks, leading to a decline in the overall performance of the LLM inference system. In recent years, the development and maturity of chiplet technology has become a significant driving force for the design of LLM inference hardware accelerators. Chiplet technology integrates small chips with different functions and processes through advanced packaging and interconnect technologies to construct complex system-on-chips (SoCs), and is considered an important technological path to continue Moore's Law. Therefore, by leveraging the inherent heterogeneous integration capabilities of the Chiplet architecture, the different computational and memory access characteristics of Prefill and Decode in LLM inference can be addressed by assigning these two tasks to different types of chips, thus designing a more efficient multi-chip heterogeneous hardware architecture for LLM inference. However, applying Chiplet technology to LLM inference acceleration hardware architecture still faces the following technical challenges: First, there is the challenge of adapting to multiple storage characteristics and hardware architectures. LLM inference involves two distinct data storage requirements: one is model weights, which are massive in volume but remain static during inference; the other is the key-value cache, which grows dynamically with sequence length and requires frequent reads and writes. Existing chiplet solutions often employ a unified storage architecture (such as HBM or DDR), making it difficult to simultaneously balance the high-density, low-power storage of static weights with the high-bandwidth, low-latency access of dynamic key-value caches, resulting in low storage resource utilization and difficulty in achieving breakthroughs in energy efficiency.

[0006] Secondly, there are data flow disruptions and communication bottlenecks caused by the difference in computational paradigms between the Prefill and Decode phases. The Prefill phase involves computationally intensive matrix operations, while the Decode phase involves memory-intensive vector operations. Traditional homogeneous chiplet clusters are ill-suited to simultaneously accommodate these two computational modes. If a simple physical separation (complete PD separation) is used, the massive data transfer bottleneck during the transmission of the KV cache generated by the Prefill phase to the Decode cluster becomes a significant problem, easily leading to severe transmission delays due to the existing D2D interface bandwidth becoming a system bottleneck.

[0007] Finally, the challenge lies in fine-grained pipeline coordination and load balancing among heterogeneous computing units. When integrating various heterogeneous computing chips such as NPU, PIM, and CIM, the computational throughput and response latency of different chips vary significantly. Designing an efficient hardware scheduling mechanism to mask processing latency between different chips while ensuring data dependencies, and dynamically balancing load fluctuations caused by different input sequence lengths to avoid pipeline bubbles, are key technical problems that urgently need to be solved to achieve an efficient multi-chip integrated architecture. Summary of the Invention

[0008] To address the aforementioned shortcomings in the prior art, this invention provides a PD semi-separated heterogeneous multi-core integrated architecture for accelerating LLM inference, and also provides a ping-pong pipeline design and hardware scheduling method that supports the PD semi-separated architecture.

[0009] According to one aspect of the present invention, a PD semi-discrete heterogeneous multi-chip integrated architecture for LLM inference is provided, comprising: an NPU chip, a DRAM-PIM chip, and a 3D-RRAM-CIM chip; wherein: The NPU chip, designed for the computationally intensive nature of the Prefill stage, employs a parallel processing unit to perform matrix operations on the long sequence output of the Prefill stage. The DRAM-PIM chip, designed for the memory-intensive nature of the Decode stage, employs on-chip KVCache and sets up a near-memory computing unit to perform calculations on the attention mechanism during the Decode stage. The 3D-RRAM-CIM core, taking advantage of the characteristic that the weight values ​​do not change and do not need to be refreshed during LLM inference, uses a 3D-RRAM-CIM core structure to store model weights and calculate weight activation, which is used to calculate KV and FFN in the Decode stage.

[0010] Preferably, each DRAM-PIM chip is provided with a portion of schedulable computing units. When the Prefill computing latency is higher than that of the Decode stage, the schedulable computing units are scheduled from the Decode stage to the Prefill stage.

[0011] According to another aspect of the present invention, a ping-pong pipeline design method supporting a PD semi-discrete heterogeneous multi-core integrated architecture is provided, comprising: The NPU chip divides a single batch into two MiniBatches after calculation in the Prefill stage; Based on two MiniBatch, a ping-pong pipeline is used to design uninterrupted data flow for the attention mechanism in the DRAM-PIM chip computing Decode stage and the data flow required for the KV and FFN processes in the 3D-RRAM-CIM chip computing Decode stage.

[0012] Preferably, the uninterrupted data stream design includes: While the DRAM-PIM chip computes the Attention of the current MiniBatch, the 3D-RRAM-CIM chip computes the FFN layer of another MiniBatch and the KV of the next iteration, achieving the uninterrupted data flow required for computation.

[0013] According to a third aspect of the present invention, a hardware scheduling method supporting a PD semi-discrete heterogeneous multi-core integrated architecture is provided, comprising: Based on a heterogeneous architecture mainly composed of NPU chips, DRAM-PIM chips, and 3D-RRAM-CIM chips, when the computation delay of the Prefill stage of a single batch is greater than or equal to a set threshold compared to the Decode stage, the schedulable computing units in the DRAM-PIM chip perform hardware scheduling based on the difference in computation delay between the two stages and the computing power of a single computing unit, so as to maintain similar computation time between the Prefill stage and the Decode stage and achieve dynamic balance of computational load among different chips.

[0014] Preferably, the schedulable computing units in the DRAM-PIM chip are adapted for hardware scheduling based on the difference in computational latency between the two stages and the computing power of a single computing unit, including: By deploying hardware performance counters on each chip, the first execution time of the current inference batch in the Prefill stage and the second execution time in the Decode stage are obtained in real time, and the latency difference between the first execution time and the second execution time is calculated. The latency difference is compared with a preset scheduling trigger threshold. If the latency difference is greater than the scheduling trigger threshold, the number of target computing units that need to be scheduled from the Decode stage to the Prefill stage is calculated based on the latency difference, the unfinished computing load in the current Prefill stage, and the preset computing power value of a single schedulable computing unit in the general matrix operation mode. A hardware reconfiguration command is sent to the DRAM-PIM chip to switch the operating mode of the schedulable computing units of the target number of computing units from near-memory attention computing mode to general matrix computing mode. At the same time, the routing configuration table of the on-chip network is updated, and the data packets of some prefill computing tasks originally intended to be sent to the NPU chip are redirected and distributed to the schedulable computing units after the mode switch.

[0015] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art: This invention significantly improves system throughput and hardware utilization: Based on a heterogeneous multi-core integrated architecture supporting semi-separation of PD (Production Processing) and memory-intensive tasks, computationally intensive and memory-intensive tasks are respectively assigned to the optimal hardware for execution, avoiding resource mismatch problems when a single hardware handles both types of tasks. Furthermore, the adaptive dynamic scheduling algorithm can flexibly allocate computing units in the PIM (Production Inference Module) core to participate in prefill calculations based on real-time load, effectively eliminating pipeline "bubbles" caused by PD latency mismatch, thereby significantly improving the overall throughput of the system in processing LLM (Limited Least Metric) inference requests.

[0016] This invention effectively reduces inference latency: the ping-pong pipeline design and micro-batch partitioning mechanism achieve high parallelism for different chip computation tasks, reducing task switching overhead. The dynamic scheduling algorithm alleviates the computational bottleneck in the Prefill phase, shortening the end-to-end processing time of requests. This invention features architectural flexibility and strong scalability: the heterogeneous multi-core integrated architecture of this invention has a natural modular advantage. The number and ratio of various core types can be flexibly configured according to the needs of different LLM models (e.g., different model depths, different parameter counts) or different application scenarios (e.g., long prompt question-and-answer versus short prompt dialogue), achieving a highly customized and cost-effective hardware solution with excellent scalability.

[0017] This invention implements a semi-separate Prefill-Decode (PD) inference architecture, achieving efficient collaborative processing of various heterogeneous chips through micro-batch partitioning and ping-pong pipeline. Addressing the PD computation latency mismatch issue, a dynamic scheduling algorithm adaptively schedules some processing units in the Processing In Memory (PIM) chip to compute prefill, further optimizing the pipeline, significantly improving overall system throughput, and reducing inference latency. Attached Figure Description

[0018] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of a PD semi-discrete heterogeneous multi-core integrated architecture for LLM inference in a preferred embodiment of the present invention.

[0019] Figure 2 Figures (a) to (c) are detailed internal architecture diagrams of the NPU chip, DRAM-PIM chip, and 3D-RRAM-CIM chip in a preferred embodiment of the present invention.

[0020] Figure 3 This is a schematic diagram of the pipeline design and data flow of a PD semi-discrete heterogeneous multi-core integrated architecture for LLM inference in a preferred embodiment of the present invention. Detailed Implementation

[0021] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0022] Traditional PD-fused / separated inference architectures fall into two categories: the first uses a PD-fused inference architecture, and the second uses a completely separated PD inference architecture. The first architecture suffers from problems such as a large number of idle computing units, low system throughput, and increased inference latency. The second architecture suffers from problems such as frequent cluster idle waiting, low overall hardware utilization, storage redundancy, data communication issues between the Prefill cluster and the Decode cluster, and high system inference latency.

[0023] To address the aforementioned problems of traditional PD fusion / separate inference architectures, and based on the advantages of multi-core heterogeneous architectures compared to traditional LLM inference hardware platforms, this invention provides a PD semi-separate heterogeneous multi-core integrated architecture for LLM inference. This architecture is a high-efficiency PD semi-separate LLM inference acceleration hardware architecture, implementing Prefill-Decode (PD) semi-separate inference. Through micro-batch partitioning and ping-pong pipeline, it achieves efficient collaborative processing of multiple heterogeneous cores. To address the PD computation latency mismatch problem, this invention uses a dynamic scheduling algorithm to adaptively schedule some processing units in the Processing In Memory (PIM) core to compute Prefill, further optimizing the pipeline, significantly improving overall system throughput, and reducing inference latency.

[0024] Specifically, the PD semi-discrete heterogeneous multi-chip integrated architecture for LLM inference provided in this embodiment may include: NPU chip, DRAM-PIM chip, and 3D-RRAM-CIM chip; wherein: The NPU chip is specifically designed for the computationally intensive nature of the Prefill stage, with a built-in massively parallel processing unit, designed to efficiently process matrix operations of long sequence outputs in the Prefill stage and maximize computational throughput.

[0025] To address the memory-intensive nature of the Decode stage, the DRAM-PIM chip stores a KV cache on-chip and incorporates near-memory computing units to reduce communication overhead during memory access. This chip is primarily used for calculating the attention mechanism in the Decode stage. Furthermore, in some preferred embodiments, each DRAM-PIM chip includes a subset of schedulable computing units. When the Prefill computation latency is significantly higher than that of the Decode stage, some computing units are scheduled to Prefill to alleviate the mismatch between Prefill and Decode computation latency.

[0026] The 3D-RRAM-CIM core utilizes the characteristic that the weight values ​​remain unchanged and do not need to be refreshed during LLM inference. It uses RRAM-CIM cores to store model weights and calculate weight activation. To address the issue of the small storage size of two-dimensional RRAM-CIM, this core adopts a 3D-RRAM-CIM core structure, which can accommodate a large number of model weights. This core is responsible for calculating KV generation and FFN calculation in the Decode stage.

[0027] In some preferred embodiments, the NPU chip, in its specific internal architecture, may further include: The Systolic Array, consisting of multiple Processing Units (PEs) interconnected in a two-dimensional network, is used to perform large-scale Generalized Matrix Multiplication (GEMM) calculations in the Prefill phase. A vector processing unit, coupled to the systolic array, is used to perform activation functions, layer normalization, and element-wise operations. The Global SRAM Buffer includes an Input Buffer and a Weight Buffer, which are used to temporarily store the input token embedding vectors read from off-chip memory, some weight data, and intermediate results generated during calculation. The instruction scheduling and control unit is used to parse instructions from the main controller, coordinate the data flow between the pulse array and the vector processing unit, and manage the transfer of data between the SRAM and the computing unit.

[0028] In some preferred embodiments, the DRAM-PIM chip, in its specific internal architecture, may further include: Multiple DRAM Bank Groups are used for high-density storage of dynamically generated KV values ​​during LLM inference; The Near-Memory Logic Layer is vertically stacked with the DRAM memory layer or located on the periphery of the memory cell via through-silicon vias (TSVs) or hybrid bonding technology. Furthermore, the Near-Memory Logic Layer includes multiple Single Instruction Multiple Data (SIMD) processing cores and is equipped with a dedicated attention computing engine, which is used to directly read the KV data in the local bank without having to move the complete KV data out of the chip, and perform dot product operation and Softmax normalization processing with the received query vector. A configurable data flow controller for switching between pure storage mode and near-memory computing mode.

[0029] In some preferred embodiments, each DRAM-PIM chip is provided with a portion of schedulable computing units. When the Prefill computation delay is higher than that of the Decode stage, the schedulable computing units are scheduled from the Decode stage to the Prefill stage.

[0030] In some preferred embodiments, the internal architecture of the 3D-RRAM-CIM chip may further include: A 3D Crossbar Array, in which RRAM devices are located at the intersection of word lines and bit lines, non-volatilely stores the weight matrix of an LLM feedforward neural network (FFN) layer by being programmed to different conductance states; A digital-to-analog converter (DAC) array, connected to the word lines of a cross array, is used to convert the input digital activation signal into an analog voltage signal; An analog-to-digital converter (ADC) array, connected to the bit lines of a cross array, is used to sample and quantize the multiply-accumulate (MAC) current signal in the analog domain, which is performed by Kirchhoff's laws and Ohm's laws. The digital post-processing circuit, including a shift-and-add and an accumulator, is used to reassemble the low-bit portion of the ADC output to recover the high-precision calculation result and to complete the nonlinear activation operation of the FFN layer.

[0031] The PD semi-separated heterogeneous multi-core integrated architecture for LLM inference provided by the above embodiments of the present invention addresses the performance bottleneck problem of traditional xPU-based homogeneous hardware architecture in LLM inference by implementing a hardware architecture design based on multi-core heterogeneity; and addresses the system performance problems caused by the architectural defects of PD fusion and full separation in existing LLM inference architectures by implementing a PD semi-separated hardware architecture, data flow and scheduling algorithm design.

[0032] Based on the architecture provided in the above embodiments of the present invention, an embodiment of the present invention also provides a ping-pong pipeline design method that supports a PD semi-separated architecture.

[0033] Specifically, the ping-pong pipeline design method supporting a semi-separate PD architecture provided in this embodiment may include: After Prefill computation, a single batch is divided into two micro-batches. A ping-pong pipeline enables uninterrupted data flow design for PIM to compute Attention, and 3D-RRAM-CIM to compute Key Values ​​and Forward FFN. Furthermore, in some preferred embodiments, this design specifically involves: while PIM computes the Attention of the current MiniBatch, 3D-RRAM-CIM computes the FFN layer of the other MiniBatch and performs Key Value computation for the next iteration, thereby significantly improving the hardware utilization of both types of chips.

[0034] The ping-pong pipeline design method supporting the PD semi-separated architecture provided in this embodiment aims to achieve deep pipeline matching between the Prefill and Decode stages. After Prefill calculation, a single batch is divided into two mini-batches. Through the ping-pong pipeline, the uninterrupted data flow design of PIM calculation of Attention, CIM calculation of KV and FFN is realized, thereby significantly improving the hardware utilization of the two types of chips.

[0035] Based on the architecture provided in the above embodiments of the present invention, an embodiment of the present invention also provides a hardware scheduling method that supports a PD semi-separated architecture.

[0036] Specifically, the hardware scheduling method supporting the PD semi-separated architecture provided in this embodiment may include: Based on a heterogeneous architecture composed of NPU, DRAM-PIM, and 3D-RRAM-CIM chips, the NPU is responsible for the Prefill computation task, and the Decode task is split into Attention and KV+FFN parts, which are mapped to DRAM-PIM and 3D-RRAM-CIM respectively. At the same time, some schedulable ALU units are set in DRAM-PIM to support Prefill. When the system detects that the Prefill computation latency of a single batch reaches a threshold compared with Decode, the schedulable ALU units in DRAM-PIM will be adapted and scheduled according to the PD computation latency difference and the computing power of a single computing unit, so as to keep the Prefill and Decode computation times as similar as possible and avoid pipeline stalls caused by excessively long Prefill stages.

[0037] In some preferred embodiments, the schedulable ALU units in the DRAM-PIM are adaptively scheduled based on the PD calculation latency difference and the computing power of a single computing unit, and may further include: By deploying hardware performance counters on each chip, the first execution time of the current inference batch in the Prefill stage and the second execution time in the Decode stage are obtained in real time, and the latency difference between the first execution time and the second execution time is calculated. The latency difference is compared with the preset scheduling trigger threshold. If the latency difference is greater than the scheduling trigger threshold, the number of target computing units that need to be scheduled from the Decode stage to the Prefill stage is calculated based on the latency difference, the unfinished computing load in the current Prefill stage, and the preset computing power value of a single schedulable computing unit in the general matrix operation mode. Additionally, a hardware reconfiguration command is sent to the DRAM-PIM chip to switch the operating mode of the schedulable computing units of the target number of computing units from near-memory attention computing mode to general matrix computing mode. At the same time, the routing configuration table of the on-chip network is updated to redirect and distribute some of the prefill computing task data packets originally intended to be sent to the NPU chip to the schedulable computing units after the mode switch.

[0038] The hardware scheduling method provided in this embodiment supports a semi-separated PD architecture. When the Prefill and Decode stages suffer from latency mismatch due to different computational loads, the scheduling algorithm dynamically allocates some processing units in the PIM core to handle the Prefill task. This achieves dynamic balance of computational load among different types of cores, maximizes hardware utilization, and optimizes the overall pipeline execution efficiency.

[0039] The technical solutions provided by the above embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific application examples.

[0040] This specific application example involves a heterogeneous multi-chip integrated architecture, including an NPU chip, a DRAM-PIM chip, and a 3D-RRAM-CIM chip. The NPU chip is specifically designed for the computationally intensive nature of the Prefill stage, incorporating a massively parallel processing unit to efficiently handle matrix operations with long output sequences and maximize computational throughput. The DRAM-PIM chip addresses the memory-intensive nature of the Decode stage by storing a KV cache on-chip and setting up near-memory computing units to reduce communication overhead during memory access. This chip is primarily used for calculating the attention mechanism in the Decode stage. Each DRAM-PIM chip has a portion of schedulable computing units; when the Prefill computation latency is significantly higher than the Decode stage, some computing units are scheduled to Prefill to alleviate the latency mismatch between Prefill and Decode. Finally, leveraging the characteristic that weight values ​​remain unchanged and do not require refreshing during LLM inference, the RRAM-CIM chip is used to store model weights and simultaneously calculate weight activation. To address the issue of the small storage size of 2D RRAM-CIM, a 3D-RRAM-CIM core structure is adopted, which can accommodate a large number of model weights. This core is responsible for calculating the KV generation and FFN calculation in the Decode stage.

[0041] Specifically, such as Figure 1As shown, the NPU chip is used to handle large-scale parallel computing tasks in the Prefill stage, including operators such as General Matrix Multiply (GEMM) and General Matrix-Vector Multiply (GEMV). The DRAM-PIM chip includes on-chip DRAM and an Algorithm Logic Unit (ALU), used for storing the KV cache generated in the Prefill stage and calculating attention in the Decode stage, respectively. The darker-colored ALUs in the diagram are schedulable computing units used to alleviate pipeline mismatch problems caused by excessively long Prefill stages. The 3D-RRAM-CIM is a three-dimensional stacked RRAM-CIM chip used for KV generation and FFN calculation in the Decode stage. The RRAM-based CIM unit takes advantage of the characteristic that the weights remain unchanged and do not need to be refreshed during LLM inference, resulting in better power consumption and read speed.

[0042] like Figure 2 As shown in (a) to (c), more detailed internal architecture diagrams of the three types of chips are presented respectively. The chip quantity ratio of the entire system can be flexibly configured according to the parameters of the LLM model to be deployed. The number of RRAM-CIM chips needs to fully accommodate the model's weight parameters. The ratio of DRAM-PIM and NPU chip quantities needs to be adjusted according to relevant parameters such as the number of model input / output tokens and Batch Size, so that the computational latency of Prefill and Decode in a single batch is basically equal, so as to achieve a compact ping-pong pipeline design, maximize system throughput, and reduce inference latency.

[0043] The PD semi-separated hardware scheduling method involved in this specific application example includes a heterogeneous architecture based on NPU, DRAM-PIM, and 3D-RRAM-CIM chips. The NPU is responsible for the Prefill calculation task, and the Decode task is split into Attention and KV+FFN parts, which are mapped to DRAM-PIM and 3D-RRAM-CIM respectively. At the same time, some schedulable ALU units are set in DRAM-PIM to support Prefill. When the system detects that the Prefill calculation latency of a single batch reaches a threshold compared with Decode, the schedulable ALU units in DRAM-PIM will be adapted and scheduled according to the difference in PD calculation latency and the computing power of a single ALU, so as to keep the Prefill and Decode calculation times as similar as possible and avoid pipeline stalls caused by excessively long Prefill stages.

[0044] The PD semi-separated ping-pong pipeline design method involved in this specific application example includes: after a single batch is calculated in Prefill, it is divided into two micro-batches. Through the ping-pong pipeline, the uninterrupted data flow design of PIM calculating Attention and 3D-RRAM-CIM calculating KV and FFN is realized. Specifically, when PIM calculates the Attention of the current MiniBatch, 3D-RRAM-CIM calculates the FFN layer of the other MiniBatch and the KV calculation of the next iteration, thereby greatly improving the hardware utilization of the two types of chips.

[0045] Specifically, such as Figure 3 As shown, after completing the Prefill task, Batch-A is divided into two MiniBatches, A1 and A2. These two MiniBatches perform parallel computation in the pipeline in a ping-pong manner. When one MiniBatch is assigned to DRAM-PIM for Attention computation, the other MiniBatch is assigned to RRAM-CIM to perform the FFN layer computation of the previous iteration and the KV generation of the next iteration. When the Prefill computation time is significantly longer than the Decode time, it will cause pipeline stalls. Therefore, several schedulable ALU units are allocated in DRAM-PIM for Prefill computation to alleviate pipeline mismatch problems.

[0046] In this specific application example, based on the PD semi-separated heterogeneous multi-core integrated architecture, to achieve deep pipeline matching between the Prefill and Decode stages, a single batch is divided into two mini-batches after Prefill computation. Through a ping-pong pipeline, uninterrupted data flow design is achieved for PIM computation attention and CIM computation key-value and FFN, thereby significantly improving the hardware utilization of the two core types. Furthermore, when latency mismatch occurs between the Prefill and Decode stages due to different computational loads, the scheduling algorithm dynamically allocates processing units from some PIM cores to handle Prefill tasks. This achieves dynamic balance of computational load across different core types, maximizing hardware utilization and optimizing the overall pipeline execution efficiency.

[0047] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0048] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A PD semi-discrete heterogeneous multi-core integrated architecture for LLM inference, characterized in that, include: NPU chips, DRAM-PIM chips, and 3D-RRAM-CIM chips; among which: The NPU chip, designed for the computationally intensive nature of the Prefill stage, employs a parallel processing unit to perform matrix operations on the long sequence output of the Prefill stage. The DRAM-PIM chip, designed for the memory-intensive nature of the Decode stage, employs on-chip KV Cache and includes a near-memory computing unit for calculating the attention mechanism during the Decode stage. The 3D-RRAM-CIM core, taking advantage of the characteristic that the weight values ​​do not change and do not need to be refreshed during LLM inference, uses a 3D-RRAM-CIM core structure to store model weights and calculate weight activation, which is used to calculate KV and FFN in the Decode stage.

2. The PD semi-discrete heterogeneous multi-core integrated architecture for LLM inference according to claim 1, characterized in that, The NPU chip includes: The pulsating array computing unit, consisting of multiple processing units (PEs) interconnected in a two-dimensional network, is used to perform large-scale general matrix multiplication calculations in the Prefill stage. The vector processing unit, coupled to the systolic array computing unit, is used to perform activation functions, layer normalization, and element-level operations. The global SRAM cache, including the input cache and the weight cache, is used to temporarily store the input token embedding vectors read from off-chip memory, some weight data, and intermediate results generated during calculation. The instruction scheduling and control unit is used to parse instructions from the main controller, coordinate the data flow between the pulse array and the vector processing unit, and manage the transfer of data between the SRAM and the computing unit.

3. The PD semi-discrete heterogeneous multi-core integrated architecture for LLM inference according to claim 1, characterized in that, The DRAM-PIM chip includes: Multiple DRAM Bank Groups are used for high-density storage of dynamically generated KV values ​​during LLM inference; The near-memory computing logic layer is vertically stacked with the DRAM Bank Group or located on the periphery of the memory cell via through-silicon vias or hybrid bonding technology. The near-memory computing logic layer contains multiple single instruction multiple data processing cores, each core is equipped with a dedicated attention computing engine, which is used to directly read the KV data in the local bank without having to move the complete KV data out of the chip, and perform dot product operation and Softmax normalization processing with the received query vector. A configurable data flow controller for switching between pure storage mode and near-memory computing mode.

4. The PD semi-discrete heterogeneous multi-core integrated architecture for LLM inference according to claim 1, characterized in that, Each DRAM-PIM chip is provided with a portion of schedulable computing units. When the Prefill computing latency is higher than that of the Decode stage, the schedulable computing units are scheduled from the Decode stage to the Prefill stage.

5. The PD semi-discrete heterogeneous multi-core integrated architecture for LLM inference according to claim 1, characterized in that, The 3D-RRAM-CIM chip includes: A three-dimensional stacked RRAM cross array, wherein RRAM devices are located at the intersection of word lines and bit lines, can non-volatilely store the weight matrix of an LLM feedforward neural network layer by being programmed to different conductance states. A digital-to-analog converter array, connected to the word lines of a cross array, is used to convert the input digital activation signal into an analog voltage signal; An analog-to-digital converter array, connected to the bit lines of a cross array, is used to sample and quantize the multiplicative current signal performed in the analog domain by Kirchhoff's laws and Ohm's laws. The digital post-processing circuit, including a shift adder and an accumulator, is used to reassemble the low-bit portion of the ADC output to recover the high-precision calculation result and to complete the nonlinear activation operation of the FFN layer.

6. A ping-pong pipeline design method supporting a PD semi-separated heterogeneous multi-core integrated architecture, characterized in that, include: The NPU chip divides a single batch into two MiniBatches after calculation in the Prefill stage; Based on two MiniBatch, a ping-pong pipeline is used to design uninterrupted data flow for the attention mechanism in the DRAM-PIM chip computing Decode stage and the data flow required for the KV and FFN processes in the 3D-RRAM-CIM chip computing Decode stage.

7. The ping-pong pipeline design method supporting a semi-separated heterogeneous multi-core integrated architecture for PD as described in claim 6, characterized in that, The uninterrupted data stream design includes: While the DRAM-PIM chip computes the Attention of the current MiniBatch, the 3D-RRAM-CIM chip computes the FFN layer of another MiniBatch and the KV of the next iteration, achieving the uninterrupted data flow required for computation.

8. A hardware scheduling method supporting a PD semi-separated heterogeneous multi-core integrated architecture, characterized in that, include: When the computation delay of the Prefill stage of a single batch is greater than or equal to a set threshold compared to the computation delay of the Decode stage, the schedulable computing units in the DRAM-PIM chip perform hardware scheduling based on the difference in computation delay between the two stages and the computing power of a single computing unit, so as to maintain similar computation time between the Prefill stage and the Decode stage and achieve dynamic balance of computing load among different chips.

9. The hardware scheduling method supporting PD semi-separated heterogeneous multi-core integrated architecture according to claim 8, characterized in that, The schedulable computing units in the DRAM-PIM chip are adapted for hardware scheduling based on the difference in computational latency between the two stages and the computing power of a single computing unit, including: By deploying hardware performance counters on each chip, the first execution time of the current inference batch in the Prefill stage and the second execution time in the Decode stage are obtained in real time, and the latency difference between the first execution time and the second execution time is calculated. The latency difference is compared with a preset scheduling trigger threshold. If the latency difference is greater than the scheduling trigger threshold, the number of target computing units that need to be scheduled from the Decode stage to the Prefill stage is calculated based on the latency difference, the unfinished computing load in the current Prefill stage, and the preset computing power value of a single schedulable computing unit in the general matrix operation mode. A hardware reconfiguration command is sent to the DRAM-PIM chip to switch the operating mode of the schedulable computing units of the target number of computing units from near-memory attention computing mode to general matrix computing mode. At the same time, the routing configuration table of the on-chip network is updated, and the data packets of some prefill computing tasks originally intended to be sent to the NPU chip are redirected and distributed to the schedulable computing units after the mode switch.

Images