Large model high-bandwidth ai computing chip based on chiplet core grain architecture and acceleration method

Abstract

This invention belongs to the field of integrated circuit design and microarchitecture technology, and provides a high-bandwidth AI computing chip based on the Chiplet chip architecture and its acceleration method. The method constructs a hardware-enforced data closed loop including read and adaptive prefetching steps, data format rearrangement and inversion steps, systolic tensor calculation triggering steps, bandwidth prediction and allocation steps, dynamic data stream scheduling steps, full-reduce parallel communication steps, and pipeline bubble elimination steps. The memory access acceleration engine outputs prefetched key-value cache data blocks and unformatted weight data blocks through an adaptive prefetching algorithm; the tensor rearrangement engine performs address mapping inversion operations to output column-major weight tensors; the systolic array performs point accumulation and addition operations to generate transmission request trigger signals; and the dynamic bandwidth allocation module outputs predicted bandwidth allocation instructions based on an exponentially weighted moving average algorithm. This invention significantly reduces communication latency, eliminates pipeline bubbles, and improves energy efficiency at the physical level.

Description

Technical Field

[0001] This invention belongs to the field of integrated circuit design and microarchitecture technology, specifically relating to a large-model, high-bandwidth AI computing chip based on the Chiplet chip architecture and its acceleration method. Background Technology

[0002] With large language model parameters spanning hundreds of billions, the resistive-capacitive (RC) delay and parasitic power consumption generated when transporting massive tensor data across physical boundaries in the von Neumann computing architecture have become core physical bottlenecks restricting the overall throughput of computing systems. The maximum manufacturable area of ​​existing monolithic chips is limited by the size of photolithography masks, making it impossible to simultaneously accommodate all the computational logic transistors of a large model and a sufficiently large static random access memory array on a single silicon wafer. The industry has introduced heterogeneous chip-to-chip integration technology to attempt to alleviate the physical limitations of area.

[0003] Existing heterogeneous chip interconnect optimization schemes primarily target general-purpose scalar or vector computing scenarios, employing passive request-response bandwidth arbitration mechanisms. These mechanisms fail to perform hardware-level hard mapping of the periodic matrix block synchronization characteristics determined by the large model. During tensor parallel communication and pipelined parallel synchronization in large models, general-purpose computing scheduling systems rely on compilers to insert synchronization barrier instructions at the software level. These software-level synchronization barriers, coupled with unpredictable off-chip memory read / write latency, lead to severe execution time skew between different computing chips.

[0004] Existing on-chip interconnect network routing nodes are highly susceptible to physical channel buffer overflows and network deadlocks when faced with sudden bursts of large-granularity continuous tensor data blocks in large models. Congestion backpressure signals propagate to the source, forcibly blocking systolic array computation operations and triggering large-scale pipeline bubbles (i.e., pipeline stalls during clock cycles where computation units have no valid data input and are idling). Simply relying on high-level compiler task partitioning optimizations cannot mask cross-network data read / write latency at the physical timing level. Numerous tensor computation units remain idle while waiting for packet transport, routing node computation, and main memory refresh, resulting in significant static power consumption waste. There is an urgent need to construct a forced closed-loop hardware architecture at the chip physical architecture level, incorporating input data prefetching and formatting, deterministic network routing allocation, and hardware-level time difference parallelism. Summary of the Invention

[0005] The purpose of this invention is to provide a high-bandwidth AI computing chip and acceleration method for large models based on the Chiplet chip architecture, which solves the technical problems of RC delay bottleneck, severe pipeline bubbles and large static power consumption waste in large model tensor parallel and pipeline parallel scenarios caused by the reliance on software synchronization barriers and passive bandwidth arbitration mechanisms in the existing Chiplet architecture.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] A method for accelerating large-scale, high-bandwidth AI computing power chips based on the Chiplet chip architecture, applied to a heterogeneous physical interconnect network consisting of computing chip clusters, memory chip clusters, and interface chip clusters, comprising the following steps:

[0008] Step S1: Reading and Adaptive Prefetching Step; The memory access acceleration engine reads the initial word sequence received by the interface core and the original large model weight matrix stored in the memory core cluster, and outputs the prefetched key-value cache data block and the unformatted weight data block to the distributed cache subsystem through the adaptive prefetching algorithm.

[0009] Step S2: Data format rearrangement and inversion step; The tensor rearrangement engine extracts the unformatted weight data block from the distributed cache subsystem, performs a one-dimensional address mapping inversion operation from row-major to column-major, and outputs a column-major weight tensor;

[0010] Step S3: Pulsating tensor calculation trigger step; The pulsating array of the tensor calculation subunit synchronously receives the prefetched key-value cache data block and the column main order weight tensor from the distributed cache subsystem, performs point accumulation and addition operations, outputs the local intermediate result tensor to the distributed cache subsystem, and generates a transmission request trigger signal;

[0011] Step S4: Bandwidth prediction and allocation step; The dynamic bandwidth allocation module receives the transmission request trigger signal, calculates and outputs the predicted bandwidth allocation instruction based on the requested bandwidth occupancy value of the current scheduling period and the predicted bandwidth capacity scalar generated in the previous scheduling period, using the exponential weighted moving average algorithm.

[0012] Step S5: Dynamic data stream scheduling step; The intelligent scheduler receives the predicted bandwidth allocation instruction and the real-time bandwidth status data of the on-chip interconnect network, performs optimal path mapping based on graph neural network, outputs the optimal transmission path identifier, and reads the local intermediate result tensor from the distributed cache subsystem and encapsulates it into a local result tensor to be transmitted.

[0013] Step S6: Full Reduce Parallel Communication Step; The on-chip interconnect network identifies the optimal transmission path, and the pipeline-level double buffering mechanism alternately sends and receives the local result tensor to be transmitted, and the global reduced tensor sequence is generated by assembling the downstream computing chips.

[0014] Step S7: Pipeline bubble elimination step; The dynamic pipeline depth adjustment unit of the downstream computing chip receives the global reduced tensor sequence, dynamically adjusts the depth of the asynchronous pipeline buffer register according to the extracted computing speed difference data of adjacent computing chips, and transmits the synchronous operation feature map, wherein the synchronous operation feature map is seamlessly used as the input tensor for the next layer model calculation, forming a hardware data closed loop.

[0015] Furthermore, the method also includes step S8 throughout the entire process: dynamic power consumption management step; the performance monitoring module captures computing power utilization data at each stage in real time, and when the computing power utilization data is lower than a preset threshold, it outputs a voltage frequency adjustment signal to the power distribution network control circuit.

[0016] Furthermore, in step S1, the adaptive prefetch algorithm is based on a smooth prefetch hit rate scalar. For the prefetch data block window byte length Dynamic scaling is performed, and the length of the prefetch window in bytes is constrained to a preset minimum value. With the maximum value The iterative relationship between them satisfies:

[0017]

[0018] in, To prefetch the reference step size constant for a single adjustment of the window, This is the preset prefetch hit rate activation threshold parameter.

[0019] Furthermore, the smoothed prefetch hit rate scalar From instantaneous hit indicator scalar With preset smoothing coefficient The calculation is obtained using the following formula:

[0020]

[0021] in, This represents the instantaneous hit indicator scalar, which is set to 1 for hits and 0 for misses when the pulsating array accesses the prefetch cache space.

[0022] Furthermore, in step S2, the one-dimensional address mapping inversion operation converts the row-major physical address into a column-major one-dimensional target physical address. The conversion logic is as follows:

[0023]

[0024] in, To output the constant starting address of the video memory segment, For column coordinate index variables, The total number of rows is a constant. This is the row coordinate index variable.

[0025] Furthermore, in step S4, the dynamic bandwidth allocation module uses an exponentially weighted moving average algorithm to calculate the bandwidth capacity scalar in the predicted bandwidth allocation instruction. satisfy:

[0026]

[0027] in, This is the smoothing weighting factor constant.

[0028] Furthermore, in step S5, the intelligent scheduler comprises a graph neural network with three layers of graph convolution operations and one fully connected layer, and its node feature aggregation operation formula is as follows:

[0029]

[0030] in, It is the ReLU activation function. For the adjacency matrix constant of the full mesh topology of the on-chip interconnect network with self-loops, The degree matrix is ​​constant. For the first The layer can learn the weight matrix parameters.

[0031] Furthermore, in step S6, the tensor of the local result to be transmitted is divided into data packets of a size matching the maximum transmission unit of 1024 bytes.

[0032] Furthermore, the pipeline-level double buffering mechanism includes buffer group A and buffer group B, and the total number of clock cycles consumed to complete the model calculation and communication synchronization of one layer is [not specified]. satisfy:

[0033]

[0034] This reduces the single-packet communication consumption cycle. Basic folding in the calculation cycle Inside, where The number of computation cycles for a single data block. The total number of blocks, Set a fixed delay for the production line startup.

[0035] Furthermore, in step S7, the target depth level of the asynchronous pipeline buffer register is... Calculate the periodic value of the speed difference Trigger step adjustment, the adjustment logic is as follows:

[0036]

[0037] in, The maximum depth level, This is the current level number. To minimize the step size constant, Let be the step decision function. The threshold constant for tolerating jitter clock fluctuations.

[0038] Furthermore, in step S8, the target power supply voltage analog value controlled by the voltage frequency adjustment signal... Scalar of normalized computing power utilization Linear mapping:

[0039]

[0040] in, Minimum retention voltage, This is the rated supply voltage.

[0041] Furthermore, the smoothing coefficient The system is dynamically adjusted based on changes in hit rate. The adjustment logic is as follows:

[0042]

[0043] in, To adjust the step size factor, This is the amplitude limiting function.

[0044] Furthermore, in step S1, the memory access acceleration engine also generates a sparse mask bitmap corresponding to the original large model weight matrix and stores it in the prefetch key-value cache data block; in step S3, each multiply-accumulate unit of the systolic array receives the corresponding mask bit in the sparse mask bitmap. When the mask bit is 0, the multiply-accumulate unit skips multiplication and addition operations while maintaining pipeline synchronization to reduce dynamic power consumption.

[0045] In addition, this invention also discloses a large-model, high-bandwidth AI computing chip based on the Chiplet chip architecture, used to implement the above-described acceleration method for a large-model, high-bandwidth AI computing chip based on the Chiplet chip architecture, including:

[0046] The distributed caching subsystem is used to cache the prefetched key-value cache data block and the unformatted weight data block output from step S1, as well as the local intermediate result tensor output from step S3.

[0047] The video memory access acceleration engine is used to read the initial word sequence received by the interface core and the original large model weight matrix stored in the video memory core cluster. It outputs the prefetch key-value cache data block and the unformatted weight data block to the distributed cache subsystem through an adaptive prefetch algorithm.

[0048] The tensor rearrangement engine is used to extract the unformatted weighted data block from the distributed caching subsystem, perform a one-dimensional address mapping inversion operation from row-major to column-major, and output a column-major weight tensor.

[0049] Tensor computation subunit is used to synchronously receive the prefetch key-value cache data block and the column main order weight tensor from the distributed cache subsystem through its pulsating array, perform point accumulation and addition operations, output local intermediate result tensor to the distributed cache subsystem, and generate a transmission request trigger signal;

[0050] The system includes a dynamic bandwidth allocation module and an intelligent scheduler. The dynamic bandwidth allocation module receives the transmission request trigger signal, calculates and outputs a predicted bandwidth allocation instruction based on the requested bandwidth occupancy value of the current scheduling period and the predicted bandwidth capacity scalar generated in the previous scheduling period using an exponentially weighted moving average algorithm. The intelligent scheduler receives the predicted bandwidth allocation instruction and real-time bandwidth status data of the on-chip interconnect network, performs optimal path mapping based on a graph neural network, outputs the optimal transmission path identifier, and reads the local intermediate result tensor from the distributed caching subsystem and encapsulates it into a local result tensor to be transmitted.

[0051] An on-chip interconnect network configured with a double buffering mechanism and a full mesh topology is used to identify the optimal transmission path. The pipeline-level double buffering mechanism is used to alternately send and receive the local result tensor to be transmitted, and the global reduced tensor sequence is generated by assembling the downstream computing particles.

[0052] The dynamic pipeline depth adjustment unit is used to receive the global reduced tensor sequence, dynamically adjust the depth of the asynchronous pipeline buffer register based on the extracted data of the difference in computing speed between adjacent computing particles, and transmit out the synchronous operation feature map.

[0053] Furthermore, it also includes a performance monitoring module and a power distribution network control circuit, which are used to capture computing power utilization data at each stage in real time. When the computing power utilization data is lower than a preset threshold, a voltage frequency adjustment signal is output to the power distribution network control circuit.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] This invention breaks down the abstraction layer barrier between on-chip interconnect networks and large-scale distributed algorithms at the chip physical architecture level by constructing a complete hardware closed loop from memory prefetching, data rearrangement, systolic computation to bandwidth prediction, dynamic scheduling, full Reduce communication, and pipeline bubble elimination. Unlike existing technologies that rely on compiler-inserted software synchronization barrier instructions and passive request-response bandwidth arbitration mechanisms, this invention utilizes a hybrid-bonded silicon interposer full-mesh network at the physical level to reduce parasitic parameters of package-level interconnects. At the network level, it establishes a dynamic bandwidth allocation mechanism based on wormhole routing predicted by an exponentially weighted moving average algorithm. At the timing level, it configures a double-buffering mechanism including asynchronous pipeline buffer registers, concealing cross-network data transmission time within the tensor matrix dot product operation cycle. This eliminates large-scale pipeline bubbles, significantly reduces communication latency, improves the overall throughput of the computing system, and reduces static power consumption waste caused by idle computing units.

[0056] This invention introduces a dynamic power consumption management mechanism. The performance monitoring module captures computing power utilization data at each stage in real time. When the computing power utilization is lower than a preset threshold, the power distribution network control circuit implements voltage and frequency adjustment, enabling the computing chip to switch to low voltage retention mode at nanosecond speed. This achieves strong coupling and adaptive voltage drop between dynamic power consumption and communication data flow, further improving the energy efficiency ratio.

[0057] In terms of adaptive prefetching, this invention reduces the standard deviation of the prefetch hit rate scalar fluctuation by dynamically adjusting the smoothing coefficient, thereby reducing overshoot in the prefetch window adjustment and improving the effective bandwidth utilization of the video memory. Regarding the systolic array, sparse-aware multiply-accumulate control is introduced. By attaching a sparse mask bitmap to the prefetch key-value cache data block and adding gating logic to the input of the multiply-accumulate unit, zero-weight triggered operation skipping and pipeline synchronization are achieved. This effectively reduces dynamic power consumption while maintaining constant array computation latency, resulting in a significant improvement in single-chip energy efficiency. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained from these drawings without creative effort.

[0059] Figure 1 This is a flowchart of the method described in this invention.

[0060] Figure 2 This is a flowchart of the dynamic power consumption management steps of the present invention.

[0061] Figure 3This is a flowchart of the adaptive prefetching smoothing coefficient dynamic adjustment sub-process of the present invention.

[0062] Figure 4 This is a flowchart of the pulsating array sparse sensing multiplication and accumulation control sub-process of the present invention. Detailed Implementation

[0063] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0064] The following is in conjunction with the appendix Figures 1-4 The embodiments of the present invention will be described in detail below.

[0065] Example 1: This example provides a method for accelerating large-model, high-bandwidth AI computing power chips based on the Chiplet chip architecture. The physical architecture includes a computing chip cluster, a memory chip cluster, and an interface chip cluster.

[0066] The input data chain traces back to the initial word sequence of external input, which is received and carried by the PCIeGen5 interface subunit in the interface chip; the original large model weight matrix is ​​fixed or preloaded into the HBM storage medium unit of the memory chip cluster. It is understood that PCIeGen5 is only an exemplary interface type; this invention can also be implemented using other high-speed interfaces such as PCIeGen6 and CXL, specifically including the following steps:

[0067] Step S1: Reading and Adaptive Prefetching; The initial lexical sequence and the original large model weight matrix serve as the initial physical input at the system bus level. The memory access acceleration engine sends continuous burst read commands to the memory chip cluster via a physical bus composed of a hybrid bonding microbump array on the silicon interposer. The memory access acceleration engine is internally configured with programmable address filtering logic. This logic, based on a pre-configured address mapping table, demaps the data stream read from the HBM in real time, identifying data belonging to the prefetch key-value cache data block as prefetch key-value cache data blocks and data belonging to the original large model weight matrix as unformatted weight data blocks, thus completing the physical separation of the two data streams.

[0068] To avoid memory access latency blocking subsequent arithmetic logic units, the memory access acceleration engine internally instantiates a prefetch unit. The prefetch unit contains embedded adaptive prefetch algorithm logic circuitry. By monitoring bus address transition patterns, the prefetch unit predicts the upcoming physical memory address segment and pre-moves data to the prefetch cache space within the compute core. The adaptive prefetch algorithm relies on iterative updates of two core state variables over clock cycles, including prefetch hit rate evaluation logic and dynamic prefetch window scaling logic.

[0069] When calculating the smoothed prefetch hit rate scalar, the evaluation logic executes the following iterative formula within the digital signal processor module:

[0070]

[0071] in, Indicates clock period The smoothed prefetch hit rate scalar calculated at each time step. Indicates clock period The instantaneous hit indicator scalar generated by the time-lapse array accessing the prefetch cache space is set by a trigger when a cache space address label is hit. When a miss occurs, the trigger resets to its original value. , This represents the prefetch hit smoothing coefficient, initially supplied by the on-chip configuration register and set to [value missing]. , Indicates clock period The time register stores the smooth prefetch hit rate scalar.

[0072] Using the smooth prefetch hit rate scalar obtained through iteration, the prefetch window dynamic scaling logic adjusts the burst read length of video memory:

[0073]

[0074] in, Indicates clock period The length of the prefetch data block window that will be written to the video memory controller register at any given time. Indicates clock period The current prefetch block window length in bytes at any given time. This represents the constant of the reference step size for a single adjustment of the prefetch window, set to [value]. byte, This represents the preset prefetch hit rate activation threshold parameter, which is fixed. The threshold is calibrated based on massive simulation data from a large-scale model memory prefetching scenario with hundreds of billions of models. This represents a sign extraction function circuit that outputs a value when the input value is greater than zero. =Equal to zero output Output less than zero .

[0075] The prefetch window byte length The value is constrained by the hardware to a preset minimum value. With the maximum value Between; if the value calculated according to the formula is less than Then take If greater than Then take . and As a constant permanently stored in read-only memory, this embodiment sets byte, byte.

[0076] After step S1 is completed, the memory access acceleration engine generates two independent data streams with clearly defined destinations: the extracted prefetch key-value cache data block and the unformatted weight data block. Both data streams are stored in the static random access memory array of the distributed caching subsystem, awaiting downstream consumption.

[0077] Step S2: Data format rearrangement and inversion; unformatted weight data blocks follow a row-major, one-dimensional linear physical address arrangement in the storage medium. To match the spatial locality characteristics of large-model intensive matrix dot product calculations, the tensor rearrangement engine extracts unformatted weight data blocks from the distributed cache subsystem via a 1024-bit wide internal bus.

[0078] The tensor rearrangement engine comprises a non-blocking crossbar switch matrix consisting of multiplexers and address mapping logic. The tensor rearrangement engine performs a row-major to column-major address mapping inversion operation. The address mapping logic performs a one-dimensional address translation for each matrix element in a single clock cycle.

[0079]

[0080] in, This indicates the column-major one-dimensional target physical address to be output to the next level buffer. This represents the constant address of the first segment of the output memory pre-allocated by the tensor rearrangement engine. This variable represents the column coordinate index of a matrix element, with a value range from... To the total number of columns minus , This is a constant representing the total number of rows in the weight matrix of the original large model to be processed. This variable represents the row index of a matrix element, with a value ranging from... Total number of lines reduced .

[0081] After the S2 stage operation is completed, the tensor rearrangement engine accurately outputs the column-major weight tensor to the data bus. At this point, the unformatted weight data block is completely transformed, with no redundant intermediate data remaining.

[0082] Step S3: Pulsating tensor computation triggered; The prefetch key-value cache data block produced in step S1 and the column-major weight tensor produced in step S2 are both stored in the distributed cache subsystem. Under the same clock tick, the pulsating array of the tensor computation subsystem synchronously reads both from the distributed cache subsystem to the input latch.

[0083] The core of the tensor computation subunit consists of a 64x64 row systolic array of 4096 independent multiply-accumulate units. This array is designed to handle the granularity of single-round matrix block computation in a Transformer model with hundreds of billions of nodes. Internal register transfers within the array utilize a single-clock global synchronization cycle, with all multiply-accumulate units synchronously performing data latching, floating-point operations, and result accumulation. Prefetch key-value cache data blocks are pipelined from left to right into the array row ports, while column-major weight tensors are pipelined from top to bottom into the array column ports. The multiply-accumulate units capture cross-point input signals and perform a fusion operation of floating-point dot product and accumulation. This stage removes all memory scheduling logic and purely executes arithmetic operations. After 64 computation clock cycles of full-load pipeline filling, the bottom pins of the tensor computation subunit continuously output local intermediate result tensors to the distributed cache subsystem.

[0084] The output state of the local intermediate result tensor triggers subsequent chain hardware actions: on the one hand, the local intermediate result tensor is written to the distributed cache subsystem for backup; on the other hand, the output state of the local intermediate result tensor transmits the physical level changes to the request input pin of the dynamic bandwidth allocation module, serving as the core triggering condition for communication topology reconstruction.

[0085] Step S4: Bandwidth Prediction and Allocation Stage; Within the same clock cycle in which the local intermediate result tensor is generated in step S3, a trigger signal arrives at the request input pin of the dynamic bandwidth allocation module along the hardware control cable. Simultaneously, the register set inside the dynamic bandwidth allocation module provides the predicted bandwidth capacity scalar generated in the previous scheduling cycle. And the bandwidth usage of local intermediate result tensor transfer requests actually recorded in the current scheduling cycle. .

[0086] Faced with sudden and highly reactive data surges in large-scale model computations, passive responses can lead to severe network queuing delays. To bridge these time differences, the dynamic bandwidth allocation module incorporates a pure hardware multiply-accumulator tree with an exponentially weighted moving average algorithm. This multiply-accumulator tree performs smoothing and prediction operations on the aforementioned data.

[0087]

[0088] in, Indicating the upcoming [number] The predicted bandwidth allocation capacity scalar required for each scheduling cycle This represents the smoothing weighting factor constant, stored in read-only memory, and has a value of [value missing]. The settings are based on offline calibration results of historical network traffic fluctuation frequencies. Indicates the first The bandwidth usage value of local intermediate result tensor transfer requests actually recorded in each scheduling cycle. Indicates the first The predicted bandwidth allocation capacity scalar generated before each scheduling cycle.

[0089] After a single-cycle calculation by the multiply-accumulator tree, the dynamic bandwidth allocation module outputs a predicted bandwidth allocation command to the internal bus. This predicted bandwidth allocation command is essentially a set of control messages containing the target physical channel number and the reserved time slot length; these control messages are directly transmitted to the intelligent scheduler.

[0090] Step S5: Dynamic Data Stream Scheduling Phase; While receiving the predicted bandwidth allocation command, the intelligent scheduler reads real-time bandwidth status data from the status registers of each routing node in the on-chip interconnect network, including the physical channel congestion flag and the virtual channel remaining depth value. The core task of the intelligent scheduler is to map communication intentions to deterministic physical routing paths.

[0091] The intelligent scheduler internally instantiates a lightweight graph neural network acceleration operator comprising three layers of graph convolution operations and one fully connected layer. The graph neural network uses the physical topology of the on-chip interconnect network as fixed input graph nodes and real-time bandwidth state data as edge weights. The graph convolution operator performs node feature aggregation operations through hardware logic gates.

[0092]

[0093] in, Indicates the first The hidden state feature matrix of the nodes output by the layer graph convolution. This represents a non-linear activation function, implemented in hardware as a ReLU logic gate. This represents the constant of the adjacency matrix of the full mesh topology of an on-chip interconnect network with self-loops. This represents the degree matrix constant corresponding to the adjacency matrix. Indicates the first The hidden state feature matrix of the nodes input to the first layer. The initial value comes from the concatenated vector of the predicted bandwidth allocation instruction and the real-time bandwidth status data. Indicates the first Learnable weight matrix parameters for layer graph convolution.

[0094] The matrix dimension in this embodiment is: The learnable weight matrix parameters are matched to the on-chip full-mesh interconnection network topology and are stored in a dedicated static random access memory within the intelligent scheduler. The values ​​are obtained by offline supervised training and convergence of a massive tensor communication traffic dataset with hundreds of billions of models, and are therefore read-only and cannot be modified during hardware operation. The training process aims to minimize end-to-end transmission latency and avoid deadlocks, with the supervision signal being the actual transmission latency under different paths.

[0095] After three layers of graph convolutional computation and fully connected layer classification, the intelligent scheduler outputs the optimal transmission path identifier. Simultaneously, the intelligent scheduler reads local intermediate result tensors from the distributed caching subsystem and packages them into a tensor of the local result to be transmitted. Both the optimal transmission path identifier and the tensor of the local result to be transmitted are pushed in parallel into the network interface controller of the on-chip interconnect network.

[0096] Step S6: Full Reduce Parallel Communication Stage; After the network interface controller of the on-chip interconnect network receives the local result tensor to be transmitted, the network layer hardware splitting logic is immediately activated, dividing the local result tensor to be transmitted into data packets with a size matching the maximum transmission unit of 1024 bytes. The split data packets are then transmitted to other computing chips along the optimal transmission path identifier's designated physical port.

[0097] This stage executes the full Reduce operation in large model tensor parallelism.

[0098] To completely mask cross-network transmission delays, this invention employs a pipelined double-buffering mechanism. Sending and receiving operations are strictly alternated between buffer group A and buffer group B. The communication time folding effect is defined by the following timing relationship:

[0099]

[0100] in, This represents the total number of clock cycles consumed to complete tensor computation and communication synchronization for a single layer of a large model. This represents the number of clock cycles consumed by the tensor computation subunit to process a single data block. This indicates the number of clock cycles required for the on-chip interconnect network to transmit a single 1024-byte data packet. This represents the total number of data blocks generated by the full Reduce operation. This represents the fixed delay period constant generated during the pipeline startup phase; in this embodiment, it is fixed at [value]. One clock cycle is the inherent fixed hardware delay for inter-core link initialization, packet header parsing, and routing port activation.

[0101] Through a double-buffered transmit / receive masking mechanism. Basically folded into Internally. After step S6 is completed, the complete global reduced tensor sequence is assembled in the pipeline buffer register of the downstream computational core.

[0102] Step S7: Pipeline Bubble Elimination Stage; In a pipelined parallel architecture, the communication arrival time of upstream computing chips is prone to nanosecond-level jitter. If software barrier synchronization is used, the jitter will be directly converted into pipeline bubbles. The dynamic pipeline depth adjustment unit of the downstream computing chip detects the arriving global reduced tensor sequence in real time and extracts the computation speed difference data between adjacent computing chips. The computation speed difference data is calculated by the difference of the timestamps generated by the network interface controller when receiving data packets, after calibration by the global synchronization clock, and its unit is the working clock cycle of this system.

[0103] The dynamic pipeline depth adjustment unit adjusts the effective depth pointer of the internal asynchronous pipeline stage buffer register based on the calculation speed difference data. The asynchronous pipeline stage buffer register is configured with a preset maximum number of depth stages. In this embodiment Fixed configuration is This level effectively covers the maximum clock jitter range during large-model tensor communication. The deep adjustment control logic is as follows:

[0104]

[0105] in, This indicates the target depth level of the adjusted asynchronous pipeline buffer register. This indicates the currently set asynchronous pipeline buffer register level. The minimum discrete step rate constant for depth adjustment is constant. A series of stages, This represents the periodic value indicating the difference in calculation speed between adjacent computational cores obtained from measurements. The physical clock threshold constant representing the tolerance for jitter is set to... One clock cycle, This represents the step decision function, with the function having only one input parameter. The function outputs when the input is greater than zero. Otherwise, the function outputs... .

[0106] By fine-tuning the depth pointer, the dynamic pipeline depth adjustment unit effectively absorbs timing jitter in the hardware buffer pool. The global reduced tensor sequence is passed through asynchronous registers and is finally transformed into a synchronous computation feature map without interruption. This synchronous computation feature map, as the final result of this layer's computation, is directly latched into the input latch of the tensor computation subunit in step S3 through a hardware path, serving as the initial stimulus for the next layer's model inference. This achieves seamless data flow circulation at the hardware level, forming a forced closed loop without software intervention.

[0107] Step S8: The dynamic power management phase runs throughout the entire process; from step S3 to step S7, various arithmetic components and storage queues within the computing chip are in a high-frequency switching state. Multiple hardware probes distributed within the performance monitoring module capture the computing power utilization data of all components in real time with a fixed sampling window of 1024 clock cycles.

[0108] When the computing power utilization rate falls below a preset threshold of 60% due to sparsity in a specific model or routing node allocation issues, the performance monitoring module forcibly triggers a core-level dynamic voltage and frequency adjustment mechanism. Based on the output of the performance monitoring module, the power distribution network control circuit generates and sends out voltage and frequency adjustment signals:

[0109]

[0110] in, This represents the simulated target supply voltage after adjustment by the power distribution network. This represents the minimum retention voltage limit constant for maintaining the state of the flip-flops within the chip without loss. In this embodiment, based on the 7nm Chiplet process, it is set to... , This represents the rated supply voltage limit constant under full-speed operation. In this embodiment, it is set to [value] based on the 7nm Chiplet process. , This indicates the summary results obtained by the performance monitoring module. to The normalized computing power utilization scalar between them.

[0111] The voltage and frequency adjustment signal ensures that during network congestion or idle waiting moments, the computing chip switches to a low-voltage hold-up mode at nanosecond speeds, achieving strong coupling and adaptive voltage drop between dynamic power consumption and communication data flow.

[0112] In one specific implementation plan, the forward propagation process of the first-layer attention mechanism in the Transformer architecture of the large language model is selected for numerical and data flow deduction.

[0113] Bus Start-up: The interface chip receives an initial sequence of 2048 tokens. The memory access acceleration engine sends a read request to the HBM in step S1, and the adaptive prefetch algorithm... Calculate the smoothed prediction hit rate scalar .because Threshold, prefetch window expand The system pre-fetches key-value pairs into key-value cache blocks and extracts the original large model weight matrix into unformatted weight blocks. These two data streams are then separated by programmable address filtering logic and stored in the distributed cache subsystem.

[0114] In step S2, the tensor rearrangement engine captures unformatted weighted data blocks from the distributed cache subsystem. Based on the one-dimensional mapping relationship of the matrix coordinate system, it splits and inverts the row-major physical addresses within one clock cycle, outputting column-major weight tensors. At this point, all original data blocks generated upstream are consumed.

[0115] In step S3, the aforementioned prefetch key-value cache data block and column-major weight tensor are pushed from the distributed cache subsystem into a 64-row by 64-column systolic array. After 64 cycles of pure floating-point operations (multiplication and addition) at pipeline depth, the local intermediate result tensor is output to the distributed cache subsystem from the bottom pin of the array, and the data ready level signal is simultaneously pulled high.

[0116] A high-level signal enters step S4, where the dynamic bandwidth allocation module captures the transmission intent. The theoretical bandwidth request is then calculated using the 2048-word sequence attention mechanism computational scenario in this embodiment. and attenuation parameters The system outputs a predicted bandwidth allocation instruction containing the required bandwidth amount. In step S5, the intelligent scheduler compares the predicted bandwidth allocation instruction with the adjacency matrix of the on-chip network topology. Substituting the graph neural network operator, the hardware solves the optimal transmission path identifier (the best deadlock-free path), and at the same time slices the local intermediate result tensor into the local result tensor to be transmitted.

[0117] The sliced ​​local result tensor to be transmitted enters the full mesh network in step S6. Relying on independently operating buffer groups A and B, the physical packet sending and array arithmetic computation times overlap. (Covered up). Data from each chip is assembled to form a global reduced tensor sequence. In step S7, facing the arrival time difference of 6 working clock cycles due to different network hop counts, the dynamic pipeline depth adjustment unit makes a determination. The clock cycle threshold is used to maintain the current depth register reception, and finally, a synchronous operation feature map is smoothly assembled. The entire data chain is completely connected end to end, and any prediction instructions or identification control quantities generated in the intermediate steps terminate at a clearly specified subsequent hardware node. This synchronous operation feature map then seamlessly flows back to the input of step S3 to start the next layer of computation.

[0118] Example 2: This example is a further optimization based on Example 1. In this example, an adaptive prefetching implementation method different from Example 1 is provided. The difference lies in the prefetch hit smoothing coefficient in step S1 and the adaptive prefetching step. Dynamic online adjustments are made to further improve prefetch hit rate and system throughput.

[0119] In Example 2, the prefetch hit rate evaluation logic inside the prefetch unit, in addition to calculating the smoothed prefetch hit rate scalar... In addition, a smoothing coefficient dynamic adjustment logic circuit was added. This circuit is used in each prefetch evaluation cycle (with... (Same update cycle) Calculate the change in hit rate .based on and Relative to target threshold The relationship is used to dynamically generate the smoothing coefficient for the next period. :

[0120]

[0121] in, The smoothing factor for the next clock cycle will be replaced by the original fixed value written into the configuration register inside the prefetch unit. calculate, The smoothing factor used in the current clock cycle. Adjust the step size factor for the smoothing coefficient, and fix it at 1. , For symbol extraction functions, Output , Output , Output , The same prefetch hit rate activation threshold parameter as in Example 1. , This is a limiting circuit to ensure the result falls within a preset range. Inside, among which , .

[0122] When the hit rate is on an upward trend ( And the current hit rate When the value is above the threshold, increase the value appropriately. Accelerate response and expand the prefetch window more quickly; when the hit rate decreases ( When the value is below the threshold, reduce the value. This mechanism suppresses noise and avoids drastic window jitter, allowing the smoothing coefficient to adapt to the memory access patterns of different layers of the model.

[0123] After adopting the dynamic smoothing coefficient of Example 2, the pre-selected hit rate scalar is obtained. The standard deviation of the fluctuation is compared to the fixed The solution reduces the overshoot phenomenon of prefetch window adjustment by about 30%, and improves the effective bandwidth utilization of video memory by about 15%. The smoothing coefficient dynamic adjustment logic circuit works in conjunction with the original prefetch hit rate evaluation logic and prefetch window dynamic scaling logic, and still outputs prefetch key value cache data blocks and unformatted weight data blocks, without changing the subsequent data flow interface of step S1.

[0124] Example 3: This example is based on the same principles as Example 1, except that the step S3, which is the systolic tensor calculation triggering step, is further optimized by introducing sparse-aware multiplication accumulation control to reduce the power consumption of the systolic array ineffective flipping.

[0125] In large model inference and training, the original large model's weight matrix and activation matrix often exhibit high sparsity. In step S3 of Example 1, each multiplication-accumulation unit in the systolic array performs a complete floating-point multiplication and accumulation operation regardless of whether the weight value is zero, generating unnecessary dynamic power consumption. Example 3 addresses this by appending a sparse mask bitmap to the prefetch key-value cache data block generated in step S1 and adding gating logic to the input of each multiplication-accumulation unit in the systolic array, thus achieving operation skipping triggered by zero-value weights and maintaining pipeline synchronization.

[0126] The specific implementation method is as follows:

[0127] In step S1, when the memory access acceleration engine reads the original large model weight matrix, it simultaneously detects the zero-value state of each weight element and generates a binary sparse mask with the same dimension as the original large model weight matrix. This sparse mask is stored in the distributed cache subsystem along with the prefetch key-value cache data block.

[0128] In step S3, the sparse mask and column-major weight tensor are synchronously injected into the systolic array. Each multiply-accumulate unit receives a weight value and a corresponding mask bit. If the mask bit is 0 (i.e., the weight value is zero), the gated clock circuit within the multiply-accumulate unit shuts off the input latches of the multiplier and adder in that unit and passes the accumulation input directly to the output, equivalent to performing a zero operation without flipping the arithmetic unit; if the mask bit is 1, multiplication and addition are performed normally. To maintain the synchronous pipeline timing of the entire systolic array, all multiply-accumulate units, regardless of whether they are gated, still latch data in their inter-pipeline registers on clock edges, ensuring that the data flow is not interrupted. Thus, the array computation delay remains unchanged, but the power consumption caused by zero-value operations is effectively reduced.

[0129] Let the sparsity of the original large model weight matrix of the current model layer be... (Zero-value ratio), then the theoretical dynamic power consumption reduction ratio of the pulsating array is approximately: For the 50% sparsity common in Transformer models, the measured array power consumption decreases by approximately 40%, and the additional latency introduced by the gating logic is less than 5% of one clock cycle, which does not affect the overall frequency. Furthermore, by utilizing the saved power budget, the array operating frequency can be moderately increased through the power distribution network control circuitry to compensate for potential throughput loss, resulting in an approximately 20% improvement in single-chip energy efficiency.

[0130] This embodiment 3 only changes the operational details inside the pulsating array. The input of step S3 is still the prefetch key-value cache data block and the column-major order weight tensor, and the output is still the local intermediate result tensor, which is fully compatible with the subsequent steps of embodiment 1.

[0131] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0132] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for accelerating large model high-bandwidth AI computing chips based on Chiplet core particle architecture, characterized in that, The method is applied to a heterogeneous physical interconnect network consisting of computing chip clusters, memory chip clusters, and interface chip clusters. The method includes the following steps: Step S1: Reading and Adaptive Prefetching Step; The memory access acceleration engine reads the initial word sequence received by the interface core and the original large model weight matrix stored in the memory core cluster, and outputs the prefetched key-value cache data block and the unformatted weight data block to the distributed cache subsystem through the adaptive prefetching algorithm. Step S2: Data format rearrangement and inversion step; The tensor rearrangement engine extracts the unformatted weight data block from the distributed caching subsystem, performs a one-dimensional address mapping inversion operation from row-major to column-major, and outputs a column-major weight tensor. Step S3: Pulsating tensor calculation trigger step; The pulsating array of the tensor calculation subunit synchronously receives the prefetched key-value cache data block and the column main order weight tensor from the distributed cache subsystem, performs point accumulation and addition operations, outputs the local intermediate result tensor to the distributed cache subsystem, and generates a transmission request trigger signal; Step S4: Bandwidth prediction and allocation step; The dynamic bandwidth allocation module receives the transmission request trigger signal, calculates and outputs the predicted bandwidth allocation instruction based on the requested bandwidth occupancy value of the current scheduling period and the predicted bandwidth capacity scalar generated in the previous scheduling period, using the exponential weighted moving average algorithm. Step S5: Dynamic data stream scheduling steps; The intelligent scheduler receives the predicted bandwidth allocation instruction and the real-time bandwidth status data of the on-chip interconnect network, performs optimal path mapping based on graph neural network, outputs the optimal transmission path identifier, and reads the local intermediate result tensor from the distributed cache subsystem and encapsulates it into a local result tensor to be transmitted. Step S6: Full Reduce Parallel Communication Step; The on-chip interconnect network identifies the optimal transmission path, and the pipeline-level double buffering mechanism alternately sends and receives the local result tensor to be transmitted, and the global reduced tensor sequence is generated by assembling the downstream computing chips. Step S7: Pipeline bubble elimination step; The dynamic pipeline depth adjustment unit of the downstream computing chip receives the global reduced tensor sequence, dynamically adjusts the depth of the asynchronous pipeline buffer register according to the extracted computing speed difference data of adjacent computing chips, and transmits the synchronous operation feature map, wherein the synchronous operation feature map is seamlessly used as the input tensor for the next layer model calculation, forming a hardware data closed loop.

2. The large model high-bandwidth AI computing power chip acceleration method based on the Chiplet core particle architecture according to claim 1, characterized in that, The method also includes step S8, which runs through the entire process: dynamic power consumption management step; the performance monitoring module captures computing power utilization data at each stage in real time, and when the computing power utilization data is lower than a preset threshold, it outputs a voltage frequency adjustment signal to the power distribution network control circuit.

3. The method for accelerating large-model, high-bandwidth AI computing power chips based on the Chiplet chip architecture according to claim 1, characterized in that, In said step S1, the adaptive prefetching algorithm is based on a smoothed prefetch hit rate scalar The prefetch data block window byte length is dynamically scaled The prefetch window byte length is constrained between a pre-set minimum value and a maximum value with an iterative relationship satisfying: wherein, is a pre-fetch window single adjustment reference step constant, is a pre-set pre-fetch hit rate activation threshold parameter.

4. The method for accelerating large-model, high-bandwidth AI computing power chips based on the Chiplet chip architecture according to claim 3, characterized in that, the smoothed prefetch hit rate scalar by the instantaneous hit indicator scalar with a predetermined smoothing coefficient is calculated, and the calculation formula is: wherein, represents a temporal hit indicator scalar that is set to one for a hit and zero for a miss when the systolic array accesses the prefetch cache space.

5. The method for accelerating large-model, high-bandwidth AI computing power chips based on the Chiplet chip architecture according to claim 1, characterized in that, In the step S2, the one-dimensional address mapping inversion operation converts the row-major physical address into the column-major one-dimensional target physical address The conversion logic is: wherein, is an output video memory segment first address constant, is a column coordinate index variable, is a row total constant, is a row coordinate index variable.

6. The method for accelerating large-model, high-bandwidth AI computing power chips based on the Chiplet chip architecture according to claim 1, characterized in that, In the step S4, the dynamic bandwidth allocation module employs an exponential weighted moving average algorithm to calculate the bandwidth capacity scalar in the predicted bandwidth allocation instruction satisfies: wherein is a constant for the smoothing weighting factor.

7. The large model high-bandwidth AI computing power chip acceleration method based on the Chiplet core particle architecture according to claim 1, characterized in that, In step S5, the intelligent scheduler comprises a graph neural network with three layers of graph convolution operations and one fully connected layer, and its node feature aggregation operation formula is as follows: wherein, is a ReLU activation function, is a constant of the full Mesh topology adjacency matrix of the on-chip interconnect network with self-loops, is a constant of the degree matrix, is a constant of the degree matrix, is a constant of the degree matrix, 8. The method for accelerating large-model, high-bandwidth AI computing power chips based on the Chiplet chip architecture according to claim 1, characterized in that, In step S6, the tensor of the local result to be transmitted is divided into data packets of a size matching the maximum transmission unit of 1024 bytes.

9. A large-model, high-bandwidth AI computing chip based on the Chiplet chip architecture, used to implement the acceleration method for the large-model, high-bandwidth AI computing chip based on the Chiplet chip architecture as described in any one of claims 1 to 8, characterized in that, include: The distributed caching subsystem is used to cache the prefetched key-value cache data block and the unformatted weight data block output from step S1, as well as the local intermediate result tensor output from step S3. The video memory access acceleration engine is used to read the initial word sequence received by the interface core and the original large model weight matrix stored in the video memory core cluster. It outputs the prefetch key-value cache data block and the unformatted weight data block to the distributed cache subsystem through an adaptive prefetch algorithm. The tensor rearrangement engine is used to extract the unformatted weighted data block from the distributed caching subsystem, perform a one-dimensional address mapping inversion operation from row-major to column-major, and output a column-major weight tensor. Tensor computation subunit is used to synchronously receive the prefetch key-value cache data block and the column main order weight tensor from the distributed cache subsystem through its pulsating array, perform point accumulation and addition operations, output local intermediate result tensor to the distributed cache subsystem, and generate a transmission request trigger signal; The system includes a dynamic bandwidth allocation module and an intelligent scheduler. The dynamic bandwidth allocation module receives the transmission request trigger signal and calculates and outputs a predicted bandwidth allocation instruction based on the requested bandwidth occupancy value of the current scheduling period and the predicted bandwidth capacity scalar generated in the previous scheduling period, using an exponentially weighted moving average algorithm. The intelligent scheduler is used to receive the predicted bandwidth allocation instruction and the real-time bandwidth status data of the on-chip interconnection network, perform optimal path mapping based on graph neural network, output the optimal transmission path identifier, and read the local intermediate result tensor from the distributed cache subsystem and encapsulate it into a local result tensor to be transmitted. An on-chip interconnect network configured with a double buffering mechanism and a full mesh topology is used to identify the optimal transmission path. The pipeline-level double buffering mechanism is used to alternately send and receive the local result tensor to be transmitted, and the global reduced tensor sequence is generated by assembling the downstream computing particles. The dynamic pipeline depth adjustment unit is used to receive the global reduced tensor sequence, dynamically adjust the depth of the asynchronous pipeline buffer register based on the extracted data of the difference in computing speed between adjacent computing particles, and transmit out the synchronous operation feature map.

10. The large-model, high-bandwidth AI computing chip based on the Chiplet chip architecture according to claim 9, characterized in that, It also includes a performance monitoring module and a power distribution network control circuit, which are used to capture computing power utilization data at each stage in real time. When the computing power utilization data is lower than a preset threshold, a voltage frequency adjustment signal is output to the power distribution network control circuit.

Images