Systems and methods for near memory compute

Abstract

Systems and methods for accelerating computations are described. An example system can include a semiconductor system including: a first near memory compute package including at least a first stacked memory device communicatively coupled to at least a first processing element (PE) disposed in a layer of the first near memory compute package; a second near memory compute package comprising at least a second stacked memory device communicatively coupled to at least a second PE disposed in the layer, wherein the second near memory compute package is communicatively coupled to the near memory compute package die via a interconnect, and wherein at least one of the first PE and the second PE are configured to accelerate at least one matrix multiplication operation using first data stored on the first stacked memory device.

Description

SYSTEMS AND METHODS FOR NEAR MEMORY COMPUTERELATED APPLICATIONS

[0001] The present application claims the benefit of U.S. provisional application Serial No. 63 / 666,105 entitled "ADVANCED HIGH BANDWIDTH MEMORY (A-HBM)" filed on 6 / 28 / 2024 and U.S. provisional application Serial No. 64 / 760,905 entitled "ADVANCED HIGH BANDWIDTH MEMORY (A-HBM)" and filed on 2 / 20 / 2025, which is incorporated herein by reference in its entirety.FIELD

[0002] The disclosure relate generally to memory and computing systems and more particularly to systems and methods for near memory compute using high bandwidth memory.BACKGROUND

[0003] The present background section is intended to provide context only, and the disclosure of any concept in this section does not constitute an admission that said concept is prior art.

[0004] Advancements in Large Language Models (LLMs) have led to increased applications in Natural Language Processing (NLP) tasks, such as language understanding, generation, and translation. However, the increasing complexity and size of LLMs have created significant challenges for efficient inference. In particular, some computing architectures, such as central processing units (CPUs) and graphics processing units (GPUs), may be ill-suited for the massive matrix multiplications and high memory bandwidth requirements of LLM inference. The relatively large number of parameters in these models is increasingly mandatingrelatively large amounts of memory storage and bandwidth, making it difficult to deploy them in resource-constrained environments. For example, inference with some LLMs (e.g., GPT-3) on a GPU can require a relatively significant amount of memory (e.g., approximately 350 GB) and can consume a relatively large amount of power (e.g., over 5600 watts of power). This highlights the need for more efficient and scalable solutions that can address the growing demands of LLM inference.

[0005] The above information disclosed in this background section is only for enhancement of understanding of the background of the disclosure and therefore it may contain information that does not constitute prior art.SUMMARY

[0006] In various embodiments, described herein include systems, methods, and apparatuses for accelerating and / or offloading operations and / or computations to a near memory compute system (e.g., an advanced high bandwidth memory (AHBM) system).

[0007] In various embodiments, a system is described. In some respects, the system can include a semiconductor system (e.g., a package including at least one near memory compute system such as an AHBM system). In particular, the system can include a first near memory compute package which can include at least a first stacked memory device that is communicatively coupled to at least a first processing element (PE) disposed in a layer of the near memory compute package. The system can further include a second near memory compute package that includes at least a second stacked memory device that is communicatively coupled to at least a second PE disposed in the layer, wherein the second near memory compute package is communicatively coupled to the first near memory compute package via a interconnect. Further, at least one of the first PE and the second PE can be configured to accelerate at least one matrix multiplication operation using first data stored on the first stacked memory device. In some respects, the system can further include a network interface (e.g., a network interfacecard, NIC) that can be configured to connect the semiconductor system to a switch (e.g., an Ethernet switch) over a network connector and transmit information based on the matrix multiplication operation.

[0008] In one aspect, the network connector can include at least one optical connector of a first type that transmits at least a portion of the information at a first rate. The system can further include a processor that is connected to the network interface and to the semiconductor system for managing the semiconductor system. The system can further include a storage device connected to the processor for storing at least a portion of the information.

[0009] In some aspects, the semiconductor system can further include a stacked memory controller configured to manage a transfer of second data between at least one of the first stacked memory device and the first PE. In some aspects, the semiconductor system can include a silicon interposer that communicatively connects the first near memory compute package and the second near memory compute package. In an aspect, the interconnect can include a die-to- die interconnect configured to operate according to an interconnect protocol (e.g., universal chiplet interconnect express, UCIe). In some respects, the semiconductor system can include the first near memory compute package being further connected to at least one of a first connector configured to operate with a first protocol (e.g., peripheral component interconnect express, PCIe) or a second connector configured to operate with a second protocol (e.g., ultra accelerator link, UALink), wherein the first protocol and the second protocol are different.

[0010] In an aspect, the first connector can be configured to connect the first near memory compute package of the semiconductor system to a host over a third connector configured to operate with the first protocol. In another respect, the second connector can be configured to connect the second near memory compute package of the semiconductor system to a third near memory compute package of another semiconductor system over a third connector that is configured to operate with the second protocol.

[0011] In an embodiment, the first near memory compute package can include a testing device configured to test at least one of the first PE or the first stacked memory device. The semiconductor system can further include a management processor configured to manage a kernel execution on the semiconductor system by configuring at least one parameter associated with at least one of the first PE or the second PE. In an aspect, the management processor can be configured to modify a clock speed of at least one of the first PE based on a temperature associated with at least one portion of the first near memory compute package. In another aspect, the first PE can include a scheduler that schedules operations associated with the matrix multiplication operation using one or more of a tightly coupled memory, a functional unit, a tensor processing element, or a math engine.

[0012] A system of one or more computing devices such as the near memory compute system can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

[0013] In some aspects, a semiconductor device is described. The semiconductor device can include a first stacked memory device for storing first data associated with at least one matrix multiplication operation. The semiconductor device can further include a first PE communicatively coupled to the first stacked memory over a channel of a network on chip (NOC) interconnect and via a first stacked memory device controller. In some respects, the first stacked memory device controller is configured to manage transfer of first data between the first stacked memory device and the first PE. Moreover, the first PE can be configured to execute the matrix multiplication operation using the first data.

[0014] In some respects, the semiconductor device can include a shared memory configured to store a portion of the first data. In other aspects, the semiconductor device can further comprise a management processor configured to manage the execution of the matrix multiplication operation by configuring at least one parameter associated with the first PE. In an aspect, the first PE can include a scheduler that schedules operations associated with the one matrix multiplication operation using one or more of a tightly coupled memory, a functional unit, a tensor processing element, or a math engine.

[0015] In various respects, a method for accelerating computations is described. The method can include receiving, at a management processor, a request to execute a kernel for matrix multiplication using at least a PE of a near memory compute package. The method can further include obtaining, by the management processor, input data and the kernel from at least one of a stacked memory device or a shared memory of the near memory compute package. The method can include executing, by the PE, an operation associated with the kernel using the input data to obtain a result, and storing the result in a second memory device.

[0016] In some aspects, the second memory device can include tightly coupled memory (TCM) and wherein the method further comprises flushing the TCM. In other respects, the method further includes assigning an identifier to the PE, and the execution of the operation is based on the identifier. In some embodiments, the method can include executing at least a first portion of the operation on one or more first channels associated with the PE. In some respects, the method further includes executing at least a second portion of the operation on one or more second channels associated with the PE.

[0017] Other embodiments of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods.

[0018] Implementations of the described techniques may include hardware, a method or process, or a computer tangible medium.BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above-mentioned aspects and other aspects of the present techniques will be better understood when the present application is read in view of the following figures in which like numbers indicate similar or identical elements. Further, the drawings provided herein are for purpose of illustrating certain embodiments only; other embodiments, which may not be explicitly illustrated, are not excluded from the scope of this disclosure.

[0020] These and other features and advantages of the present disclosure will be appreciated and understood with reference to the specification, claims, and appended drawings wherein:

[0021] FIG. 1 shows an example of an overall system architecture for near memory compute using an advanced high bandwidth memory (A-HBM), in accordance with embodiments of the disclosure.

[0022] FIG. 2 shows high-level architectures associated with an A-HBM SiP for the disclosed systems, in accordance with embodiments of the disclosure.

[0023] FIG. 3 shows a diagram of an semiconductor system including a A-HBM SiP for the disclosed systems, in accordance with embodiments of the disclosure.

[0024] FIG. 4 shows a block diagram of an A-HBM base die for the disclosed systems, in accordance with embodiments of the disclosure.

[0025] FIG. 5 shows a cluster of processing elements (PEs) for the disclosed systems, in accordance with embodiments of the disclosure.

[0026] FIGs. 6A and 6B shows an example of control flow for the disclosed systems, in accordance with embodiments of the disclosure.

[0027] FIG. 7 shows an example of a fine-grained Matmul kernel for communication between a CPU and PEs for the disclosed systems, in accordance with embodiments of the disclosure.

[0028] FIG. 8 illustrates example matrix multiplication and tiling operations for the disclosed systems, in accordance with embodiments of the disclosure.

[0029] FIG. 9 shows a single kernel-based execution example for the disclosed systems, in accordance with embodiments of the disclosure.

[0030] FIG. 10 illustrates a fused graph mode execution example for the disclosed systems, in accordance with embodiments of the disclosure.

[0031] FIG. 11 describes parallel access to HBM by PEs for the disclosed systems, in accordance with embodiments of the disclosure.

[0032] FIGs. 12A and 12B depict data movement between dies for the disclosed systems, in accordance with embodiments of the disclosure.

[0033] FIG. 13 shows a microprocessor for PE kernel execution for the disclosed systems, in accordance with embodiments of the disclosure.

[0034] FIG. 14 illustrates a management CPU (M-CPU) subsystem for the disclosed systems, in accordance with embodiments of the disclosure.

[0035] FIG. 15 shows a cluster direct memory access (DMA) diagram for the disclosed systems, in accordance with embodiments of the disclosure.

[0036] FIG. 16 depicts a block diagram of a 3D-DRAM physical layer interface (PHY) for the disclosed systems, in accordance with embodiments of the disclosure.

[0037] FIG. 17 illustrates a Digital Front-end Interface to through-silicon vias (DFI-to- TSV) read timing diagram for the disclosed systems, according to an embodiment of the disclosure.

[0038] FIG. 18 illustrates a DFI-to-TS V write timing diagram for the disclosed systems, according to an embodiment of the disclosure.

[0039] FIG. 19 provides a general overview of a DRAM die stack with channels for the disclosed systems, in accordance with embodiments of the disclosure.

[0040] FIG. 20 depicts an HBM (e.g., HBM4) subsystem for use in connection with the disclosed systems, in accordance with embodiments of the disclosure.

[0041] FIG. 21 shows a direct access (DA) ball signal path at a SIP level and DA pads at a Chip-on-Wafer (CoW) level for use in connection with the disclosed systems, according to an embodiment of the disclosure.

[0042] FIG. 22 illustrates HBM4 DA Ball allocation for multiple HBM4 DRAM devices for the disclosed systems, in accordance with embodiments of the disclosure.

[0043] FIG. 23 describes DA pBump (ball) and DA signal path for the disclosed systems, according to an embodiment of the disclosure.

[0044] FIG. 24 outlines A-HBM test coverage scenario for the disclosed systems, in accordance with embodiments of the disclosure.

[0045] FIG. 25 depicts a die-to-die interconnect for the disclosed systems, according to an embodiment of the disclosure.

[0046] FIG. 26 explains protocol (e.g., UCIe) layers and functionalities for use by the disclosed systems, in accordance with embodiments of the disclosure.

[0047] FIG. 27 shows signal and power TSVs for the disclosed systems, according to an embodiment of the disclosure.

[0048] FIG. 28 illustrates a middle TSV for the disclosed systems, in accordance with embodiments of the disclosure.

[0049] FIG. 29 depicts an A-HBM SiP package with a silicon interposer for the disclosed systems, according to an embodiment of the disclosure.

[0050] FIG. 30 shows an A-HBM package with a silicon interposer for the disclosed systems, in accordance with embodiments of the disclosure.

[0051] FIG. 31 illustrates various configurations of a SiP used with the disclosed systems which can be implemented based on performance needs, according to an embodiment of the disclosure.

[0052] FIG. 32A and 32B depicts an A-HBM system for the disclosed systems, in accordance with embodiments of the disclosure.

[0053] FIG. 33 provides a fully connected interconnect topology example for the disclosed systems, according to an embodiment of the disclosure.

[0054] FIG. 34 shows a combined fully connected and hybrid cube mesh interconnect topology example for the disclosed systems, in accordance to embodiments of the disclosure.

[0055] FIGs. 35 A and 35B illustrates a 2D torus interconnect topology example for the disclosed systems, according to an embodiment of the disclosure.

[0056] FIG. 36 depicts static and dynamic operations of the disclosed systems, in accordance with embodiments of the disclosure.

[0057] FIG. 37 shows a low-level API user interactions with the disclosed systems, according to an embodiment of the disclosure.

[0058] FIG. 38 illustrates a kernel and thread hierarchy for the disclosed systems, in accordance with embodiments of the disclosure.

[0059] FIG. 39 provides a memory hierarchy diagram for the disclosed systems, according to an embodiment of the disclosure.

[0060] FIG. 40 describes matrix split between HBMs, PEs, and channels that can be used to conduct operations by the disclosed systems, in accordance with embodiments of the disclosure.

[0061] FIG. 44 illustrates GEMM modeling with blocks and tiles of the disclosed systems, according to an embodiment of the disclosure.

[0062] FIG. 45 depicts a flash attention use case for the disclosed systems, in accordance with embodiments of the disclosure.

[0063] FIG. 46 shows an interaction diagram with a simplified self-attention calculation for a pre-fill stage for the disclosed systems, in accordance with embodiments of the disclosure.

[0064] FIG. 47 shows an A-HBM SiP local reduction mechanism for the disclosed systems, in accordance with embodiments of the disclosure.

[0065] FIG. 48 illustrates a local broadcast mechanism for the disclosed systems, according to an embodiment of the disclosure.

[0066] FIG. 49 depicts a worker A-HBM kernel pseudocode for the disclosed systems, in accordance with embodiments of the disclosure.

[0067] FIG. 50 describes an example of a pipelined local reduce mechanism for the disclosed systems, in accordance with embodiments of the disclosure.

[0068] FIG. 51 provides an example master A-HBM kernel pseudocode for the disclosed systems, in accordance with embodiments of the disclosure.

[0069] FIG. 52 is an illustration of an exemplary operational flow illustrating example operations for performing computations using the disclosed system, in accordance with example embodiments of the disclosure.

[0070] While the present techniques are susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described. The drawings may not be to scale. It should be understood, however, that the drawings and detailed description thereto are not intended to limit the present techniques to the particular form disclosed, but to the contrary, the intention is to cover allmodifications, equivalents, and alternatives falling within the spirit and scope of the present techniques as defined by the appended claims.DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

[0071] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0072] Embodiments of the present disclosure relate to semiconductor systems and methods for accelerating computations, particularly matrix multiplication operations but also other general computations including mathematical operations such as trigonometric operations, Exponential and Logarithmic Functions, fast Fourier transform operations, BLAS (Basic Linear Algebra Subprograms), and the like. In an aspect, a semiconductor system is disclosed, comprising a first near memory compute package (also referred to herein as an advanced high bandwidth memory (AHBM) package, device, and / or system) with a stacked memory device (also referred to herein as an HBM) and one or more processing elements (PEs). The semiconductor system can further include a second near memory compute package with another stacked memory device and PEs, which can be communicatively coupled to the first near memory compute package via an interconnect. At least one of the PEs is configured to accelerate matrix multiplication operations using data stored on the stacked memory device. The system may further comprise a stacked memory controller (e.g., an AHBM controller), a silicon interposer, and connectors configured to operate with different protocols (e.g., PCIe, UALink, UCIe, and the like discussed further herein). The system may also include a management processor that manages kernel execution, modifies parameters based on system conditions (e.g., clock speeds based on temperature), and schedules operations using variouscomponents of the semiconductor system such as tightly coupled memory, functional units, tensor processing elements, or math engines.

[0073] The semiconductor system may be configured to execute matrix multiplication operations using data stored on the stacked memory device, with the PE comprising a scheduler that schedules operations associated with the matrix multiplication operation. The system may also include shared memory, management processors, and testing devices to test PEs or stacked memory devices. Methods for accelerating computations are also disclosed, including receiving requests to execute kernels for matrix multiplication, obtaining input data and kernels from stacked memory devices or shared memory, executing operations associated with the kernel using input data to obtain results; storing results in second-level storage; assigning identifiers; flushing tightly coupled memories (TCMs); and executing portions of an operation across multiple channels.

[0074] Overall, there are many advantages to the disclosed systems including improved performance for matrix multiplication operations through near-memory computing architectures; increased flexibility through support for multiple protocols (e.g., PCIe; UALink); enhanced thermal management via dynamic clock speed adjustments based upon monitored temperatures within certain parts of chipsets housing numerous cores / elements; capability of running demanding mathematical models requiring considerable computational resources efficiently allocated according to application needs dynamically assessed during runtime phases thereof, which overall ensures optimal throughput efficiency gains feasible under diverse operating conditions encountered during real-world applications.

[0075] Various embodiments of the present disclosure now will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments are shown. Indeed, the disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicable legal requirements. The term “or” is used herein in both the alternative and conjunctive sense, unless otherwise indicated. The terms “illustrative” and “example” are used to be examples with no indication of quality level. Like numbers refer to like elements throughout. Arrows in each of the figures depict bi-directional data flow and / or bi-directional data flow capabilities. The terms “path,” “pathway” and “route” are used interchangeably herein.

[0076] Embodiments of the present disclosure may be implemented in various ways, including as computer program products that comprise articles of manufacture. A computer program product may include a non-transitory computer-readable storage medium storing applications, programs, program components, scripts, source code, program code, object code, byte code, compiled code, interpreted code, machine code, executable instructions, and / or the like (also referred to herein as executable instructions, instructions for execution, computer program products, program code, and / or similar terms used herein interchangeably). Such non- transitory computer-readable storage media include all computer-readable media (including volatile and non-volatile media).

[0077] In one embodiment, a non-volatile computer-readable storage medium may include a floppy disk, flexible disk, hard disk, solid-state storage (SSS) (for example a solid- state drive (SSD)), solid state card (SSC), solid state module (SSM), enterprise flash drive, magnetic tape, or any other non-transitory magnetic medium, and / or the like. A non-volatile computer-readable storage medium may also include a punch card, paper tape, optical mark sheet (or any other physical medium with patterns of holes or other optically recognizable indicia), compact disc read only memory (CD-ROM), compact disc-rewritable (CD-RW), digital versatile disc (DVD), Blu-ray disc (BD), any other non-transitory optical medium, and / or the like. Such a non-volatile computer-readable storage medium may also include readonly memory (ROM), programmable read-only memory (PROM), erasable programmableread-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory (for example Serial, NAND, NOR, and / or the like), multimedia memory cards (MMC), secure digital (SD) memory cards, SmartMedia cards, CompactFlash (CF) cards, Memory Sticks, and / or the like. Further, a non-volatile computer-readable storage medium may also include conductive-bridging random access memory (CBRAM), phasechange random access memory (PRAM), ferroelectric random-access memory (FeRAM), nonvolatile random-access memory (NVRAM), magnetoresistive random-access memory (MRAM), resistive random-access memory (RRAM), Silicon-Oxide-Nitride-Oxide-Silicon memory (SONOS), floating junction gate random access memory (FJG RAM), Millipede memory, racetrack memory, and / or the like.

[0078] In one embodiment, a volatile computer-readable storage medium may include random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), fast page mode dynamic random access memory (FPM DRAM), extended data-out dynamic random access memory (EDO DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), double data rate type two synchronous dynamic random access memory (DDR2 SDRAM), double data rate type three synchronous dynamic random access memory (DDR3 SDRAM), Rambus dynamic random access memory (RDRAM), Twin Transistor RAM (TTRAM), Thyristor RAM (T-RAM), Zero-capacitor (Z-RAM), Rambus inline memory component (RIMM), dual in-line memory component (DIMM), single in-line memory component (SIMM), video random access memory (VRAM), cache memory (including various levels), flash memory, register memory, and / or the like. It will be appreciated that where embodiments are described to use a computer-readable storage medium, other types of computer-readable storage media may be substituted for or used in addition to the computer-readable storage media described above.

[0079] As should be appreciated, various embodiments of the present disclosure may also be implemented as methods, apparatus, systems, computing devices, computing entities, and / or the like. As such, embodiments of the present disclosure may take the form of an apparatus, system, computing device, computing entity, and / or the like executing instructions stored on a computer-readable storage medium to perform certain steps or operations. Thus, embodiments of the present disclosure may also take the form of an entirely hardware embodiment, an entirely computer program product embodiment, and / or an embodiment that comprises combination of computer program products and hardware performing certain steps or operations.

[0080] Embodiments of the present disclosure are described below with reference to block diagrams and flowchart illustrations. Thus, it should be understood that each block of the block diagrams and flowchart illustrations may be implemented in the form of a computer program product, an entirely hardware embodiment, a combination of hardware and computer program products, and / or apparatus, systems, computing devices, computing entities, and / or the like carrying out instructions, operations, steps, and similar words used interchangeably (for example the executable instructions, instructions for execution, program code, and / or the like) on a computer-readable storage medium for execution. For example, retrieval, loading, and execution of code may be performed sequentially such that one instruction is retrieved, loaded, and executed at a time. In some example embodiments, retrieval, loading, and / or execution may be performed in parallel such that multiple instructions are retrieved, loaded, and / or executed together. Thus, such embodiments can produce specifically-configured machines performing the steps or operations specified in the block diagrams and flowchart illustrations. Accordingly, the block diagrams and flowchart illustrations support various combinations of embodiments for performing the specified instructions, operations, or steps.

[0081] Terminology

[0082] As used herein, there are various terms listed below that are given some context and definition, although it is understood not to be limited to the descriptions provided below.

[0083] An activation function can refer to a function in a neural network that introduces non-linear properties to the model, allowing it to learn complex patterns. Some example activation functions used in machine learning models can include ReLU (Rectified Linear Unit), GELU (Gaussian Error Linear Unit), and SwiGLU (Switchable Gaussian Linear Unit).

[0084] A batch can refer to a set of data points processed together in one iteration of training or inference in machine learning. Batching can aid the disclosed systems in efficiently managing computation resources.

[0085] Context length can refer to the number of tokens a model can consider at one time from the input data. It defines the extent of previous information that models associated with the disclosed systems use to make predictions.

[0086] A decode stage can refer to a stage of a LLM (large language model) inference where the model generates output tokens autoregressively one at a time, until a predetermined stopping criterion is met.

[0087] Finetuning can refer to the process of adjusting a pre-trained model on a new, typically smaller, dataset to specialize its knowledge for specific tasks or domains.

[0088] Flash attention can refer to a memory-efficient attention mechanism that improves the speed of training and inference.

[0089] GELU (Gaussian error linear unit) can refer to a type of activation function that is used to add non-linearity to the model predictions, often used in transformers.

[0090] GEMM (general matrix multiply) can refer to an operation in many Al model computations involving the multiplication of matrices.

[0091] GEMV (general matrix-vector multiply) can refer to a basic operation used inAl model for multiplying a matrix with a vector.

[0092] GenAI (Generative Al) can refer to a type of artificial intelligence designed to generate new content, from text to images, by learning from vast amounts of data.

[0093] GPT (generative pre-trained transformer) can refer to a type of large-scale language model designed to generate human-like text based on the input it receives.

[0094] Inference can refer to the process by which a trained model makes predictions on new data.

[0095] KV Cache (key -value cache) can refer to a caching mechanism that stores keyvalue pairs for quick retrieval, used to enhance the performance of large-scale models like LLMs.

[0096] An LLM (large language model) can refer to a type of Al model trained on extensive text datasets capable of understanding and generating human language.

[0097] MLLM (multimodal large language model) can refer to an Al model that processes and generates predictions based on multiple forms of data, such as text, sound, and images.

[0098] Model deployment can refer to the process of integrating a trained Al model into an existing production environment where it can start providing predictions on real- world data.

[0099] Model serving can refer to the process of deploying a trained Al model into a production environment where it can perform inference tasks based on new input data, typically to provide real-time responses in applications.

[0100] MoE (Mixture of experts) can refer to a model architecture that involves multiple specialist models or experts that are selectively activated based on the input they receive.

[0101] Multimodal models can refer to Al models that are capable of processing and understanding multiple types of data input, such as text, audio, and visual data.

[0102] NLP (natural language processing) can refer to the branch of artificial intelligence focused on giving machines the ability to understand, interpret, and generate human language.

[0103] Prefill can refer to an initial stage of LLM inference where the model processes the input tokens to compute the intermediate states (keys and values), which are used to generate the first new token.

[0104] Pruning can refer to a technique in neural network training to reduce the size of the model by removing neurons that contribute little to the output predictions.

[0105] Quantization can refer to the process of reducing the precision of the numbers used to represent model weights, which can decrease the model size and speed up inference.

[0106] ReLU (rectified linear unit) can refer to a popular activation function in neural networks, especially effective in hidden layers.

[0107] Sequence Length and Context Length are related concepts, whereby sequence length can refer to the maximum number of tokens (words, subwords, or characters) that a model can process in a single input. Context length, often used interchangeably with context window, defines how much of that input sequence the model can remember or utilize when generating its output.

[0108] Softmax can refer to a mathematical function commonly used in the output layer of neural network classifiers to convert logits — raw prediction values — into probabilities that sum to one.

[0109] Sparsity can refer to the inclusion of many zero-valued elements in a data structure.

[0110] SwiGLU (Sigmoid-Weighted Linear Unit) can refer to an activation function that modifies the Gated Linear Unit (GLU) by incorporating the sigmoid function to manage the weighting of inputs, used to introduce non-linearity in neural networks.

[0111] A token can refer to the smallest unit of data processed by a language model, which can be a word, part of a word, or even punctuation.

[0112] Training can refer to the process of teaching a model to perform a specific task by adjusting its weights based on feedback from a training dataset.

[0113] A-HBM (Advance HBM) and AHBM (also referred to herein as a near memory compute package, device, die, and / or chip) can refer to a type of custom HBM with computational functionality in at least one of the layers of the device (e.g., the base layer / die), which can, in an example include PIM (Processing-in-Memory) technology, which embeds compute functions such as for Al LLM inference in a base / buffer die.

[0114] AALC (air-assisted liquid cooling) can refer to a cooling technology that uses a combination of air and liquid to remove heat from electronic components, offering improved cooling efficiency compared to traditional air cooling.

[0115] Air Cooling can refer to a method of dissipating heat from electronic devices and systems by using air flow. It is one of the most common cooling techniques in computing hardware.

[0116] Buffer die can refer to a base die of HBM stack. In some respects, the buffer die can include a high speed interconnect PHY or D2D with fine pitch micro-bump and test logical circuitry for testing, I / O testing, loopback testing of the DRAM core die at the memory cell level.

[0117] CoW (chip on wafer) can refer to a method in semiconductor manufacturing where the chip is bonded to the wafer level, often used in the production of advanced packaging technologies like 3D integrated circuits (ICs).

[0118] DA can refer to direct access for HBM core die testing that is, directly interact with the HBM memory through an AHBM interface, bypassing the limitations of traditional memory bus architectures and / or processors.

[0119] D2D (die-to-die) can refer to the interconnect technology used for communication between different dies.

[0120] DFI (DDR PHY interface) can refer to Part of a Joint Electron Device Engineering Council (JEDEC) standard, this is the physical interface layer for DDR SDRAM memories.

[0121] PIM (processing-in-memory) can refer to a technology that integrates processing capabilities within the memory.

[0122] FWC (facility water cooling) can refer to an advanced cooling technology that uses liquid immersion or other extensive methods to manage heat in high-performance computing systems, allowing for higher power densities and efficiency.

[0123] High bandwidth memory (HBM) can refer to a type of 3D stacked memory architecture that offers higher bandwidth and lower power consumption compared to traditional RAM types. It is often used in high-performance computing and graphics processing.

[0124] IEEE 1500 can refer to a standard test interface for DRAM using a serial protocol.

[0125] MBIST (memory built-in self-test) can refer to an architecture embedded within memory chips to enable self-testing for detecting and diagnosing manufacturing faults and errors in memory cells.

[0126] NPU (neural processing unit) can refer to a specialized hardware unit designed for efficiently processing neural network computations, which can also be referred to herein,based on context, as a processing element (PE) and / or components associated with the PE (e.g., scratchpad memory, controller, etc.).

[0127] SM (streaming multiprocessor) can refer to the basic unit of computation in some GPUs. An SM includes multiple cores that execute instructions in parallel, significantly accelerating computing tasks.

[0128] 0AM (Open Compute Project (OCP) accelerator module) can refer to an industry standard defines an open-hardware compute accelerator module form factor.

[0129] Open Rack V3 can refer to the third version of a standard for rack design, developed under the OCP, that focuses on enhancing the physical infrastructure for data centers in terms of efficiency, modularity, and scalability.

[0130] PCI (peripheral component interconnect) can refer to a standard for connecting peripheral devices to a computer.

[0131] PE (processing element) can refer to an Al accelerator compute processor element, which include tensor, math engine and micro-processor for software kernel processing, which can be integrated in the AHBM system.

[0132] Power density can refer to a measurement of the amount of power (energy per time) generated per unit volume of an electronic device, which affects cooling and performance characteristics.

[0133] SiP (system in package) can refer to an integration of multiple electronic components into a single chip package, which can include processors, memory, and other subsystems, offering a compact and efficient solution.

[0134] TSV (through silicon via) can refer to a technology used in creating high- performance integrated circuits by vertically interconnecting stacked silicon wafer or dies.

[0135] UALink (ultra-accelerator link) can refer to an open-standard interconnect for Al accelerators. It enables high-speed, low-latency interconnection for datacenter-grade Al and HPC accelerators.

[0136] UCIe (universal chiplet interconnect express) can refer to a standard for interconnecting chiplets within a package to create a more modular and scalable architecture, enhancing data transfer rates and flexibility in system design.

[0137] A / B testing can refer to an experiment where two or more variants of a page are shown to users at random, and statistical analysis is used to determine which variation performs better for a given conversion goal.

[0138] A compiler can refer to, in the context of Al, a software tool that translates code written in a high-level programming language into machine code that can be efficiently executed on various platforms, including GPUs and specialized Al hardware.

[0139] Eclipse can refer to an open-source integrated development environment (IDE) used for programming in various languages, including Java, C++, and others. It provides tools for code development, debugging, and testing.

[0140] Fault tolerance can refer to the ability of a system or application to continue operating properly in the event of a failure of some of its components. In software, fault tolerance techniques can include redundancy, error handling, and failover procedures.

[0141] A library can refer to, in the context of Al, a collection of modules or functions specifically designed to assist in developing and deploying Al models. These libraries often include pre-built algorithms, neural network architectures, and statistical tools that streamline the process of machine learning model development.

[0142] CUDA (Compute Unified Device Architecture) can refer to a parallel computing platform and application programming interface (API) model. It allows software developers to use a CUDA-enabled GPUs for general purpose processing.

[0143] CUDA-GDB can refer to a tool for debugging CUDA applications. It extends the functionality of the standard GNU Debugger (GDB) to support debugging of parallel computations on some GPUs.

[0144] NSight can refer to a suite of performance analysis tools to help developers optimize the performance of applications running on GPUs.

[0145] A programming model can refer to a conceptual framework that defines the types of constructs and techniques used for coding within a programming environment. It guides developers in how they can write their programs for specific hardware or software platforms.

[0146] Runtime can refer to, in the context of Al, the environment that supports the execution of Al models. It manages the allocation and optimization of system resources such as memory and processing threads while the program is running.

[0147] Visual studio code can refer to a lightweight, open-source integrated development environment (IDE). It supports a wide range of programming languages and frameworks and is highly customizable with extensions.

[0148] Now turning to the inventive concepts, embodiments of the disclosure are described at least to address the challenges of LLM inference. In particular, the disclosed systems can use various techniques to reduce the memory footprint and computational cost. Some algorithmic techniques, such as quantization, pruning, and sparsity, aim to compress model parameters or eliminate redundant computations without significantly sacrificing accuracy. Additionally, the disclosed systems can use some Mixture-of-Experts (MoE) architectures that can distribute model parameters across a large number of specialized expert networks that are dynamically selected based on the input, enabling scaling up model size while maintaining computational efficiency. Furthermore, the disclosed systems can use multimodal LLMs (MLLMs) to process and generate multiple forms of data, including text,images, audio, and video. While MLLMs enable applications by processing multiple types of data, they also present significant memory challenges.

[0149] In addition to ML specific technologies, some embodiments of the disclosure may include an approach to solving memory -related challenges by integrating compute with memory technology, called Processing-in-Memory (PIM). PIM for LLM inference may include one or more of the following characteristics. In one aspect, a PIM-enabled system such as that of the disclosed embodiments can provide several advantages, including reduced energy consumption. By placing memory and compute units in close proximity, the disclosed systems can use PIM to minimize energy -intensive and time-consuming data movement between them. PIM can enable dense and scalable memory configurations, unlike CPU / GPU- attached DRAM. This is because PIM is not limited by chip shoreline constraints, allowing for more storage capacity and better resource utilization.

[0150] In one embodiment, the disclosed systems can use PIM to leverage higher internal memory bandwidth compared to off-chip interfaces. Provided that computing power can be proportionally increased, this can lead to faster data processing and reduced latency. In one respect, the disclosed systems can use PIM to enable massively parallel in-memory computation by exploiting memory-level parallelism across various dimensions such as banks channels dies. This capability makes it particularly suitable for large-scale parallel computations like those required for LLMs. In one respect, PIM supports fine-grained low- overhead data flows surpassing CPU / GPU architectures due its ability simplify optimize logic specific application ranges. This translates to improved system efficiency and responsiveness.

[0151] PIM enabled systems such as that of the disclosed embodiments may improve computing by placing computation closer to memory. Some embodiments of the disclosed systems can address one or more of the following issues: (1) Thermal: As computation isbrought closer to memory, the memory may become more prone to heat-related problems. This increased susceptibility to temperature fluctuations can lead to errors, slow down performance, or even cause permanent damage to the system. (2) Scalability: As LLMs continue to grow in size and complexity, it becomes increasingly difficult to develop systems that can efficiently process them without sacrificing performance. (3) Reliability: The accuracy and consistency of results must be maintained across various workloads and environmental conditions. Any deviations from expected outcomes could have serious consequences, making reliability a top priority. (4) Programmability: Users may need programming models that can tap into the unique advantages of PIM architectures. (5) Software stack: Building a comprehensive software ecosystem that seamlessly integrates with PIM-based hardware is essential. A robust software stack of the disclosed systems can enable developers to harness the power of PIM technologies, ensuring widespread adoption and driving innovation forward. The hardware and software components of PIM-enabled systems such as that of the disclosed embodiments may be co-designed, taking into account of the distinct characteristics of LLMs.

[0152] Some embodiments utilize stacked memory with computational capabilities which can also be referred to as AA-HBM or AHBM to add further capabilities to PIM-based systems and address the challenges associated with PIM. In various respects, A-HBM enabled systems such as those of the disclosure may not only incorporate PIM capabilities, but also enhance them, thereby providing a comprehensive solution for optimizing the performance and efficiency of LLM inference. By achieving these objectives, an A-HBM system according to some embodiments may enable efficient and effective processing of complex language models.

[0153] In some aspects, the disclosed systems can include an A-HBM that features a memory device that combines PIM with custom High-Bandwidth Memory (cHBM). In somerespects, AHBM can be used to support Al LLMs. For example, A-HBM can integrate compute functions optimized for LLMs directly into a base die (e.g., an HBM4E-compatible base die), which can represent one of the dies (e.g., the bottom die) on which the memory modules can be placed. This can enable A-HBM to provide higher memory bandwidth with larger HBM stacks, resulting in improved performance by reducing memory bottlenecks when used by NPUs / GPUs and similar processing elements.

[0154] In some respects A-HBM systems of the present disclosure provide numerous advantages for computational activities such as LLM inference, including but not limited to the below. For example, the present disclosure provides an A-HBM based inference system that achieves significant improvements in power efficiency. For instance, the disclosed systems can include an A-HBM-based inference system comprising multiple cards with multiple A-HBM tiles can achieve a substantial improvement in power efficiency, such as approximately 5 times the power efficiency as compared to alternative systems (particularly applicable in LLM computations such as GPT-3 175B and the like). This enhancement in energy efficiency enables datacenters to achieve more performance within current power- constrained operation conditions.

[0155] In one embodiment, another advantage of the present disclosure is that the A- HBM system accommodates diverse LLM types by supporting various data types and general cores as well as tensor cores. This versatility enables A-HBM compatibility with emerging LLM architectures and methodologies, providing a future-proof system for evolving Al and machine learning applications.

[0156] In one embodiment, yet another advantage of the present disclosure is that the disclosed A-HBM system is scalable in terms of model size, sequence length, batch size, and number of users supported. The system based on A-HBM can serve massive models (e.g., those exceeding about at least 1 trillion parameters) and long context lengths (e.g., about atleast 1 million sequence length). The use of interconnect protocols including, but not limited to, Universal Chiplet Interconnect Express (UCIe) 1.1, Ultra Accelerator Link (UAL) 1.1, and Ultra Ethernet enables scalability and flexibility in the system.

[0157] The advancements in ML have both driven and been enabled by developments in hardware as discussed in the following. For example, specialized Al accelerators such as GPUs other Al-specific chips have dramatically increased the compute capability available for ML tasks. Memory and bandwidth: The increasing size of models and datasets has driven the need for higher memory capacity and bandwidth in Al hardware. Distributed and parallel computing: Improvements in interconnect technologies and distributed computing frameworks have enabled training of ever-larger models. Heterogeneous computing: Integration of different types of processors (CPUs, GPUs, TPUs, field programmable gate arrays (FPGAs), etc.) has become relevant for handling diverse ML workloads efficiently. Hardware-software co-design: Closer integration of hardware and software design has led to more efficient ML systems, with hardware features specifically tailored for ML workloads. Neuromorphic computing: Advancements in brain-inspired computing architectures promise energy-efficient processing for certain types of neural networks. These hardware developments have been crucial in enabling the rapid progress in ML, and the co-evolution of ML algorithms and hardware is expected to continue driving innovation in both fields.

[0158] The disclosed systems can be configured to enable Al applications. As noted, recent advancements in generative Al have been predominantly driven by LLMs. Some models (e.g., GPT-3.5, GPT-4, and PaLM) have demonstrated unprecedented capabilities in generating human-like text across various domains and tasks. ChatGPT and instruction-tuned models have shown abilities in understanding and generating contextually appropriate responses. While not strictly LLMs, models like DALL-E 2 and stable diffusion leverage techniques from LLMs to generate images from textual descriptions, showcasing the cross-modal influence of LLM advancements. Extension of capabilities based on LLM leverage technologies such as MoE and MLLM. MoE activates only a portion of expert(s) best suited to a given input out of many experts in LLM and it is known to have 4-8x capacity increase with similar computing power. Another technology is multimodal LLM, in which multimodal inputs are converted to LLM inputs and LLM outs are converted to multimodal outputs, with the same LLM for text being used.

[0159] The disclosed systems can be configured to enable LLMs that can create a unified representation space for both image and text data. This LLM-driven approach to generative Al has led to a paradigm shift in how we approach various generative tasks. The ability of these models to understand context, follow instructions, and generate coherent and diverse outputs has opened up new possibilities in creative applications, content creation, and ALassisted tools across numerous domains.

[0160] In one embodiment, the disclosed systems can serve to accelerate the inference process of these LLMs using memory. In this context, the description herein describes the requirements of LLMs from two perspectives: user experience metrics and system metrics.

[0161] Overall System Architecture

[0162] FIG. 1 shows an example of an overall system architecture, in accordance with embodiments of the disclosure. At a high level, FIG. 1 shows how multiple A-HBM units may be arranged to form a card, multiple cards may be put on a tray, and multiple trays in a server. In various aspects, the disclosed system architecture can enable scalable and high- bandwidth computing and is optimized for applications requiring high memory bandwidth and low latency. The disclosed systems are configured to utilize a scalable architecture comprising multiple A-HBM SiP modules interconnected via different interfaces as described variously herein. This architecture enables high-bandwidth and low-latency communicationbetween compute units, making it suitable for any suitable application, including but not limited to, large-scale HPC or Al applications.

[0163] In particular, the diagram shows a server rack 102 with multiple slots, each containing a compute module, including various components in a rack format, for example, a network card 104, described further below. The server rack 102 also includes a switch tray, which can include multiple switches and various ports, and can serve to connect multiple trays to one another to provide a scalable architecture and to collect and distribute data and network traffic between various cards in the trays to one another and to other servers. The middle section 114 of the diagram illustrates a compute tray 106 including multiple A-HBM SiP cards 103, which are connected to one another via UAL interfaces (e.g., at a predetermined bandwidth of, for example, 448 Gbps) and a CPU 107 that is connected to the A-HBM SiP Cards 103 via another interface (e.g., a PCIe7 based interface at a predetermined bandwidth of, for example, 128 Gbps). The right side of the diagram 126 shows an expanded view of a given A-HBM SiP Card including actual A-HBM dies connected in a network topology with multiple nodes. As shown in diagram 126, the A-HBM dies can be connected to one another via a D2D interface such as UCIe (e.g., at a predetermined bandwidth of about 512 Gbps) and can be connected to other A-HBM cards via UAL links interconnects 116 or to external devices (e.g., a host) via a PCIe interface.

[0164] Various components of FIG. 1 are discussed further below in the context of embodiments of the disclosure. As noted, the disclosed systems can include an A-HBM SiP card such as A-HBM SiP card 103, which can include A-HBM devices (e.g., including PEs and HBMs) and interconnects (e.g., UAL, UCIe, and PCIe interconnects and / or supporting chiplets). The disclosed systems can be configured to use a suitable protocol for storage, compute, and / or memory operations such as PCIe to transfer data and commands between a host and an A-HBM SiP, thereby providing bandwidth for LLM performance. In some cases,the data can have a predetermined size which can be, in some examples, around tens to hundreds of gigabytes which can be loaded into the disclosed system’s memory through one or more PCI-e links before serving inference.

[0165] The disclosed systems can, in some examples, employ a given interconnect such as UAL, an open-standard interconnect for Al accelerators, although the disclosed systems are not limited to this interconnect type. In some aspects, a UAL 116 (or any other similar) based connection can be used to enable the transfer of data between A-HBM SiP cards. In LLM, collective communications such as all-reduce and all-gather operations that synchronize data between A-HBM SiPs may occur per transformer layer, providing relatively low communication overheads. In some embodiments, as noted, the disclosed systems can include a compute tray 106 that can include multiple A-HBM SiP cards 103 that can be connected to one another using a predetermined UAL configuration (e.g., a UAL xl6 configuration) which can provide a predetermined bandwidth (e.g., about 448 GB / s of bandwidth).

[0166] In one embodiment, as shown in the middle diagram 114 the disclosed systems can include a compute tray 106 that can serve as a single server system containing CPUs (e.g., CPU 107), memory (e.g., DRAM or flash, not shown), a predetermined number (e.g., 8) of A-HBM SiPs, and a network card (e.g., network card 105). The compute tray can be, in some examples, configured as a minimal deploy platform for processing LLMs; for example, a given compute tray can execute a predetermined number (e.g., 96) of multi-head attention heads of a given LLM model concurrently without tray-to-tray communication.

[0167] When required to support higher concurrency levels or larger models than what can fit within one compute tray, the disclosed systems can use multiple trays that may be interconnected using UAL links and via a switch tray as shown in FIG. 1 allowing scalingacross trays via Ultra Ethernet technology thereby facilitating efficient execution of LLMs requiring high degrees of parallelism across distributed hardware configurations.

[0168] A-HBM HW Architecture

[0169] FIG. 2 shows high-level architectures associated with an A-HBM device / unit for the disclosed systems, in accordance with embodiments of the disclosure. In particular, the figure depicts a system architecture that can be used for high-performance computing or Al applications. The left side of the diagram 202 shows an A-HBM device which comprises a near memory compute device that can have an A-HBM 3D stacked memory structure, comprising multiple layers of memory and compute units. The stacked memory structure can include a base die 201 (also referred to as a logic die or a buffer die herein) that can include computational functionality (e.g., Al acceleration functionality such as GEMM computational functionality). Further, the A-HBM device / unit (also referred to herein as a near compute device) can include memory (e.g., DRAM) core die that include multiple layers of memory and compute. The memory can further include HBM memory (e.g., HBM4E memory) and can, in some examples, also include some computational functionality (e.g., via processing elements, PEs) that can perform various computational functions, as described further herein.

[0170] The central part of the diagram 214 illustrates a base or buffer die (e.g., as shown as base die 201) for the A-HBM which includes logical functionality such as multiple PEs 203, which are connected to each other via a connection such as UCIe-A 205 (UCIe- Advanced, which features greater bandwidth, scalability, and latency compared to UCIe) interfaces. A physical layer interface UCIe PHY for UCIe-A is shown, which enables communication between PEs (e.g., PE 203) and other components. TSVs (Through-Silicon Vias such as TSV 207 are used to connect multiple layers of the 3D stacked memory structure). The right side of the diagram 226 illustrates the architecture of PEs (e.g., PE0 223 through PEI 5 253) which are configured to communicate over a network-on-chip (NOC)interconnect 219 over multiple pseudochannels 217 and 226, which can include logical channels for separate, high-bandwidth paths for data transfer over a single physical channel. Each PE (e.g., PEO 223 or PEI 5 253) can include one or more of the following: a processing element that executes instructions. TCM (Tightly Coupled Memory) which includes small, low-latency memory that stores data and instructions. An instruction Cache that includes a cache that stores instructions for the processor. A hardware accelerator GEMM for matrix multiplication operations. A MATH hardware accelerator for mathematical operations.

[0171] As noted, the diagram shows a NOC interconnect, which enables communication between PEs and other components over the pseudochannels (e.g., psuedochannels 0-63 217).

[0172] Overall, the disclosed systems utilize a 3D stacked SoC architecture comprising multiple PEs (e.g., PEs 203) interconnected via various interfaces such as UCIe-A (e.g., UCIe-A 205), PCIe, and NOC interconnect (e.g., NOC interconnect 219) interfaces. This architecture enables high-bandwidth and low-latency communication between compute units, making it suitable for large-scale HPC or Al applications.

[0173] To describe the embodiments in more detail, as shown in the figure, each A- HBM unit / device (e.g., shown in diagram 202) can include multiple stacks of HBM core dies on top of a buffer die 201. In some respects, the A-HBM unit / device 202 can include stacks of HBM core dies, for which the disclosed systems can include HBM4 and HBM4E core dies in some embodiments. Such dies can provide a predetermined memory capacity and bandwidth at a given power level (e.g., about 36 GB of capacity and about 2 TB / s of bandwidth at about a 40 W power level).

[0174] The diagram 214 shows a relatively simplified architecture of A-HBM buffer die (for more detailed views, please refer to FIGs. 4 and 5, below). As noted, diagram 214 includes PEs 203 that compute, UCIe interfaces (e.g., UCIe-A 205), and TSV channels 207 toaccess HBM memory (e.g., the HBMs in the HBM stack on top of the base die 201). For die- to-die connections, the disclosed system can use a predetermined number (e.g., 2 or 3) modules of UCIe-A connections (e.g., part of UCIe-A 205), which includes a predetermined number (e.g., 8 or 12) modules of UCIe-A PHY (shown in diagram 214), to provide a predetermined total data bandwidth (e.g., about 512 GB / s) for die-to-die communication.

[0175] In some respects, the disclosed systems can include an A-HBM comprising PEs such as PE 203 in which each PE is associated with a group of HBM pseudo channels like those shown as pseudochannels 217. The diagram 226 shows an example configuration where 4 pseudo channels (e.g., pseudochannel 0-3) are mapped to one PE (e.g., PE PE0 223). Thus, in a case where an HBM memory module (e.g., HBM4 module) has 64 pseudo channels, there can be a total of 16 PEs that are configured to access the HBM. In some aspects, the NOC interconnect 219 can be configured to provide mapping between pseudochannels and PEs, allowing one PE to access any pseudochannel to retrieve data from any HBM memory on the AHBM device / unit.

[0176] In some aspects, a PE (e.g., PE0 223 or PEI 5 253) can include various sub-units as shown as sub blocks of the PE in diagram 226. For example, a given PE (e.g., PE0 223) can include processor and an instruction cache to run the kernels associated with operations of a ML task, coordinating memory and compute resources. In some embodiments, the A- HBM can use a general processor (e.g., a CPU or coprocessor) for this purpose. The PE (e.g., PE0 223) can include tightly coupled memory (TCM) (or similar) can be a relatively fast memory that the PE can access. For example, it may be used as a scratchpad memory that can store prefetched data from HBM, user data, and intermediate data. The TCM can further be private to each PE or can have some parts of the TCM be exposed to other PEs in the sameA-HBM.

[0177] In some aspects, the PE’s compute logic for LLM inference can be divided at least into two parts to provide general computing capabilities as shown in sub-blocks of the PE (e.g., PEO 223): GEMM and MATH. In an example, a GEMM engine can be responsible for matrix-to-matrix multiplication acceleration. The MATH engine can be configured to process element- wise operations on floating point numbers, including basic math, exponentiation, and trigonometric functions.

[0178] In various aspects, the disclosed systems can be configured to use any number of PEs that serve as computing logic for each A-HBM device / unit based on various system parameters including, but not limited to, available area, thermal and power constraints, and memory bandwidth.

[0179] A-HBM SW Architecture

[0180] This section on the AHBM SW architecture and the below on the detailed A- HBM architecture is now described to provide context for the figures and description throughout and is provided here to give the reader additional information about details of the disclosed systems. In some embodiments, the disclosed systems can include a PE architecture that features a memory architecture allowing efficient access to memory from compute units. Moreover, the system can include A-HBM LLM inference kernel libraries and a compiler that can be configured to utilize such hardware features.

[0181] The disclosed systems can use A-HBM inference-optimized libraries to work with the hardware features of the PEs, and provide custom designed kernels for each LLM inference operation including GEMM, all-reduce and self-attention. The disclosed systems permit users to execute inference models in at least two modes: an eager mode and a graph mode. In the eager mode, each operation can be evaluated immediately without building graphs. Weight and activation data placement can be handled by users. In graph-mode, the disclosed systems can build an efficient graph before evaluation, allowing an Al compiler tooptimize kernels and data. For example, in the graph mode, the A-HBM Al compiler can place weight and activation data evenly across HBM groups for high PE utilization. The features of other Al stacks are similar to that of GPUs. The detailed optimization mechanisms are described further below.

[0182] In some aspects, the disclosed systems can include an A-HBM host program that can be configured to execute the following steps: Copy input data from host memory to device memory; this can be handled by a model compiler or a memory copy APIs, which can split the data across PEs. Load and execute the kernel; when a kernel load request is queued, the runtime fetches the command and deliver it to every A-HBM package in each A-HBM SiP through a device driver. Copy the results back to host memory; the results are fetched from HBM memory to a host; the contents in the scratchpad TCM memory can be discarded after exiting the kernel. The computation results temporarily stored in TCM can be copied to HBM by the kernel, before exiting the kernel.

[0183] In some respects, the A-HBM programming model can be similar to that of some GPUs. Processing entities, PE and GPU cores or A-HBM and SM can play the same role processing data in a distributed and parallel manner. In some respects, A-HBM SiP may not need to include L2 caches. For example, any operations optimized to use L2 caches can still be executed in A-HBM but it may not be as efficient. A-HBM programs have better performance when they are optimized by aligning data and compute units. A-HBM kernels internally use local memory addresses to access PE’s internal resources. When a kernel accesses remote memory or resources, the disclosed systems can include a memory management unit (MMU) can translate the local address to physical one.

[0184] Detailed A-HBM Architecture

[0185] As noted, the section above on the AHBM SW architecture and the below on the detailed A-HBM architecture is described to provide context for the figures anddescription throughout and is provided here to give the reader additional information about details of the disclosed systems.

[0186] Level 1 : Die

[0187] In one embodiment, the disclosed systems can include an A-HBM architecture that introduces a unified compute node clustering as a building block. This building block can include of a single HBM memory, multiple PEs such as GEMM and math engines, a management CPU, and shared SRAM memory for activation updates. In another embodiment, this building block can serve to group compute nodes around high- capacity / high-bandwidth HBMs. By doing so, latency can be minimized and data throughput of the HBM is maximized, avoiding traditional L2 cache architecture. Within a cluster, PEs extensively access data, particularly for large-sized model data. Additionally, key and value data from user queries are created and updated during inference operations.

[0188] In one embodiment, the disclosed systems can include multiple instances of these A-HBM units / devices in a given A-HBM die (which can be placed on a card) to be interconnected in a mesh network of multiple A-HBM units / devices to achieve scalability and expansion. The mesh network architecture allows a given PE (or neural processing unit (NPU) which can include a PE and / or a PE along with other supporting components like cache, memory, or controllers) of a given A-HBM unit / device to not only have connectivity within its own A-HBM unit / device but also access another A-HBM unit / device’ s PE resources on the same die. In one embodiment, when extended through die-to-die interconnects as described variously herein, the disclosed systems can feature a multi-die network of A-HBM dies. Within this network, the disclosed systems can include PEs located on one A-HBM die that can access PEs on neighboring A-HBM dies.

[0189] In one embodiment, the disclosed systems can include an A-HBM SiP package to package (i.e., card-to-card) interconnect (e.g., UAL interconnect) that allows expansioninto large multi-card networks. In some respects, even the PEs of individual A-HBM dies and A-HBM units / devices within the A-HBM dies that are on the cards can be interconnected at various levels of hierarchy / granularity, which can permit data exchange among them without CPU intervention. On each card, the disclosed systems can maximize scalability, the disclosed systems can include an A-HBM unit / device architecture that incorporates a 4- direction interconnection connecting all adjacent A-HBM dies. This arrangement enables connectivity on the Left / Right / Top / Bottom sides of the die through a UCIe interconnect (e.g., at an example bandwidth of about 512 GB / s), as further shown and described in connection with FIG. 3, below. The bandwidth of each interconnect can be consistent across all four directions.

[0190] In some examples, the disclosed systems can include an interposer (e.g., a silicon interposer) that connects the various A-HBM dies and / or interconnects. For example, if the size of the interposer can be, in some examples, about 52 mm x 68 mm, the die size that can integrate 16 A-HBM dies in a 4 x 4 array format can be, in some examples, about 12.5 x 16.5 mm, under the condition of using a silicon interposer in a SiP. This assumes an edge margin of about 1 mm on each side. This results in, a given example, an about 80mm x 80mm package that can also comply with various requirements, such as for example, standards based sizing requirements (e.g., 0AM sizing requirements). In other aspects, the disclosed systems can include packaging systems that can serve to reduce the limitations of silicon interposer enabling bigger base die size and / or bigger tile configuration like 5x5 as long as power levels allows.

[0191] A-HBM SiP

[0192] FIG. 3 shows a diagram of a semiconductor system that includes an A-HBMSiP for the disclosed systems, in accordance with embodiments of the disclosure. The diagram shows aspects of the disclosed systems which utilize a scalable architecturecomprising multiple A-HBM dies (e.g., A-HBM die 305) interconnected via D2D interfaces (e.g., D2D interface 311) within an A-HBM SiP package. In some respects, the HBM dies can also be referred to as a near memory compute die or package herein. This architecture enables high-bandwidth and low-latency communication between compute units, making it suitable for large-scale HPC or Al applications, specifically designed for large-scale processing of complex tasks.

[0193] The diagram shows an A-HBM SiP package 300 at a card level with predetermined dimensions (e.g., about 80x80mm). The package contains multiple A-HBM dies (e.g., A-HBM die 305), each with predetermined dimensions (e.g., about 12.5x16.5mm), which can be placed on one or more an interposers (e.g., a silicon interposer), not shown. The architecture shows 16 A-HBM Dies, where each A-HBM die is a 3D stacked memory structure, comprising multiple layers of memory and compute units. The diagram shows interconnects that can include I / O chiplets 303, which handle input / output operations between the A-HBM SiP package and external devices using one or more interconnects (e.g., PCIe, UALink). The diagram shows D2D interconnects, which enable communication between individual A-HBM dies. The package includes UALINK interfaces 327, which provide highspeed connectivity between the system and external devices. The package includes PCIe interfaces 323, which provide connectivity between the system and a host. In an aspect, the I / O chiplets can perform operations on and control the data (e.g., filter, scan, perform error correction on, etc.) flowing through the interfaces / interconnects (e.g., PCIe and / or UALink interfaces / interconnects) .

[0194] As noted, the A-HBM SiP package contains multiple A-HBM dies, each with its own onboard I / O chiplets and D2D interconnects such as I / O chiplet 303 and D2D interconnect 311. The dies can be arranged in a grid-like pattern, with each die connected to its neighbors via respective adjacent D2D interconnects; it is understood that other patternscan be used and that the provided example is non-limiting. The diagram highlights an optional feature comprising A-HBM direct storage (e.g., via an SSD 317 over a PCIe link 323), which enables quick context switching or model switching; the A-HBM SiP may access (EG, with local or remote access control) an attached storage In some aspects, the disclosed systems can include a predetermined number of A-HBM base dies (e.g., 16 A-HBM base dies) can be integrated into a multichip package using an interconnect such as a silicon interposer or a silicon bridge.

[0195] In one embodiment, a package comprising approximately 16 HBM chips is provided. The memory capacity of the package can be, in an example, approximately 768 GB, which is achieved by including approximately 16 HBM4E chips each having a capacity of about 48 GB. The package provides a memory bandwidth of, in an example, approximately 32 TB / s to about 40 TB / s, which is achieved by including, in an example, 16 HBM4E chips each having a bandwidth of about 2 TB / s to about 2.5 TB / s. In one embodiment, the package can have, in an example, an estimated size of about 80 mm x 80 mm. The package may include a card-to-card interconnect with a data transfer rate of approximately 448 GB / s per chiplet, using a UALink interface. Additionally, the package may include a die-to-die interconnect with a data transfer rate of approximately 512 GB / s per side, using a UCIe-A two-module interface.

[0196] In one embodiment, the disclosed systems can include a SiP that comprises an I / O chiplet that supports a host interface and chip-to-chip interface, including, in an example, PCIe Gen7 x8 with a data transfer rate that can be, in an example, approximately 128 GB / s. The I / O chiplet is configurable to support multiple protocols, including cache coherent protocols such as: CXL2.x with CXL.IO and CXL.MEM and any other suitable protocol; PCIe Endpoint for traditional server farms or clouds that do not support CXL; and PCIe Root Complex for compute node direct storage (for example, non-volatile memory (NVMe) basedsolid state drives). The I / O chiplet is also configurable for PCIe Gen7 Endpoint / Root Port and UALink. In one embodiment, the VO chiplet can be connected to an A-HBM die via an UCIe-A interconnect.

[0197] In one embodiment, although not specifically shown on the figures (e.g., FIG.3) the disclosed systems can include a SiP that includes hardware features for data encryption and decryption (IDE) for user data / model security, although the specific implementation details can be configurable via one or more logical chips.

[0198] In one embodiment, the A-HBM die comprises a base die stacked with a DRAM core die. The DRAM core die may be implemented using any suitable memory technology, such as either HBM4 or HBM4E technology, with a memory capacity, in an example, of approximately 24 Gbit or 32 Gbit per die, respectively, and approximately 12 stacks in a non-limiting example. In some cases, if there are significant differences in power TSV location between HBM4 and HBM4E, the A-HBM base die may be able to support one type of HBM core die. The speed of the HBM core die may range from approximately 8 Gbps to about 10 Gbps. In one embodiment, the bandwidth of the HBM core die can range from approximately 2 TB / s to about 2.5 TB / s in some examples.

[0199] A-HBM Die Block

[0200] FIG. 4 shows a block diagram of an A-HBM base die for the disclosed systems, in accordance with embodiments of the disclosure. The diagram shows the AHBM base die and various components and their interconnections, discussed below. This diagram provides an overview of some aspects of the AHBM architecture, highlighting its key components, including die-to-die interconnects, system bus mapping, testing logic, TSV landing zones, physical layers, and memory controllers. These components work together to enable efficient processing, data transfer, and memory access within modern computing systems.

[0201] In some respects, the PE blocks (e.g., PE block 402) are connected to the system bus mapping (e.g., system bus mapping 411) and D2D interconnect 407 (e.g., featuring a UCle-Adv PHY 2 module at approximately 512GB / s) to distribute data and commands to various portions of the device and enable communication between different dies (or chips) in the A-HBM SiP. The AHBM base die also includes test logic (e.g., the memory built-in test (MBIST / IEEE1500 435 and the HBM TEST Logic block 437) which can enable testing and validating the AHBM components. Further, the AHBM base die shows a TSV landing zone and 3D DRAM PHY (Physical Layer) 415. The TSV landing zone refers to the area where the vias interconnecting to memory stacked on the base die (EG, HBM) land on or connect to the base die’s surface. The 3D DRAM PHY controls access to the stacked DRAM memory. The AHBM base die further features controllers 411 (e.g., a 32- channel HBM4 controllers) that manages access to the HBMs. There are several PEs, and each PE can serve, in some aspects, as an NPU with GEMM / GEMV matrix multiplication capability, a math engine for operations such as Softmax, activation functions like SWIGLU and GELU / RELU, and a DMA engine for data movement.

[0202] In more detail, the diagram depicts an A-HBM base die block diagram which, in some examples, can have a single HBM TSV and multiple PEs. The A-HBM base die has the following sub blocks. In one embodiment, a 3D-DRAM-PHY IP is integrated on a TSV landing zone area to support 3D stacking of HBMs. The 3D-DRAM-PHY can support, E.G. HBM4 / HBM4E at a predetermined speed, for example, of about 10 Gbps / bit. The interface type between the 3D-DRAM-PHY and the HBM controller IP can include, in an example, a DFI interface. The DFI clock needs to be set up to a predetermined amount, for example, to about 1.25 GHz. A phase lock loop (PLL) can be placed near the 3D-DRAM-PHY to minimize skew. The PE core can support various data types, including FP16, BF16, FP8, INT4, INT8, and INT32, with optional FP32 for reduction precision. The total PE's computeperformance for example, on a given die, can be, in some examples, about 90 TFLOPs atFP16.

[0203] In some aspects, the disclosed systems can include internal TCM memory, not shown in FIG. 4, but which is shown and further discussed in connection with PE internal configurations shown in FIG. 5, below. In an aspects, the TCM memory can be, in an example, about 2 MB, although more TCM may be preferable given die space availability. A compute cluster of A-HBM can include, in an example, about 16 PEs surrounding a single HBM. Two additional PEs can be placed for defect replacement, providing fault tolerance. The total number of PE cores can be, in an example, 18, considering optional redundant PE cores if area allows. Kernel execution on a core (e.g., an ARM-based core) with a coprocessor interface enables the creation of custom instructions for tensor / math engines / FUs. Fused instruction support is also provided by the coprocessor interface. The operating clock speed can, in some examples, range from about 1 GHz to about 1.25 GHz, and clock scaling between about 60% and 100% may be required for power throttling.

[0204] The system can include a NOC network for the HBMs and PEs, which is not shown in FIG. 4, but which is shown and further discussed in connection with FIG. 5, below. A NOC router interface bus protocol can use any suitable protocol such as AXI and AXI- Stream for D2D inbound / outbound. In one embodiment, the bandwidth between HBM and 16 PE can be, in an example, about 2 to 2.5TB / s. Four pseudochannels can be logically mapped to one PE. The bus operation frequency ranges from, in an example, about 1 GHz to 1.25 GHz, with clock scaling between 60% and 100%. A die-to-die protocol block such as CCG (CCIX / CXL Gateway), along with a die-to-die interconnect (such as UCIe) can enable multichip mesh network connections to other dies or cards.

[0205] The die-to-die interconnect can use, in an example, UCIe-A x64 advance package with 2 (or 3) modules, providing a required bandwidth of about 512 GB / s. Theinterconnect features 4-side connectivity, with about 512 GB / s per side, on the Top / Left / Right / Bottom sides. Hardware internal / external loopback test support is also provided for lane repair. In some examples, as shown further in connection with 5, the disclosed systems can include shared SRAM that is used for storing various types of data, including, but not limited to, input activations or intermediate activations for various batch size, KV caching, and kernel binary.

[0206] The disclosed systems can also use a management CPU, not shown in FIG. 4, but which is shown and further discussed in connection with FIG. 5, below. The management CPU employs a core processor such as an ARM-based core, with tightly coupled memory such as ITCM, DTCM (ITCM can include a 64-bit memory interface and DTCM can include two 32-bit memory interfaces), instruction cache (I-Cache), and / or data cache (D-Cache). The core can also features debugging and operates at a predetermined frequency, such as approximately 2 GHz.

[0207] In one embodiment, the disclosed systems can include a CPU subsystem (not shown) that can include peripherals such as a thermal sensor (not shown), which can be used for power throttling. For example, the thermal sensor can be placed in any suitable location on an AHBM die, an AHBM device / unit and can monitor temperature and provide signals to adjust power consumption accordingly. The CPU subsystem peripherals can also comprise a timer, Universal Asynchronous Receiver-Transmitter (UART), Serial Peripheral Interface (SPI), Vectored Read-Only Memory (VROM), and General-Purpose Input / Output (GPIO) pins, which can facilitate system management and communication. In another embodiment, the CPU subsystem can include a clock reset generation block (clock reset gen), which can generate clock and reset signals to synchronize and manage the system's operational states. Additionally, the CPU subsystem can comprise a Power Management Unit (PMU), which can oversee and regulate power distribution across the system.

[0208] In some aspects, the disclosed systems can include a HBM design-for-test(DFT), which in one embodiment, can include an HBM test logic block as shown in FIG. 4. This test block can be an IP block in an AHBM base die. The HBM DFT architecture can support any suitable standards-based features (e.g., IEEE1500 features), including lane repair mechanisms that enable repair of faulty lanes; custom pattern tests that allow for testing with specific patterns; MBIST control for managing built-in self-test operations; feature support for comprehensive testing of the any HBM core die; and D2D loopback test control functionality.

[0209] Furthermore, in an embodiment, MBIST functionality is provided within the HBM DFT architecture. DA (direct access) support is also included for CoW testing purposes. For instance DA support may enable high speed (HS) tests or low speed (LS) tests under certain conditions. The disclosed systems may further provide on-die Automated Test Equipment (ATE) support.

[0210] In one embodiment, the disclosed systems can include various debugging features that can include telemetry support, which can enable periodic logging by a management central processing unit (M-CPU) for HBM debugging or failure analysis. For example, the telemetry support can involve logging temperature data for all core die stacks and A-HBM base dies. Additionally, the telemetry support can include logging refresh rates and performing healthiness checks by severity field, such as Error-Correcting Code (ECC) correction bits.

[0211] In another embodiment, the debugging feature can comprise debug logic with debug over power down support. This can enable efficient debugging of the system even when certain components are powered down. The system can also include an Embedded Trace circuit for instruction and data tracing, allowing for detailed analysis of system behavior. Furthermore, in an embodiment, the disclosed systems can provide PerformanceMonitors Extension support can be provided. This can facilitate software profiling and performance debugging by enabling monitoring of various system parameters.

[0212] The disclosed systems may also provide a debugging interface such as a Cross Trigger Interface (CTI) for multiprocessor debugging purposes. For instance CTI may allow triggering debug events across multiple processors enabling more comprehensive analysis under certain conditions. In addition to these features, in one implementation, the disclosed systems can include a suitable interface such as a JTAG interface although other interfaces might alternatively or additionally be utilized depending upon design requirements among other considerations. In some embodiments, virtual trace logging over PCIe are supported including timestamping logging memory allocation by PE’s microprocessor M-CPU as well as via specific logging formats.

[0213] PE Cluster

[0214] FIG. 5 shows a cluster of PEs for the disclosed systems, in accordance with embodiments of the disclosure. The figure presents a detailed diagram of an AHBM base, showcasing its various components and their interconnections. These connections enable data transfer and communication between the different components. Overall, the diagram provides a comprehensive overview of the computer chip's architecture highlighting its complex design composed of a management CPU and memory controllers as shown by section 506, and a NOC interconnect 504, and multiple PEs as shown in section 502 that work together to facilitate efficient processing, data transfer, and memory access. The diagram shows several components. A main / management CPU (M-CPU) representing the main central processing unit. Shared SRAM indicating shared static random-access memory. HBMC (High- Bandwidth Memory Controller) Channels labeled as HBMC CH-0, CH-1, CH-2, and so on, which manage access to high-bandwidth memory. NOC interconnect shown as a barconnecting the various components, facilitating communication between them. PE Blocks labeled as PEO, PEI, and PEN-1, which represent processing elements.

[0215] Each PE block can include the following. A CPU: A block within a given PE (e.g., PEO), indicating a core such as an ARM-based CPU core. LI Cache: within a given PE (e.g., PEO), representing level 1 cache memory. AXI (Advanced extensible Interface)-M that connects LI cache to the MMU at a predetermined frequency such as about a 1 GHz frequency. A MMU component responsible for managing memory access (e.g., translating physical to logical addresses). Multi-Channel DMA component that enables efficient data transfer between different parts of the system (e.g., between HBM memory and compute units (e.g., tensor units, math engines, etc.). TCM representing tightly coupled memory that enable scratch pad operations. FU (Functional Units) within each PE block, which perform specific functions such as tensor operations and math engine tasks, which can, in some examples be performed in different areas of the PE.

[0216] In more detail, in some aspects, a PE-related cluster can be logically defined and can include a combination of PE cores, HBM stacks, SRAM, DMA engine(s), and interconnects (e.g., D2D interconnects). In some examples, such as that shown in FIG. 5, such a PE-related cluster can include a predetermined number of PE cores, for example, about 16 PE cores, an HBM stack, SRAM, a DMA engine and D2D interconnect. PEs can be configured to access data from a port (e.g., a slave port) to access SRAM, HBM memory and a suitable cache coherent interconnect for processors (e.g., CCG). In particular, the diagram in FIG. 5 represents an example to demonstrate a PE-related cluster, and some differences may exist.

[0217] In some respects, the disclosed systems can include a shared SRAM that can be utilized for temporary activation data storage. For instance, in some example use cases forGPT, for each user token, double the size of the activation space may be required. Forexample, in an example algorithm for GPT-3 175B, the activation / token size can be about 24 KB (12288 x 2 Bytes). To store both the current and previous step’s activations, the disclosed systems may use at least twice this space per token. Continuing with the example, the disclosed systems can support 32 concurrent user sessions, for which the architecture may need about 24KB x 2 x 32 users, which amounts to approximately 1.5 MB of SRAM space for activation storage. However, due to the loading latency of shared memory, the disclosed systems can use TCM since it may be more efficient for activation storage if the PE has a relatively large enough TCM memory.

[0218] In other aspects, the disclosed systems can use a shared SRAM for preloading kernel binaries, which helps reduce kernel launch latency since HBM latency is longer than that of shared SRAM. Additionally, shared SRAM can be used for collecting or buffering partial reduction data from neighboring A-HBM modules and for key -value paging.

[0219] In other aspects, the HBM memory controller can have a predetermined channel usage, for example, for accessing HBM memory. For example, assuming each slave port of AXI supports, in an example, about 32 GB / s Bandwidth per pseudochannel, about 64 AXI slave ports can be configured to provide about 2 TB / s at an approximately 1 GHz clock. Accordingly, in this example, each PE can be logically allocated to 4 pseudochannels.

[0220] In other respects, the M-CPU can be configured to manage kernel launches to PEs and can set input parameter for supporting PE’s kernel execution, then create message for controlling and triggering execution. Depending on kernel operation type, the M-CPU involvement may collect one or more PE generated results or send response or report to the Host.

[0221] In some aspects, the DMA Engine can support multi-purpose data movement. In various respects, some versions of the PEs can include a system architecture that may or may not support cache and coherency. Even in an unsupported case, a CPU, other PEs, or a DMAcan access a PE’s local memory (e.g., via TCM) or registers via the AXI-slave port. This enables the current PE to send its computation results from the local TCM to the neighboring PE’s TCM memory, allowing the PE cluster to perform reduce / gather operations without the need for cluster DMA setup or CPU intervention. In this case, specific TCM region or paging need to be configured accordingly to support requisite memory access.

[0222] Control Flow Example

[0223] FIG. 6A shows an example of control flow for the disclosed systems, in accordance with embodiments of the disclosure. The diagram illustrates a portion of an HBM system architecture, specifically highlighting the interaction between a management CPU, PEs, and HBM. A management CPU is connected to multiple PEs through a communication interface. Each PE has its own associated SRAM memory. The PEs are connected to HBM controllers (i.e., the HBMCs), which manage data transfer between the PEs and HBMs. The HBMCs are also connected to a DMA component and a D2D interface. The HBM is divided into multiple channels, each with its own TSV interface. Overall, this architecture enables for high-performance applications requiring fast data transfer rates and low latency.

[0224] In one embodiment, in more detail, a management CPU is configured to send a predetermined PE MSG to a target PE to execute fine-grained kernels. Examples of such kernels include matrix multiplication (Matmul) operations, such as Matmul(Tl, T2), addition (Add) operations, such as Add(Tl, T2), and other operations like Softmax(Tl.size()), Normalize(Tl.size()), and GeLu(Tl.size()), among others. As one example operational flow, the management CPU can send a message to a queue on a PE, which can then decode the message and issue a read to a HBM. The read command is received by a HBMC, which can retrieve a weight from a corresponding HBM and provide it a PE, which can then determine an associated activation. Further, a completion command can be issued by the PE.

[0225] FIG. 6B shows an example flow chart of the operations of the disclosed systems in reference to FIG. 6A, in accordance with embodiments of the disclosure. At block 601, the management CPU sends a message to a PE (e.g., a PE MSG). At block 602, the disclosed systems can queue the message at a queue of the PE. At block 603, the disclosed systems can decode the message from the queue. At block 604, the disclosed systems can issue a read command to an HBM-C based on the decoded message. At block 605, the disclosed systems can obtain weight from an HBM accessible by the HBM-C. At block 606, the disclosed systems can obtain an activation using information from local memory (e.g., SRAM). At block 607, the disclosed systems can provide, by the PE, a completion to the management CPU.

[0226] FIG. 7 shows an example of a fine-grained matrix multiplication (Matmul) kernel for communication between CPU and PEs for the disclosed systems, in accordance with embodiments of the disclosure. At a high level, the diagram illustrates an example PyTorch's torch.matmul(Tl, T2) operation as performed by the disclosed systems. In some aspects, the example includes, a HOST sending a request to a system’s CPU (e.g., management CPU) to execute the Matmul kernel on a matrix input using PEs. The CPU decomposes the matrix into smaller blocks and assigns them to individual PEs. Each PE performs a portion of the Matmul operation using its local memory (e.g., SRAM) and HBM. The PEs exchange data with each other through direct connections. This fine-grained kernel execution enables efficient parallelization of large matrix multiplications.

[0227] The operation can be described in more specific detail below. As noted, the CPU executes a matrix multiplication operation (e.g., Matmul kernel (Torch.matmul(Tl,T2))) between two tensors, Tensorl and Tensor2. For example, Tensorl may be a weight tensor having dimensions of approximately 32 x 12288, and Tensor2 may be an activation tensor having dimensions of approximately 12288 x 32. The CPU may execute the matrixmultiplication operation by pushing a PE message to a given PE and using the torch. matmul function, which takes Tensorl and Tensor2 as inputs. The PE can then add the message to a queue, then dequeue and decode the message. In one embodiment, the kernel for executing the matrix multiplication operation is loaded from HBM. The HBM may store a plurality of kernels that can be selectively loaded and executed by the PE. By loading the kernel from HBM, the system can efficiently execute complex operations while minimizing data transfer between different memory levels. It can get weights from the HBM and activations from SRAM, and it can set a DMA. The PE can perform matrix multiplications and store results using DMA for example, at SRAM, for retrieval.

[0228] Kernel Based Execution

[0229] In some respects, the disclosed systems can use kernels to perform calculations in parallel so that tasks can be executed by multiple threads in parallel. These threads can be organized into thread blocks, and multiple thread blocks can run concurrently on a given PE. For example, when a kernel is launched, the management CPU manages the distribution of threads to PE. The disclosed systems can be configured to perform such a kernel launch, which can include the process of initiating the execution of a kernel on a computing device such as a PE. In some respects, the kernel launch can involve specifying various parameters, such as the number of threads and thread blocks to execute, as well as any data that the kernel requires for computation. The PE then processes these threads in parallel, with each thread performing a portion of the computation defined by the kernel. The disclosed systems can be configured to efficiently manage and launch kernels to increase the overall performance of parallel applications.

[0230] FIG. 8 illustrates matrix multiplication and tiling concepts for the disclosed systems, in accordance with embodiments of the disclosure. The image depicts a matrix split between multiple HBM devices, PEs, and channels. The left-hand side of the image showsthree large matrices, each representing a portion of the data. The right-hand side of the image shows a detailed view of one of these matrices, split into multiple smaller blocks. Each block is shaded differently, indicating that it is being processed by a different PE or channel. This type of matrix splitting can be used by the disclosed systems in parallel computing applications, such as machine learning and scientific simulations, to distribute the workload across multiple processing elements and improve performance.

[0231] More specifically, in some examples (e.g., as shown in reference to FIG. 8 and also to FIG. 9 discussed further below), a Kernel code can be compiled and launched for an array of PEs (e.g., an M x N associated PE, where here the disclosure provides an example with an 8x4 PE array). In this example, the same kernel can be loaded to M x N PE array. Each PE may need to have a unique node id, E.G. (0,0) through (7,3), as an input parameter before kernel launching. A node identification number (node id (X, Y) ) can be programmed to PE register by the host or given as input parameter.

[0232] In this tensor parallelism example, the distributed PEs can perform a partial operation with partitioned weight tiles. In such a case, a weight tile partitioning rule should be predefined by the developer for distributed PEs. For instance, if weight tiles are partitioned as shown in FIG. 8, then based on given root address of the whole weight data, the Kernel in the node id (0,1) can determined the weight tile’s offset address based on node id and the size of partitioned weight tiles. In this example, the PE in the node id (0,1) can load the weight tiles from HBM and load the activation data from a given address as input parameter. Then all nodes perform operations simultaneously with the different range of weight tiles.

[0233] In some aspects, the kernel can determine what to do when a given operation (e.g., a matmul) computation is completed. For example, once the computation is completed by each PE, the Kernel thread may be configured to gather all the compute result of each PE ((0,0) ~ (7,3)) to the last node (7,3) to complete the given matmul operation by utilizing 8x4array. Then, the kernel can determines the operations, and all node sends their compute result to the last node’s TCM or to the predefined data memory. Once all compute result is gathered, this matmul operation is completed and the system can then provide a response to a requesting device such as the host.

[0234] FIG. 9 shows a single kernel-based execution example for the disclosed systems, in accordance with embodiments of the disclosure. In particular, the diagram shows an example case of a single kernel that includes subsequent graph executions of GPT inferencing including a message passing interface (MPI) operation. The diagram illustrates a single kernel-based execution example for a machine learning model. The kernel is executed on PEs with different coordinates (e.g., (0,0) through (7,3)). Each PE performs a series of operations, including encoding, embedding, add, norm, ReLU and MatMul. Further, after the operations are performed by a given PE, a reduce operation along with an activation and / synchronization operation can be performed before the data is sent to the next PE. The PEs are interconnected, enabling data exchange and synchronization. Finally, the output can be provided along with a completion response to an external device (e.g., host). In various aspects, the diagram shows how data flows between PEs and how operations are executed in parallel. Overall, this diagram provides a detailed view of the execution flow for a single kernel.

[0235] FIGs. 10A and 10B illustrate a fused graph mode execution example for the disclosed systems, in accordance with embodiments of the disclosure. The diagrams illustrate a fused graph mode execution example for a machine learning model. A series of kernels (e.g., kernel 1-3) can be executed on multiple PEs with different coordinates (e.g., (0,0) through (7,3)). As before, each PE performs a series of operations, including encoding, embedding, add, norm and MatMul. The PEs are interconnected enabling data exchange and synchronization. The results from each PE can be subjected to a reduce operation and thenconsolidated. Various processes (e.g., activation) can be performed by other PEs on other AHBM SiP cards as shown in FIG 10B. As illustrated in FIG. 10B multiple kernels can be used to perform the operations and the results can be combined to thereby provide a fused output after a final reduce step. Overall, this diagram provides detailed view of execution flow for fused graph mode.

[0236] In more detail, a fused graph mode execution is shown in which a method of optimizing and executing multiple computational operations or tasks can be performed by combining them into a single, streamlined process. This process can serve to improve performance and efficiency by reducing the overhead associated with executing individual operations separately. In the context of deep learning or neural network computations, a fused graph mode execution involves combining multiple operations, such as normalization, and activation functions, into a single fused operation. By doing so, the system can avoid redundant data transfers between memory and processing units, which can lead to performance improvements. Fused graph mode execution can include graph optimization operations where the disclosed systems analyze the sequence of operations and optimize them for parallelism and minimal data movement. Once the optimization is complete, the disclosed systems can fuse the operations together into a single, composite operation that can be executed more efficiently. This approach helps maximize hardware utilization and minimize latency, resulting in faster and more efficient execution of complex computations. Overall, the disclosed system can use fused graph mode execution to enhance performance by minimizing the overhead associated with executing individual operations separately, enabling more efficient and effective use of computational resources.

[0237] Eager Mode Support

[0238] In some respects, the disclosed systems can be configured to support an imperative and interactive programming environment for working with tensors PE andbuilding computational graphs dynamically (e.g., similar to Eager mode in a developer tool such as PyTorch or other developer tools). In such a mode, the disclosed systems can operate on tensors immediately as they are called, allowing users to see the results and debug code step by step. It is useful for debugging to identify errors and understand how the code works. Assuming this mode launches a short kernel unit one at a time, each kernel can send completion to host when it complete the given computation.

[0239] HBM Access Requirements

[0240] Local pseudo channel group for PE

[0241] FIG. 11 describes parallel access to HBM by PEs for the disclosed systems, in accordance with embodiments of the disclosure. In particular, the diagram 1102 illustrates an AHBM system architecture, specifically highlighting the parallel access to HBM by multiple PEs. In one respect, the management CPU is connected to multiple PEs through a communication interface. Each PE has its own SRAM memory. The PEs are connected to HBM controllers (HBMCs), which manage data transfer between the PEs and HBM. The HBMCs are also connected to a DMA component and a D2D interface. The HBM is divided into multiple channels, each with its own TSV interface. Overall, this architecture can facilitate high-performance applications requiring fast data transfer rates and low latency.

[0242] In various aspects, the disclosed systems enable all PEs to access all HBM channel through a mesh network. However, the maximum efficiency in terms of memory bandwidth utilization can be achieved when each PE access data from their local HBM channel, which can be, in some examples, 4 pseudo-channel, while the PE access HBM with full bandwidth. Further disclosed herein are system requirement for PE core IP about methods to group pseudo-channel for PE.

[0243] In general GPU architectures utilizing L2 Cache, the HBM access pattern involves cache line fills (e.g., 64 B) and random access. Some HBM controllers can achievemaximum bandwidth under these random access conditions, making efficient bank interleaving essential. For a single channel, the maximum number of open banks can be a first threshold (e.g., 4), while using pseudo channels can increase this up (e.g., up to 8). However, increasing the number of open banks impacts power consumption. In the A-HBM architecture, weight data can be fetched directly by the PE without using cache. Therefore, to the disclosed systems can sequentially store the weight data accessed by each PE within a predetermined page (e.g., about IK page) of DRAM with the help of the compiler. This allows the PE to fetch data from the local HBM channel while minimizing the number of open banks (e.g., to 1 or 2). Once a page opens, PEs access HBM via sequential manner that reduces the switching activity of TSV address / command lines. Minimizing the number of active banks and reducing TSV command line (e.g., AWORD I / O) switching by sequential access can contribute to lower power consumption.

[0244] Data Movement via DMA

[0245] FIG. 12A depicts data movement between dies for the disclosed systems, in accordance with embodiments of the disclosure. The diagram 1200 illustrates data movement between dies in an A-HBM system. The system consists of multiple dies, each with its own PE, SRAM memory, and HBMC. The dies are connected through a HBM TSV interface, which enables data transfer between them. A DMA component is also present, facilitating data transfer between the dies and other components. The diagram shows how data moves between the dies using D2D adapters. This architecture supports high-performance applications requiring fast data transfer rates and low latency.

[0246] In more detail, FIG. 12A shows data movement control flow when system requires to copy or transfer data between internal SRAMs. In some respects, the CPU of the disclosed systems can set the DMA engine to transfer memory data from current cluster toneighbor cluster’s memory. Transferring data to other dies or cards can be done by DMA viaD2D interconnects (e.g., UCIe).

[0247] The control flow can include the following. The CPU (e.g., the management CPU) receives data movement command from host (e.g., all-gather between dies or cards). In more granular embodiments, a given PE can include low-latency scratchpad memory such as TCM, and a given PE can use internal DMA to perform similar operations such as gathering operations for lower-level operations to reduce data movement when source data is placed in a given PE’s TCM. In some aspects, if source data sits in shared SRAM, then cluster DMA can be used. In any case, in an aspect, the management CPU can set one or more parameters and initialize the DMA process, for example, by configuring DMA registers (1). The disclosed systems can set the source address with activation location in local memory and set destination address with neighborhood cluster SRAM or other die’s or card’s memory space. The disclosed systems can initiate a DMA operation to read (2) data from local memory (e.g., SRAM at (3) to get an activation) and send it to the destination address (e.g., a neighboring die), for example, via D2D adapter (4). Once it done, the CPU can receive a completion message.

[0248] FIG. 12B shows a flowchart of example operations of portions of the disclosed systems as shown / described in connection with FIG. 12A, in accordance with embodiments of the disclosure. At block 1201, the Management CPU sends a set DMA message to the DMA. At block 1202, the DMA issues a read command to SRAM. At block 1203, the DMA receives an activation from the SRAM. At block 1204, the DMA sends the information to a D2D adapter. The information can then be received at a D2D interconnect of a neighboring die.

[0249] FIG. 13 shows a system for PE kernel execution for the disclosed systems, in accordance with embodiments of the disclosure. The diagram 1302 illustrates a system 1304including PEs (e.g., PEO shown in system 1304) for kernel execution. A PE includes several components such as M-CPU, shared SRAM memory and NOC interconnect. As shown, a given PE (e.g., PEO) can be connected to an address translation unit (ATU), PE scheduler, a CPU such as an ARM-based CPU, and LI cache. In addition, the PE comprises functional units (FUs), Tensor, and Math engines. In some aspects, the ATU can translate Logical Addresses (LA) to Physical Addresses (PA) that correspond to the actual locations in the HBM memory. Overall, this diagram provides detailed view of PE microprocessor architecture.

[0250] In more detail, the PE (e.g., PEO) can include an interface that comprises at least one multiple bus based on AXI for facilitating communication and data transfer. For example, the interface can include channels (e.g., four channels) of AXLM (memory) 256-bit buses operating at about 1 GHz, which can provide a bandwidth of about 128 GB / s (which can result from 32 GB / s x 4 channels) for HBMC pseudo-channel or Shared SRAM via NOC interconnect.

[0251] In another embodiment, the PE Core IP can also include another bus (e.g., an AXLS bus, not shown) between that enables access to internal TCM and registers by the M- CPU and other PEs (e.g., PEI, etc. not depicted). During reduction operations among PEs, this AXLS bus can be utilized for neighbor PEs to forward compute results to each other. The AXLS bus can have a bandwidth of 128 GB / s with a 1024-bit data bus operating at about 1 GHz. Furthermore, in an embodiment, the disclosed systems can include a dedicated bus (e.g., an AXLM bus, not shown) for particular types of operations, such as reduction operations. For instance this dedicated bus might be used by PEO to forward compute results to neighbor PE (e.g., PEI) or even to other PEs on another SIP via D2D interfaces.

[0252] Regarding, the disclosed systems can include, in one implementation, a tensor core functionality thus enabling matrix multiplication with various tensor tile sizes such as,for example, a 32x8, 32x16, or 32x32 configurations, among possibly others not explicitly listed here. The target performance for GEMM operations can be, as an example, approximately 90 TFLOPs per 16 PEs operating at FP16 or BF16 precision, although actual achievable performance may vary depending upon implementation specifics among other factors. In terms of supported data types, the disclosed systems can support FP16, BF16, FP8, INTI 6, INT8 and INT4 and FP32 types and the like.

[0253] In one embodiment, the system can prioritize Exponential (EXP) operations performance, which may be more important for LLM applications. For example, optimized EXP operations can significantly improve the overall processing efficiency of LLMs. In another embodiment, the system can include hardware support for polynomial approximation. This can enable efficient computation of complex mathematical functions using polynomial approximations. Furthermore, in an embodiment, a high-performance divider is provided to accelerate Softmax and LayeredNorm operations. For instance an 8K FP16 divider shared across 16 PE cores could be utilized.

[0254] Regarding kernel execution, in one implementation a CPU based microprocessor (e.g., such as the CPU shown in the diagram) per PE (e.g., PE0) can be provided to manage kernel execution efficiently. For example this CPU might be based on a core with a coprocessor interface supporting custom Instruction Set Architecture (ISA). The CPU can have access to Instruction Tightly Coupled Memory (ITCM) and Data Tightly Coupled Memory (DTCM) (not shown), which can provide fast and low-latency memory access for improved performance.

[0255] The disclosed systems can include a computing system architecture optimized for area efficiency, power efficiency, and performance enhancements. In one embodiment, the system can, as an example, target an area efficiency of about 1.5 TFLOPs / mm2for FP16Multiply-Accumulate (MAC) operations. For example, the disclosed systems can achieve that through the design and layout of the MAC units.

[0256] In another embodiment, the system can also target an area efficiency of 0.5 TFLOPs / mm2for math engines, such as Exponential (EXP) and FP16 divider for Softmax and Layered-Normalization operations. This can enable efficient computation of complex mathematical functions while minimizing area usage.

[0257] In some aspects, the disclosed systems can feature a total available area for a PE cluster with sixteen PEs. For example, the area could be approximately 80 mm2with an additional 10 mm2for redundant PEs. Based on this allocation, a target area per PE could be around 5 mm2with redundant PEs, or 5.62 mm2without redundant PEs, although actual implementation specifics may vary.

[0258] In one implementation, the disclosed systems using the architectures shown and described with reference to the figures above can provide acceleration for various data manipulation operations such as data shuffling, broadcast, transpose, reshape, concat, slice, dynamic slice, and dynamic update slice, among possibly others not explicitly listed here.

[0259] The kernel execution from a core with coprocessor interface may require coprocessor instruction mapping in order to enable seamless interaction between a CPU (e.g., ARM core) and custom accelerators.

[0260] In some aspects, the disclosed systems can provide clock scaling for thermal handling purposes allowing system to dynamically adjust clock frequency and voltage levels based on temperature conditions thereby maintaining optimal performance while preventing overheating.

[0261] In some aspects, the disclosed systems can include components in the PEs (e.g., PE0) such as the tensor units, math engines, and / or other devices that feature a unified multiprecision MACs support from integer INT4 / 8 / 16 to floating point FP8 / FP16 / BF16 data withhigh area and energy efficiency. When the disclosed systems (e.g., at an PE) meets some machine learning layers needing low hardware (HW) utilization, such as depthwise convolution or shallow layers with a few input channels, the system can reconfigure the computational flow to enhance the utilization after getting basic tensor information from a compiler, such as operation types and shapes. The system can include a dynamic operation mode to cover extremely low-power to low-latency requirements (e.g., those below predetermined thresholds).

[0262] PE microprocessor for kernel execution

[0263] In some aspects, the disclosed systems can include PEs that have a small CPU cores for firmware execution and a neural tensor that is connected to NPU cores through dedicated DMAs. The CPU can orchestrate data transfer between the on-chip scratchpad memory (TCM) and an external memory (e.g., HBM, Share SRAM, and the like). In some cases, the CPU can include one or more processors having an a interface to issue custom ISA instruction to a PE scheduler, which orchestrates various operations including, but not limited to, Weight Fetcher, Tensor, Math Engines and output router operations.

[0264] The disclosed systems can be configured to use LI cache (or instruction TCM, ITCM) for kernel execution. In an aspect, a kernel binary code can be preloaded in external memory. In some respects, the disclosed systems can load the kernel binary in HBM and / or Shared SRAM). In some respects, the disclosed systems can use shared SRAM due to short latency as compared with HBM.) This may be due to the fact that, in idle condition, where no access conflict between multiple masters, HBM access latency can have a higher latency than SRAM. For example, HBM access latency can be about 23 to 25 cycles, while Shared SRAM access latency is about 4 to 6 cycles.

[0265] Coprocessor interface

[0266] The disclosed systems can configure the CPU interface. For example, the system can utilize a CPU coprocessor interface featuring custom instructions for coprocessor data engine (CDE) support, which can be implemented on some processors. In an embodiment, the encoding of these instructions in CDE is architecturally defined, allowing for customization of the actual data processing operations. In another embodiment, the CDE contains a predetermined number (e.g., 15) of classes of instructions that support different data types and various numbers of input parameters. Each instruction can include a coprocessor number, destination register, and source register. The disclosed systems provide an alternative option for microprocessors that do not support a coprocessor interface. In such cases, a command queue and response queue can be provided on the PE scheduler. The microprocessor uses store instructions to fill command context in the command queue.

[0267] PE-to-PE IPC message queue

[0268] The disclosed systems enable PE-to-PE communication by configuring a predetermined number of message queues (e.g., up to 32 sets of message queues) using TCM space. This allows PEs to communicate with other PEs or the M-CPU via messages. In an embodiment, the hardware supports Inter-Process Communication (IPC) queues that can include registers or memory locations that hold the addresses (pointers) to message queues, including a head (e.g., 16-bit) and tail (e.g., 16-bit) pointer. The hardware compares these pointers to update not-empty bitmap registers (e.g., 32-bit), which are routinely polled by the CPU to check for incoming messages. In an embodiment, the entry size of each queue can be configurable, with options including, but not limited to, 4 / 8 / 16 / 32B. The hardware implements in an example, 32 sets of queue-config registers, each comprising a queue base offset (e.g., 24 bits), queue depth (e.g., 8 bits), and queue entry size (e.g., 2 bits).Additionally, the hardware includes 32 sets of pointer registers, one bitmap register, and 32 queue-config registers.

[0269] Operation flow

[0270] The disclosed systems include a PE with multiple channels and bus, for example, a 4-channel AXI-M 256-bit at an approximately 1 Ghz bus, providing a predetermined bandwidth, for example, about 128 GB / s (32GB / s x 4 channels). This bus can be configured to connect to HBMC pseudo-channel or Shared SRAM via NOC interconnect.

[0271] In an embodiment, the PE can fetch weight data or activation data from HBM or Shared memory through this interface. The PE DMA then stores the fetched data into TCM. The functional and / or fetch Unit (FU) feeds the weight data to the Tensor unit or Math engine to perform GEMM or Math operations. Once the compute operation is completed, the result data is sent to TCM by the output router. The PE scheduler ensures that all required operations are completed and responds to completion notifications via coprocessor interface (or response queue) with CPU-PE.

[0272] TCM Flushing

[0273] In some GPU architectures that utilize L2 cache, synchronization can be automatically managed by flushing the cache after each kernel execution. In A-HBM architectures that do not employ a cache architecture, manual management of TCM flushing may be required by firmware (FW). In an embodiment, FW manages TCM flushing to ensure that all intermediate data outcomes are properly flushed to designated memory locations, such as HBM or shared memory. This is a crucial requirement before starting the next kernel operation. Additionally, in some cases, host SW may issue TCM flush instructions to guarantee that all intermediate data gets correctly flushed. Thus, the disclosed systems provide a mechanism for manual management of TCM flushing to ensure data consistency and integrity.

[0274] Reduction flow

[0275] The disclosed systems include a PE with a port such as an AXI-Slave port that allows M-CPU (or other PEs) to access internal TCM / registers. In an embodiment, during reduction among PEs, this AXI-Slave port is used to receive compute results from other PEs.The bandwidth for the AXI-S port can be a predetermined amount (e.g., approximately around 128GB / s, which corresponds to a 1024 bit at IGhz data bus due to reduction speed). The PE can be equipped with a dedicated AXI-Master bus for reduction (e.g., with approximately 128GB / s bandwidth), allowing it to forward its computed results to other PEs or SiPs. In one embodiment, when a neighboring PE's compute result is received, the PE performs on-the-fly reduction (FP32 precision) with its own compute result. This reduction operation can be executed in a pipelined manner.

[0276] Kernel Launch

[0277] In some respects, the disclosed systems can serve to minimize kernel launch latency by preloading kernels into HBM or Shared-SRAM. In an embodiment, when the host instructs the launch of a kernel, it sends the kernel index and input parameters to the A-HBM device. The PE then loads the kernel from either HBM or Shared-SRAM. In one embodiment, Shared-SRAM may be used due to its approximately shorter latency compared to HBM.

[0278] Management CPU

[0279] FIG. 14 illustrates a management CPU subsystem for the disclosed systems, in accordance with embodiments of the disclosure. The diagram specifically highlights the components of a management CPU subsystem that can be used in connection with an AHBM die (e.g., similar to M-CPU 455 shown and described in connection with FIG. 4, above). The SoC includes multiple processors, which are dual-core processors. The SoC also features a debug module, shared SRAM memory (e.g., LLRAM based memory). In addition, it comprises of a Generic Interrupt Controller (GIC), a component in ARM-based systems thatmanages interrupts from peripherals to processor cores. Overall, this diagram provides a detailed view of SoC architecture. In more detail, the disclosed systems can have a predetermined target frequency (e.g., frequency of approximately 2 GHz) for some examples. In an embodiment, the system management CPU subsystem that features ITCM and DTCM. Additionally, the management CPU subsystem includes a predetermined amount of shared memory (e.g., around 2MB of shared memory) as shown in FIG. 14 that is accessible through a low-latency interface.

[0280] Features

[0281] The processor can include a mid-performance processor for use in real-time embedded applications. Some example processors herein features a 64-bit capability, based on the architecture. In an embodiment, the system may support a secure memory architecture such as a Protected Memory System Architecture at ELI and EL2 (Exception Levels, which represent different privilege levels for software execution). The processor may also features optional Advanced Single Instruction Multiple Data (SIMD) and floating-point architecture support with two 64-bit data engine pipelines. Each core may have a separate LI data cache and LI instruction cache that are private to it. In one embodiment, two optional TCMs are provided, namely ITCM and DTCM, which can be private to each core. The management CPU processor can include various buses for communication with shared memory, and other management CPUs on different A-HBM SiP cards. For example, the management CPU can include a shared AXI5 approximately 256-bit Main Manager (MM) port for instruction and data access. Additionally, an optional and shared bus (e.g., AXI5) around 256-bit Low- latency RAM (LLRAM) port is provided for instruction and data access. The processor also features an optional and shared AXI5 about 64-bit Shared Peripheral Port (SPP) for peripheral access. Furthermore, an optional per-core AXI5 roughly 32-bit Low-latency Peripheral Port (LLPP) is provided for peripheral access. Optionally, the disclosed processorcan feature ECC, Single Error Correct or Double Error Detect protection for all of the instantiated cache tag and data RAMs and the TCM RAMs.

[0282] Debug Features

[0283] The disclosed systems can include a M-CPU that features a debug module (e.g., as shown in FIG. 14) that can have a range of debug features, including support for debugging over power-down. In an embodiment, the system includes an Embedded Trace capabilities for instruction and data trace. The system also supports performance monitors extension for software profiling and performance debugging. A Cross Trigger Interface (CTI) is provided for multiprocessor debugging. In one embodiment, optional support is available for integrating embedded logic analyzers, which provides advanced debug capability and signal observability. Additionally, the system features a MBIST for testing memories at boot time.

[0284] Cluster DMA

[0285] FIG. 15 shows a cluster DMA diagram for the disclosed systems, in accordance with embodiments of the disclosure. The cluster DMA system 1502 can include a type of interface used for high-speed data transfer between different components shown in the system. The cluster DMA system can include at least two channels, each with its own instance of the DMA (e.g., cluster DMA instance 0 1509 and cluster DMA instance 1507). The channels are labeled as Channel 0 1503 and Channel 1 1505. Each channel has several components, including a FIFO (First-In-First-Out) buffer, finite state machines (FSMs) for source and destination control, and AXI master interfaces 1523. The AXI master interfaces are used to connect one cluster DMA (e.g., cluster DMA instance 0 1509) to other components in the system, such as memory (e.g., local SRAM or neighborhood SRAM) or peripherals. The diagram shows the flow of data through the cluster DMA system, including the transfer of data between channels and the control signals used to manage the transfer.Overall, the cluster DMA system appears to be designed for high-performance applications requiring fast data transfer rates and efficient data management.

[0286] In more detail, the cluster DMA can be used with management CPU firmware (e.g., that M-CPU shown and described above in connection with FIG. 14). In an embodiment, the cluster DMA includes specialized DMA controller IP. One function of the cluster DMA is to perform memory-to-memory DMA transfers. In some aspects, the cluster DMA includes 2 unidirectional channels. One channel is used to transfer data from local SRAM to neighborhood SRAM in a different cluster, which may be located in the same die, a different die, or a different card. The other channel is used to transfer data in the opposite direction, with reverse source and destination pair.

[0287] In one embodiment, firmware programs certain registers to provide a source address, a destination address, and the number of bytes (approximately) to be copied for a single-block DMA transfer. A block of data may be broken into a sequence of burst transfers on the AXI bus interface. The disclosed systems can support multi-block DMA transfers. In an embodiment, firmware prepares multiple linked list items in local SRAM, which are fetched by the controller at the beginning of every block. During a multi-block transfer, one linked list item is fetched at a time while the data transfer for the previous linked list item is in process. Each linked list item contains a block descriptor, including source address, destination address, and approximately number of bytes to be copied), as well as a pointer that points to the location in local SRAM where the next linked list item exists. In some respects, the DMA controller outputs a predetermined number (e.g., 3) bits of interrupt. There is one combined channel interrupt bit for each channel and one combined common interrupt bit. Interrupts are generated on transfer completion, error conditions, and so on. Further, the disclosed systems can read the different interrupt status registers to understand the source of the interrupt and take appropriate actions.

[0288] In terms of data flow, the disclosed systems can configure each channel to read the data from a source and store the data in a dedicated channel FIFO temporarily, before reading it out from the FIFO memory and writing to a destination. The controller supports unaligned transfer, by handling the packing and unpacking of the data considering the unaligned address offset at both source and destination.

[0289] The Cluster DMA can include the following features, which are offered by way of example (but which can vary and substantially differ based on design choice). The bandwidth can be approximately 128 GB / s. In an embodiment, the Cluster DMA has around 2 AXI masters. The AXI data bus width can be roughly 32 bytes. The clock frequency of the Cluster DMA can be about 2GHz. There can be approximately around 2 DMA instances and roughly 2 channels. The channel FIFO size can be approximately 256 (depth) x around 32 bytes (width). The number of AXI outstanding requests is approximately 16. The maximum burst size is approximately equal to 16 x 32 bytes. Similarly, the maximum single block transfer size can be, in an example, about 32 kB. In one embodiment, the Cluster DMA supports unaligned transfers on the AXI interface. The disclosed systems can also support multi-block transfers as well as interrupt generation.

[0290] HBM 3D PHY

[0291] In some embodiments, a predetermined number (e.g., 32) of channel DFI signals can be connected between HBMC and an HBM 3D DRAM PHY. The 3D DRAM PHY receives DFI signals and converts them into TSV interface (I / F). During this process, if the DFI clock and TSV clock frequencies are identical, there may be no need for a serialization or de-serialization process between DFI I / F and TSV I / F.

[0292] FIG. 16 depicts a block diagram of a 3D-DRAM PHY for the disclosed systems, in accordance with embodiments of the disclosure. At a higher level, the diagram depicts a technical architecture for a 3D DRAM system, focusing on its physical layer andTSV connections. The 3D DRAM PHY (Physical Layer) component is comprised of multiple channels, including four channels of memory physical layer interfaces (SM PHY 1CH) and a high-speed physical layer interface with four channels (HM PHY 4CH). The TSV connections play a crucial role in enabling vertical communication between layers of the 3D stacked memory. Specifically, TSV AWROD and TSV DWROD are used to connect different layers. The architecture features multiple instances of the 3DDRAM PHY component, indicating a stacked architecture with multiple layers of DRAM. The GLOBAL TSV TOP component is also present, related to global TSV connections at the top layer. The diagram shows multiple APB (Advanced Peripheral Bus) interfaces (APB 0-3) connecting to different components. These interfaces enable access to peripherals and facilitate data transfer. Overall, this architecture enables high-performance and low-latency memory access in 3D stacked systems. The use of TSVs enables increased memory density, while the high-speed interfaces support fast data transfer rates.

[0293] With more specificity, the diagram shows HBM 3D DRAM PHY to support 32 channels. As depicted in the figure, there are multiple 3D DRAM PHY which supports 4 channels. Some features of HBM 3D DRAM PHY are summarized as follows. In an embodiment, the HBM 3D DRAM PHY supports HBM3 C-die specification or similar specifications. The system also supports approximately 4 channels. In one embodiment, the frequency ratio is 1 : 1. The HBM 3D DRAM PHY features a fully digital DLL for controlling a relevant strobe signal. The disclosed systems include a feedback loop-back test scheme for at-speed data and control channel testing. In addition, TSV repair control is supported.

[0294] DFLto-TSV conversion

[0295] FIG. 17 illustrates a digital front-end interface to through-silicon vias (DFLto-TSV) read timing diagram for the disclosed systems, according to an embodiment of the disclosure. In more detail, the diagram shows the DFI and TSV timing for the read operationin the pseudo channel mode. Once the read data is on the TSV bus, these data is converted to DFI by 3D DRAM PHY. The waveform diagram illustrates the behavior of various signals in a digital system, as related to the disclosed system’s memory and / or high-speed interconnects. The DFI signals include DFI clk, Dfi command, Dfi rdata en, andDfi rdata. The Dfi command signal includes several protocol layer commands represented as PCO, PCI, PC2, and PC3. These commands affect the state of other signals in the system. The bank / row / COL / CRC control signals include BRCK (Bank Check) signals that indicate bank selection or control. There are also CRCK (CRC Check) signals that indicate CRC operations. The DQ (Data) bus is used for actual data transfer and consists of multiple bits. The DQ_E signal indicates when data is being transferred on the DQ bus. Overall, this waveform diagram represents a part of a larger SoC or memory controller design. It emphasizes interaction between different components in digital systems.

[0296] FIG. 18 illustrates a DFI-to-TSV write timing diagram for the disclosed systems, according to an embodiment of the disclosure. The waveform diagram illustrates the behavior of various signals in a digital system, likely related to memory or high-speed interconnects. The DFI signals include DFI clk, Dfi command, Dfi wdata en, and Dfi wdata. The Dfi command signal includes several protocol layer commands represented as PCO and PCI. These commands affect the state of other signals in the system. The write data enable signal (Dfi wdata en) is asserted, allowing data to be written on the Dfi wdata bus. The DQ (Data) bus is used for actual data transfer and consists of multiple bits organized into four groups: DQ BL04, DQ BL15, DQ BL26 and DQ BL37. The figure shows the DFI and TSV timing for the write operation. Once the valid write data is on the DFI bus, the write data and other signals are converted to TSV signals. In this example, the latency for write data, from DFI wdata bus to TSV wdata bus, is 2 elk as the buffer dies consumes 2 elk for converting DFI to TSV. Overall, this waveform diagram represents a part of memorycontroller design and shows the interaction between different components in the digital subsystems of the present disclosure.

[0297] HBM Memory Controller

[0298] The disclosed systems can use HBM comprising high-speed 3D stacked Synchronous DRAM. The HBM4 uses a wide-interface architecture to achieve high performance. The disclosed systems can use the wide-interface which can be divided into independent channels where each channel is independent of one another. That is, channels can be independently clocked and need not be synchronous. According to some standards (e.g., JEDEC Standard No. 238 A), each channel interface can maintain a 64-bit data bus (DQ) operating at double data rate. The figure below from JEDEC Standard shows the general overview of a DRAM Die Stack with Channels, which can be configured to be used with a base die (e.g., an example AHBM architecture as disclosed herein).

[0299] FIG. 19 provides a general overview of a DRAM die stack with channels for the disclosed systems, in accordance with embodiments of the disclosure. The diagram depicts a stacked architecture of DRAM dies with multiple channels, which can be used in the AHBM system. The stack consists of multiple DRAM dies, each with 4 channels as an example. Each channel is further divided into smaller blocks. The stack also includes an optional base die that can include logical circuitry and functionality. This architecture enables increased memory density and bandwidth.

[0300] In some embodiments, the disclosed systems can use HBM, which can include HBM4 or HBM4E and can further use an HBM4-compatible memory controller to manage the memory. An example HBM4 memory controller may support a range of HBM4 memory devices. Specifically, the memory controller may support HBM4 memory devices having approximately 2, 4, 8, 12, or 16 stack heights (or more). The controller may also support all standard HBM4 channel densities up to about 32 GB. In one embodiment, the memorycontroller operates at a data rate of up to approximately 10 Gbps per pin, with a total of approximately 2048 DQ data pins. The controller may also support HBM4 RAS features. In one embodiment, the memory controller includes a DFI (Digital Front-end Interface) compatible interface between the controller and PHY. The controller may also provide end- to-end data parity protection to ensure data integrity. In one embodiment, the memory controller includes an interface to user logic that is either AXI or native. This allows for flexible integration with various SoC architectures.

[0301] In some embodiments, a block diagram of the HBM4 subsystem example (showing a Memory controller, PHY, and HBM device) is shown in FIG. 20. In particular, FIG. 20 depicts an HBM4 subsystem for the disclosed systems, in accordance with embodiments of the disclosure. The diagram illustrates a HBM system architecture, specifically highlighting the interaction between a requester, HBM controller, HBM PHY, and HBM DRAM. The requester is connected to 32 instantiations of the HBM controller. The HBM controller manages data transfer between the requester and the HBM DRAM through the HBM PHY. The HBM PHY is a critical component that enables high-speed data transfer between the HBM controller and the HBM DRAM. It supports multiple data rates, including 2 GHz, 4 Gbps, and 8 Gbps as an example. The system includes multiple channels for parallel data transfer. Each channel has a specific bandwidth. Overall, this architecture enables high-performance applications requiring fast data transfer rates and low latency.

[0302] In one embodiment, an A-HBM device includes design-for-test (DFT) features to support testing and failure analysis. Specifically, the A-HBM device may support CoW testing after TSV stacking or failure analysis after package assembly, similar to custom HBM buffer dies. In one embodiment, the HBM DFT blocks comprise DA, MBIST, and IEEE 1500 interfaces to support both DRAM core testing and packaging connectivity testing. In one embodiment, the DA feature provides a separate test interface that allows a vendor to accessthe HBM DRAM independently of the host. The DA feature provides a path for direct access to the HBM DRAM at the SiP level via DA balls (pBumps) and offers a path for conducting Low-Frequency (LF) and High-Frequency (HF) tests at CoW level via DA pads.

[0303] In one embodiment, the DA interface includes approximately 40-bit DA Balls (pBumps), denoted as DA[39:0] (not shown), and approximately 320 DA pads. Note that the number of pads may vary depending on the specific HBM generation, such as HBM3 or HBM4.

[0304] In particular, FIG. 21 shows a DA Ball signal path at SIP level and DA pads at CoW level for the disclosed systems, according to an embodiment of the disclosure. The DA Ball signal path at SiP level is connected to the DA pads at CoW level through a silicon interposer. The DA pads are located on the surface of the chip and can be used for testing and debugging purposes. The diagram shows multiple HBM chips connected to a core through PHY interfaces, which can be used by the disclosed systems to access HBM from the PEs. The HBM chips have multiple banks of memory, and each bank has a corresponding DA ball signal path.

[0305] In one embodiment, the DA feature of an A-HBM device includes a range of testing and validation capabilities. Specifically, the DA feature may include: A means for testing the DRAM core die via DA balls and DA pads; A high-frequency test solution that supports the use of low-frequency test equipment via DA pads; and Control interfaces for IEEE 1500 and MBIST. Test coverage that includes, but is not limited to, HBM DRAM core testing, TSV connectivity testing, package connectivity testing, PHY testing, and other related tests.

[0306] In one embodiment, these DA features enable comprehensive testing and validation of the A-HBM device, including its DRAM core, TSVs, package connectivity, andPHY. By providing these test capabilities, the DA feature helps to ensure the reliability and functionality of the A-HBM device.

[0307] FIG. 22 illustrates HBM4 DA Ball allocation for multiple HBM4 DRAM devices for the disclosed systems, in accordance with embodiments of the disclosure. The diagram illustrates the allocation of DA pins for multiple HBM DRAM devices. In this configuration, four HBM4 DRAM devices are connected to a buffer die, which is then connected to a silicon interposer. Each HBM4 DRAM device has multiple stacks of DRAM.

[0308] The DA pins of an A-HBM device, such as HBM4, are allocated for point-to- point and multi-drop connections. In one embodiment, the DA pins are divided into two groups: point-to-point and multi-drop. The point-to-point group includes approximately 8 DA pins, which can be denoted as DA[19: 12] (not shown). The multi-drop group includes approximately 32 DA pins, which can be denoted as DA[39:20] and DA[11:0] (not shown).

[0309] In one embodiment, for CoW testing, a HBM device, such as HBM3, can include a predetermined number of pads, for example, approximately 320 vendor-specific DA pads. The DA pads are allocated for various purposes, including control / address / data / clock signals (CA / RA / DQ / Clock / etc.), serializer-deserializer (SERDES) signals for high-frequency (HF) testing, and power pins. Specifically, approximately 43 PADs are dedicated to CA / RA / DQ / Clock / etc., operating at approximately 250 MHz.Approximately 66 pads are dedicated to SERDES signals for HF testing, also operating at approximately 250 MHz. The remaining pads, approximately 211 in number, are dedicated to power pins.

[0310] In one embodiment, the DA pad assignment can be configured accordingly (e.g., DA pad, ball, and IEEE1500 port configurations for HBM3). Depending on the selection of a specific signal (e.g., via a Wrapper Serial Port (WSP) signal), the DA interface can be used either for regular test mode via DA balls or IEEE1500 interface mode. In oneembodiment, the test flow for a HBM device can be grouped into three categories: CoW testing, SiP failure analysis, and MBIST. In an embodiment, the CoW test flow involves testing the HBM device through the DA pads. The test flow can include a sequence of operations including accessing the DA pads, serializing the data, expanding the channel / data queue (CH / DQ), transmitting the data through a physical layer, and testing the core.

[0311] In one embodiment, the SiP failure analysis test flow involves testing the HBM device through a DA bump or ball. The test flow comprises a sequence of operations including accessing the DA bump or ball, expanding CH / DQ signals and transmitting them to PHY for core.

[0312] In one embodiment there can be two MBIST paths. The first MBIST path involves testing through DA pad / bump as IEEE1500 interface. The sequence of operations can include accessing DA pad / bump via IEEE1500, executing MBIST, serializing data, expanding CH / DQ signals, and transmitting them to PHY for core. In another embodiment, a second MBIST path involves testing through an IEEE1500 bump via interposer. The sequence of operation includes accessing IEEE1500 bump via interposer, executing MBIST, serializing data, expanding CH / DQ signals, and transmitting them to PHY for core testing.

[0313] FIG. 23 shows a signal paths of DA ball, DA pad, and IEEE1500 pBump for the four test flows described above. In particular, FIG. 23 describes DA pBump (ball) and DA signal path for the disclosed systems, according to an embodiment of the disclosure. The diagram illustrates the signal path (1-4) for DA pBump (ball) and DA pads in the AHBM system. The DA signal path originates from the DA area and passes through various components, including PLL, DIS, MUX, and PHY. The signal then reaches the TSV area and finally connects to the core die. The diagram shows multiple DA pads, including DA pads (ANI), DA pads (W), and others. These pads are used for testing and debugging purposes. The signal path is also connected to a CoW testing interface, which enables testing of theHBM system at the CoW level. Overall, this diagram provides a detailed view of the DA signal path and its connections to various components in the HBM system.

[0314] In one embodiment, an A-HBM system as disclosed can be configured to include an IEEE1500 Wrapper that provides a standardized interface for testing and debugging. The IEEE1500 Wrapper contains a standard register stack, including a Wrapper Bypass Register (WBY), a Wrapper Data Register (WDR), and a Wrapper Instruction Register (WIR), as well as an instantiation of a MBIST controller. In one embodiment, the IEEE1500 Wrapper can be accessed through various interfaces, including IEEE1500 microbumps from the compute die, DA pads, or DA microbumps. This allows for flexible testing and debugging of the A-HBM system. In one embodiment, the IEEE1500 test features include 47 instructions that support various test functions. These test features include boundary scan, I / O loopback testing, MBIST execution, memory cell repair mechanisms, lane repair capabilities, chipping detection, device identification, temperature monitoring, channel identification, channel disablement control, Write Data Strobe (WDQS) alignment verification, and Error Check and Scrub (ECS) Error handling.

[0315] MBIST Test Features

[0316] In one embodiment, the disclosed systems can use a MBIST system that provides at-speed testing of memory cells and cores through an IEEE1500 interface. The target frequency of the MBIST clock can be, for example, approximately 312.5 MHz. In one embodiment, the MBIST system is responsible for generating test patterns, detecting fail addresses, and facilitating cell repair. The MBIST system supports two types of test patterns: march patterns for detecting "hard defects" and scan patterns for detecting "retention failures." These test patterns can be broadcast to up to 32 channels, allowing for optimized test time. In one embodiment, the MBIST system operates in two modes: SPEC mode and User-Programmable mode. In SPEC mode, pre-defined march and scan test patterns are used,and the mode is not user-programmable. In user-programmable mode, users can create custom test sequences with programmable memory addresses and write / read data values. In one embodiment, the MBIST system includes an internal Built-In Self-Repair (BISR) mechanism that operates at frequencies under a threshold, such as approximately 3.2 Gbps. When running an MBIST sequence at lower frequencies (under 3.2 Gbps), the BISR mechanism is deployed. However, when running an MBIST sequence at higher frequencies (over 3.2 Gbps), a specific instruction (e.g., an IEEE1500 HARD REPAIR instruction) may be used instead.

[0317] Test Methods

[0318] FIG. 24 (including FIGs. 24A and 24B) outlines A-HBM test coverage scenario for the disclosed systems, in accordance with embodiments of the disclosure. The diagram illustrates the test coverage for an A-HBM system. The A-HBM system includes various components such as PHY BIST, SCAN TEST, and HBM DFT TEST. These components are used to test the system's physical layer, scan chains, and DFT logic. The diagram shows multiple instances of the 3D DRAM TSV interface, which connects to the core die. The TSV interface is used for testing the DRAM's connectivity and functionality. TheHBM DFT TEST component is responsible for testing the A-HBM system's DFT logic. This includes testing of various interfaces. Overall, this diagram provides a view of the A- HBM system's test coverage.

[0319] In one embodiment, the disclosed systems can includes DFT blocks that comprise DA, MBIST, and IEEE1500 interfaces. These DFT blocks support both DRAM core testing and packaging connectivity testing. The HBM DFT test methodology includes several test items, such as HBM DFT test using DA, IEEE1500, and MBIST interfaces. The methodology can involve TSV lane testing and DRAM core die testing using MBIST. Additionally, scan testing is performed to cover all PEs, NOCs, M-CPUs, and HBMcontrollers. The scan test also covers all digital domain logic tests at-speed. Furthermore, 3D DRAM PHY testing can be performed, which includes internal and external loopback testing of the DRAM PHY. Die-to-die PHY BIST testing is also conducted, which involves internal loopback BIST for wafer testing and external loopback BIST for lane testing on interposers after assembly.

[0320] Die-to-die interconnect

[0321] In A-HBM architecture, UCIe-Advance package (32Gbps / lane) is employed for die-to-die interconnects for multiple compute dies on SiP. FIG. 25 shows an example diagram with mesh network. Coherency mesh network (CMN) gateway interface 2503 can be used and can be connected in UCIe with Credited extensible Stream (CXS) interface to connect one die (e.g., die 2502) to the gateway interface 2503.

[0322] As noted, FIG. 25 depicts a die-to-die interconnect for the disclosed systems, according to an embodiment of the disclosure. The diagram illustrates a high-bandwidth interconnect architecture, specifically highlighting the use of UCIe for die-to-die communication. The architecture consists of multiple chiplets that can be located on a given A-HBM device (e.g., on the base die of the A-HBM device), each with its own UCIe interface. These chiplets are connected using UCIe, enabling high-speed data transfer between them. The diagram shows two instances of UCIe, each with multiple lanes. These lanes are used to transmit data between the chiplets. The use of UCIe enables scalable and flexible interconnects for heterogeneous systems.

[0323] System requirement

[0324] In one embodiment, the disclosed systems can use an interface such as UCIe.For the interface, the disclosed systems can includes error handling support with a bit error rate (BER) at or below a predetermined threshold (e.g., le-15). In one embodiment, to further enhance reliability, additional forward error correction (FEC) mechanisms may be employedto achieve an even lower BER than le-15. In one embodiment, power management is optional. Although power management states are not mandatory for streaming protocols in the UCIe specification, the disclosed systems can use another protocol such as CXS (Chiplet eXtend Serialization) to detect interface activation and deactivation states. This enables the disclosed systems to use UCIe to estimate power-saving states.

[0325] Component description

[0326] In one embodiment, the disclosed systems can operate with a layered protocol such as Universal Chiplet Interconnect Express (UCIe) protocol, with each layer performing a distinct set of functions. This layered architecture enables a modular and scalable design. In one embodiment, the disclosed systems can operate with a UCIe stack including three main components, each responsible for specific functionalities. A figure illustrating the UCIe layers and their respective functionalities is provided below.

[0327] FIG. 26 explains UCIe layers and functionalities for the disclosed systems, in accordance with embodiments of the disclosure. The diagram depicts a layered architecture for D2D communication, specifically highlighting the Flit-aware D2D Interface (FDI) and Raw D2D Interface (RDI). The architecture consists of three layers: Protocol Layer, Die-to- Die Adapter, and Physical Layer. The Protocol Layer interacts with the Die-to-Die Adapter through the FDI. The Die-to-Die Adapter performs several functions, including arbitration / multiplexing, cyclic redundancy check / retry, link state management, and parameter negotiation. The Physical Layer interacts with the Die-to-Die Adapter through the RDI and performs functions such as link training, lane repair / reversal, scrambling / de- scrambling, sideband initialization / transfers, analog front-end management, and clock forwarding. Overall, this architecture can be used for high-performance die-to-die communication in 3D stacked systems such as the AHBM system.

[0328] In more detail, the disclosed systems can use the Protocol Layer of the UCIe protocol stack. This include support for various protocols, including PCIe 6.0, CXL 2.0, and CXL 3.0 protocols, and the like, enabling plug-and-play compatibility with existing ecosystems, including Control and Status Register (CSR) compatibility. Additionally, the disclosed systems can support streaming protocols for raw bits and connects to a D2D adapter through the interface. In one embodiment, the disclosed systems can use the Die-to-Die Adapter Layer to perform several functions. For example, the disclosed systems can enable initial protocol / parameter negotiation. Further, the disclosed can perform multiplexing multiple protocols when needed.

[0329] In one embodiment, the disclosed systems can use a Die-to-Die Adapter Layer for reliable data transfer through mechanisms such as cyclic redundancy check (CRC), retry, and flit. In one embodiment, the disclosed systems can use a PHY Layer which can further include several sub-layers, including PHY logical, electrical / Analog Front End (AFE), and sideband / global aspects. Moreover, based on package type, lane reversal or width degradation can be supported by the system.

[0330] The disclosed systems can use UCIe to support different flit modes based on the protocol. For instance, to minimize latency, three flit modes can be employed: CXL 256B flit formats, Latency-Optimized 256B flit formats, and Raw formats with custom Forward Error Correction (FEC) for tunneling AXI or AXI-Stream protocols.

[0331] Package Description

[0332] In one embodiment, the packaging technology disclosed herein is utilized for performance-optimized applications. As a result, a channel reach associated with the D2D interconnects may be relatively short, typically less than about 2 mm. Consequently, the interconnect can be configured for high bandwidth and low latency, ensuring optimalperformance and power efficiency characteristics. In one embodiment, the main characteristics of this packaging technology can be summarized as follows.

[0333] In some example embodiments, the supported speeds per lane for the interconnect can be approximately 4 GT / s, 8 GT / s, 12 GT / s, 16 GT / s, 24 GT / s, and 32 GT / s, and the like. In one embodiment, the bump pitch for the interconnect ranges from approximately 25 um to 55 um. In an embodiment, the channel reach for the interconnect can be approximately 2 mm. In one embodiment, the raw BER for the interconnect is targeted to be around le-27 for speeds less than or equal to about 12 GT / s and around le-15 for speeds greater than or equal to about 16 GT / s.

[0334] In one embodiment, the disclosed systems can include 3D TSV stacking technology. Specifically, the use of 3D TSV technology can introduce certain design considerations. For example, the area occupied by HBM TSVs and HBM core die test-related logic on the logic die represents one such consideration. The 3D DRAM PHY and TSV landing area on the logic die can be another. In one embodiment, the disclosed systems can include specific considerations for the TSV area for Signal TSV and Power TSV.

[0335] Routing limitation by HBM TSV related logic

[0336] Given the central positioning of TSVs in the structure of the HBM die, the alignment of HBM 3D PHY and TSV positions on the logic die may represent a design consideration. Consequently, HBM TSVs and control logic may need to be sandwiched between PE networks. As a result, constraints on network routing between upper and lower Pes may arise, centered on the HBM TSVs.

[0337] FIG. 27 shows Signal and Power TSVs, for the disclosed systems according to an embodiment of the disclosure. The diagram illustrates a 3D stacked architecture, specifically highlighting the use of TSVs for signal and power transmission to a given portion of the A-HBM device (e.g., HBM memory stack and / or base die components). In oneembodiment, the signal TSVs are used for transmitting signals between different layers of the stack (e.g., HBM memory stack). In one embodiment, the power TSVs are used for supplying power to different components of the stack. Hence, the disclosed systems can optimize routing by utilizing the space between Signal TSVs and Power TSVs. In some aspects, each Signal TSV can occupy a predetermined area, for example, an area of about 21.1um x 22.35um per I / O, with about 7.4 um of available room between Signal TSVs. Moreover, each Power TSV occupies a predetermined area, for example, an area of about 21.2um x 14um per I / O, with 14.5um of available room between Power TSVs.

[0338] Power TSV for HBM

[0339] The TSVs location can have a predetermined size, such as a package size of, for example, about 14.2 mm x 11 mm (e.g., as per HBM4E in some examples). In other respects, the disclosed systems can use different TSV locations and / or sizes, since the TSV map can be changed as the core die architecture is changed.

[0340] FIG. 28 illustrates a middle TSV for the disclosed systems, in accordance with embodiments of the disclosure. The diagram shows multiple TSVs connecting different layers of a 3D stacked chip. The middle TSV is shown, indicating its significance in the AHBM system. In one embodiment, the middle TSV can be designated as the HBM4 TSV landing bump for main command and data I / O. In another embodiment, four H' shape TSV regions can be occupied by power TSV lines, which supply power from the base die to the DRAM core die. In one embodiment, the power TSV area, similar to the middle TSV area, can be reserved for power supply purposes for the core die stack. In one embodiment, the physical layout may avoid this space as much as possible (e.g., preventing logic or metal routing from passing through that area).

[0341] Thermal Handling

[0342] In one embodiment, to optimize the placement of PEs within a base die, it may be desirable to avoid positioning them directly beneath a DRAM core die. The maximum tolerable temperature for a DRAM core die can be approximately 95°C. If the temperature exceeds a predetermined threshold such as about 85°C, the refresh rate must be increased to ensure data retention. In one embodiment, increasing the refresh rate can result in higher power consumption. Therefore thermal throttling may be used starting at 85°C. In one embodiment, the PE within the base die performs high-speed tensor operations generating significant heat. Placing PEs directly beneath DRAM core dies can elevate temperatures. In one embodiment, to mitigate thermal impact, PEs may not be overlapped with DRAM core dies when positioning PEs on base dies. However, in reality, perfectly avoiding overlay may be difficult unless the base die size is increased. In some aspects, the base die size may be constrained by the interposer size. Therefore, the overlay may be minimized by placing the PEs as far towards the outer edges of the base die as possible.

[0343] Molding material

[0344] In one aspect, the molding material used in the HBM stack can include a type of plastic that provides structural rigidity. However, due to its low thermal conductivity, heat from the base die in direct contact with the molding material may not dissipate effectively. In one embodiment, to address this challenge, molding materials with different thermal conductivity can be used (e.g., epoxy, silicon, polyurethane, polyimide, silicon carbide, aluminum oxide, etc. based materials can be used).

[0345] Dynamic thermal throttling

[0346] In one embodiment, stacking the HBM core die vertically on top of the logic die using TSVs can pose challenges in terms of heat dissipation. This structural configuration can create a thermal bottleneck, as the HBM core die may obstruct the heat dissipation from the logic die. In one embodiment, the high-speed operating compute (e.g., the PEs), HBMcontroller, and TSV area can be potential hotspots on the logic die. In one respect, the highspeed compute core beneath the HBM core-die may be susceptible to thermal issues.

[0347] In one embodiment, to mitigate these thermal challenges, a dynamic thermal throttling mechanism is used. This can be achieved by actively utilizing thermal sensors in both HBM core-die and base-die. A management CPU (M-CPU) can track thermal sensor values from HBM memory. Further, the M-CPU can check if the temperature is greater than a certain threshold, then gear down the PE clock speed with a predetermined gratuity (e.g., 1 GHz down to 800 MHz). The M-CPU can monitor temperature changes to see if the system cools down. If the system is not cooled down to a predetermined level in certain period of time, then gear down PE clock one more step (e.g., 800 MHz down to 600 MHz).Accordingly, the system can be incrementally throttle PE performance to achieve a given temperature.

[0348] In one embodiment, a dynamic thermal throttling mechanism is implemented to manage the temperature of a PE) and HBM. The system can adjust the PE clock stepping level and monitor temperature sensor values from the HBM. In an embodiment, the PE clock and HBM controller clock (DFI clock) are adjusted up or down until a sustainable temperature is reached. In one embodiment, this process may require adjustments to HBM parameters such as clock speed, memory capacity, and / or the like. In another embodiment, the PE power budget is set at a predetermined amount (e.g., about 22 W for 16 PEs), with a predetermined usage, for example, a usage of about 4 TFLOPs / W at FP16.

[0349] Level 2: Sy stem -in-Package

[0350] In some respects, an AHBM SiP card can feature multiple AHBM units / devices that can be configured on a package which can further use am interposer, e.g., a silicon interposer. In some aspects, the silicon interposer can have a predetermined size.

[0351] FIG. 29 depicts an A-HBM SiP card / package 2900 with a silicon interposer for the disclosed systems, according to an embodiment of the disclosure. It shows the network topology for the system, specifically highlighting the use of UAL 2902 and PCIe interfaces. The topology consists of multiple nodes (e.g., AHBM SiP card / package) interconnected by lines representing connections. The nodes are arranged in a grid-like structure, with some nodes connected to UAL interfaces on the left and right sides, and one node connected to a PCIe interface on the top-right side. This topology cam be used to enable high-bandwidth and low-latency communication between nodes in the system.

[0352] Alternatively, the disclosed systems can use heterogeneous integration technology that horizontally places one or more logic dies (CPU, GPU, etc.) and several HBM dies on top of a silicon interposer, making multiple dies operate as a single chip in one package, to lift silicon interposer size limitation. The disclosed systems can make a bit bigger buffer die to help thermal dissipation.

[0353] FIG. 30 shows an A-HBM SiP package / card 3000 with silicon interposers for the disclosed systems, in accordance with embodiments of the disclosure. The diagram depicts a 16 A-HBM SiP package which can use silicon interposers. In one embodiment, the package can include a type of 3D stacked package that integrates multiple A-HBM dies with a silicon interposer. The silicon interposer acts as a bridge between the A-HBM dies and the PCB, enabling high-speed data transfer and reducing latency. In some respects, the package has several features. Multiple A-HBM dies: The package contains 16 A-HBM dies. Silicon interposer: The silicon interposer is used to connect the A-HBM dies to the PCB. High-speed interfaces: The package supports high-speed interfaces such as UAL and PCIe. Overall, the package enables high-performance applications that require fast data transfer rates and low latency.

[0354] In some aspects, the disclosed systems can feature an A-HBM architecture that allows flexible configurations depending on the performance needs. FIG. 31 illustrates various configurations (e.g., minimal 2 x 2 A-HBM SiP or more complex 3 x 3 A-HBM SiP) of the disclosed systems depending on performance needs, according to an embodiment. In one embodiment, the disclosed systems can use UALink as a communication medium for chip-to-chip and tray-to-tray interactions. The diagrams illustrate two different example architectures for A-HBM systems. In one embodiment, the left diagram 3102 shows a simpler architecture with a single A-HBM SiP comprising 4 AHBM units / devices connected to UAL 3102 and PCIe 3104 interfaces. In one embodiment, the right diagram 3104 shows a more complex architecture of an A-HBM SiP 3104 with multiple A-HBM units / devices (e.g., AHBM unit / device 3116) connected to UAL 3112 and PCIe 3114 interfaces. The use of UAL enables high-speed communication between the A-HBM modules and other components. The PCIe interface provides connectivity to external devices.

[0355] FIG. 32 (i.e., FIG. 32A and 32B) depicts an A-HBM system for the disclosed systems, in accordance with embodiments of the disclosure. The diagram illustrates a high- level system architecture for an advanced computing system, specifically highlighting the interaction between various components and interfaces. In one embodiment, the system includes multiple CPUs, DRAMs, and A-HBM SiPs. The CPUs are connected to the DRAMs and A-HBM SiPs through PCIe switches (e.g., PCIe7). In one embodiment, the A-HBM SiP comprises multiple A-HBMs, which are interconnected using UCIe- Advanced.

[0356] In one embodiment, the system also includes an Ethernet switch 3204 for networking between trays and a UALink switch for high-speed interconnects. In some respects, the tray (e.g., tray 0) can be connected to the Ethernet switch at least over a first connector, which can be an optical based connector. In one example, the connector can include at least one network connector including an optical small form factor (OSPF)connector configured to transfer data at a first rate (e.g., 800 GBE per lane / channel). In some respects, the tray (e.g., tray 0) can also be configured such that various AHBM cards (e.g., AHBM 0AM 2 and AHBM 0AM 3 in FIG. 32) are connected to the UALink switch at least over another network connector including a second connector, which can be an optical based connector. In one example, the connector can include one or more OSPF connectors configured to transfer data at a second rate (e.g., 200 GB per lane / channel). In some respects, the Ethernet switch is configured to transmit one type of information from a network interface card (NIC) over a first type of protocol (e.g., Ethernet protocol), while the UALink switch is configured to transmit second type of information from one AHBM SiP card on one tray to another AHBM SiP card on the same tray or another tray. Overall, this architecture enables high-performance applications requiring fast data transfer rates and low latency.

[0357] In some aspects, the tray (e.g., tray 0) can include a NIC that is connected to a processor (e.g., a CPU). The CPU can be configured to manage the interactions and data transfers over the NIC and between one or more AHBM cards (e.g., OAMs) and other devices on other trays. The CPU can also be configured to connect to a storage device (e.g., a non-volatile memory express (NVME) based device) that can be configured to store any suitable data (e.g., information related to LLM models that are being executed, including but not limited to, input matrices, activations, weights, and the like.

[0358] In more detail, the disclosed systems can use a rack can include two network switch boxes, one for Ethernet 3204 and the other for UALink, along with multiple computing trays such as tray 0 to tray N-l (an example of 6 trays can be used). Each computing tray can include a predetermined number accelerator modules, for example, 6 A- HBM SiP cards in OCP Accelerator Modules (OAMs) format that are interconnected. In one embodiment, the A-HBM SiP on the 0AM employs a chiplet-based architecture. In one embodiment, the SIP comprises 16 A-HBMs that are fully connected using D2Dinterconnects (i.e., UCIe-Advanced interconnects) over a silicon interposer with a gap (e.g., less than about 2 mm). In one embodiment, in addition to the interconnection of A-HBMs, a predetermined number of chiplets, for example, 4 IO chiplets can be interconnected to 4 A- HBMs located at the corners using the same D2D interface. This enables external connections between SiP to SiP and SiP to PCIe.

[0359] Level 3: Card

[0360] In one embodiment, a given a thermal design power (TDP) of a predetermined amount (e.g., approximately 992 W) and an A-HBM SiP dimension of predetermined values (e.g., 80 mm x 80 mm), a PCI Express card form factor with A-HBM SiPs can be used. In one embodiment, the mechanical form factor for the PCLe card is considered to be either full or three-quarters length and full height. In one embodiment, the target board power can be approximately 1000 W or more. This can be achieved using two PCIe power connectors, each capable of delivering a predetermined power (e.g., about 600W (using a H+ 12V High Power HPWR connector) or 675 W (using an H++ 12V-2x6 connector)). In an embodiment, the thermal solution for the A-HBM SiP on a PCLe card can utilize liquid cooling. In one embodiment, liquid cooling solutions can handle up to TDP of a predetermined amount, approximately 1500 W. In one embodiment, when the TDP of A-HBM SiP is limited to a predetermined amount such as approximately 450W-600 W, air cooling solution may be used. In one embodiment, the maximum number of A-HBM SiPs per card can be determined based on available cooling solutions.

[0361] Architecture and Components of A-HBM Card

[0362] Management Interface for A-HBM Card

[0363] The disclosed systems can include an A-HBM card that communicates with theBaseboard Management Controller (BMC) on the tray via the following management interfaces, which can also be used for debugging and also profiling. In one embodiment, themanagement interface can include an I2C / I3C / SMBus 2.0 compliant interface that can serve as an out-of-band interface that supports Intelligence Platform Management Bus (IPMB) and Intelligent Platform Management Interface (IPMI) protocols, as well as Management Component Transport Protocol (MCTP) over SMBus binding. In another embodiment, the management interface can include a JTAG interface that is also out-of-band and that provides general debug access, register / memory dump capabilities. In an embodiment, the management interface can include a UART interface that is out-of-band and that enables serial console access. In one embodiment, the management interface can include a PCIe interface is an in-band interface that may be used to support MCTP over PCIe binding. In one embodiment, the management interface can include a UALink interface is an in-band interface that can be used to support MCTP over UALink binding.

[0364] A-HBM Card Sensors

[0365] The disclosed systems can include an A-HBM card that may include several silicon components, including one or more ASICs implementing acceleration functions, and memory such as HBM and / or LPDDR. The connected sensors of these components can be read over sideband interfaces for system management. The sensors can be configured to measure, among other things, at least the power / current (memory, compute logic, and other modules), voltage (memory, compute logic), ASIC temperature (hot spot, edge temperature), memory temperature for each stack, air / liquid flow inlet and outlet status, and the power state: max power mode, reduced / capped power mode. The information can be retrieved via the management interface.

[0366] A-HBM Card Debugging and Profiling

[0367] The disclosed systems can include debugging and profiling capabilities. In some aspects the hardware of disclosed systems can include debugging and profiling features that are available in some computer architectures such as those in ARM architecture. In somerespects, the debugging and profiling features can be provided after valid authorization.Further, the A-HBM Tray may include a BMC.

[0368] In one embodiment, a debugging system for an A-HBM system supports various debugging features. In an embodiment, the JTAG interface is utilized as a debugging interface, compliant with ARM's Debugging Interface (ADI) specification. In one embodiment, an event counter is provided as a lightweight debugging tool with minimal performance impact. This allows for conditional counting of specified events.

[0369] In another embodiment, an event logger with timestamping is also provided. This enables conditional logging of specified events with timestamp. To prevent buffer overwhelming, a circular buffer can be employed. The timestamp granularity can be as fine as the clock resolution. In one embodiment, an event notifier is provided for conditional notification of specified events. In one embodiment, a list of loggable events should be provided to facilitate effective debugging. In one embodiment, a register / memory dumper is provided to dump data from a set of registers or a range of memory. In one embodiment, a register / memory loader is also provided to load data to a set of registers or a range of memory.

[0370] In one embodiment, the disclosed systems can include a profiling system that includes a Performance Monitor Unit (PMU). The PMU supports at least 7 registers per core to store hardware counter values such as cycle and instruction count. However, there is a performance trade-off point to increase the number of registers. In another embodiment, the required hardware counters include cycle, instruction, load, store, stall, special instructions (e.g., matrix-multiply- accumulate instruction), all caches (L1 / L2 / LLC), and DDR access / refill. These counters provide valuable insights into the system's performance and help optimize its operation.

[0371] In an embodiment, a hardware monitor system for an A-HBM system includes several monitoring components. In one embodiment, a power monitor is provided to monitor power consumption per each power rail. In one embodiment, thermal monitors (sensors) are also provided. At least 1 sensor is required for each core and other subsystems like memory. In one embodiment, a bandwidth monitor is provided to monitor data transfer rates between memory nodes. For example, bandwidth monitoring can be performed between LI and L2 caches, L2 and LLC caches, and LLC caches and DDR memory.

[0372] In one embodiment, hardware knobs are provided to facilitate power measurement and optimization. In an embodiment, power measuring points are included to monitor power consumption of various components. These measuring points include cores, DDR memory, and total board power. In one embodiment, to enable detailed power measurement, a special board may be created to measure power consumption of each power rail.

[0373] Level 4: Tray

[0374] Interconnect Topology

[0375] In one embodiment, the disclosed systems can include a tray that provides a sufficient number of A-HBM card slots. The number of slots is determined considering the expected growth in LLM parameter size and the number of attention heads. In an embodiment, the total bandwidth of the on-board interconnect is determined based on the communication topology between A-HBM cards and the peak traffic bandwidth among them. In one embodiment, the communication topology among A-HBM cards can be flexible enough to accommodate rapidly evolving LLMs. In an embodiment, several communication topologies can be considered for LLM inference with multiple A-HBM cards in a tray.

[0376] All-to-All: In various embodiments, the disclosed systems can include A-HBM cards which can each directly communicate with all other A-HBM cards. In one embodiment,a fully connected topology for the disclosed systems offers maximum flexibility. However, this topology can become complex and expensive to manage when dealing with large numbers of A-HBM cards due to the high number of connections required. In one embodiment, a variation of this topology is a fully connected mesh. In this mesh topology, each A-HBM card connects to a specific subset of other A-HBM cards. This approach offers better scalability compared to a complete all-to-all connection but still requires a significant number of connections.

[0377] Ring or Torus: In one embodiment, the disclosed systems can include A-HBM cards that are arranged in a ring or torus-shaped network. In this configuration, data hops from one A-HBM card to the next, eventually reaching all A-HBM cards. In an aspect, this ring or torus topology offers a balance between communication overhead and scalability. However, data transfer can be slower for some A-HBM cards depending on their position in the ring. Thus, in some aspects, some A-HBM cards can be dedicated to communicating with external tray's A-HBM cards or all A-HBM cards. In one embodiment, this topology is configured based on the bandwidth of latency and end-to-end data path, as well as communication load balancing.

[0378] Tree Network: In one embodiment, the disclosed systems can include A-HBM cards that are arranged in in a hierarchical tree structure. This can be efficient for workloads with localized communication patterns, but performance can degrade for models with global dependencies.

[0379] Hybrid Topologies: In one embodiment, the disclosed systems can include A- HBM cards that are arranged in hybrid topologies in which elements of different topologies are combined. For example, a portion of the cards can be configured in a mesh network for core communication and another portion can be configured in a ring for additional communication paths can be a good approach for complex models. Further, in some respects,the disclosed systems can feature provisioning reconfigurability of topology for increased flexibility.

[0380] Interconnect Topology Example when Eight A-HBM Cards in a Tray

[0381] In one embodiment, the disclosed systems can feature various interconnect topologies, which, for the sake of explanation, are described herein with reference to an example in which the tray has up to eight A-HBM cards installed. The number of ports provided by each A-HBM card can vary depending on the interconnect topology inside the A-HBM tray. In one embodiment, different interconnect topologies have pros and cons in terms of node-to-node latency and bandwidth, network degrees and diameters, and the like. These factors ultimately affect overall system performance, power consumption, scaling cost, and programmability.

[0382] In one respect, if each A-HBM tray operates independently, there may be no need for Tray-to-Tray scaling interconnect. In this case, a fully connected topology can be used to interconnect A-HBM cards within a tray. In an aspect, if multiple A-HBM trays need to operate together, then Tray-to-Tray scaling interconnect may be required. In this scenario, a combined fully connected and hybrid cube mesh topology or 2D torus topology can be used.

[0383] FIG. 33 shows a fully connected interconnect topology example for the disclosed systems, according to an embodiment of the disclosure. The diagram illustrates a fully connected interconnect topology example for OCP Accelerator Modules (OAMs). In one embodiment, this topology may feature no or limited interaction between trays. Further, it can enables direct communication between any two OAMs, allowing for efficient data transfer and scalability. Each 0AM has multiple ports, and each port is connected to a specific number of other OAMs. The connections between OAMs are represented by arrows,indicating different communication channels. This fully connected topology allows for maximum flexibility and performance in a system with multiple OAMs.

[0384] FIG. 34 shows a combined fully connected and hybrid cube mesh interconnect topology (e.g., where tray-to-tray communications are enabled) example for the disclosed systems, in accordance to embodiments of the disclosure. The diagram illustrates a combined fully connected and hybrid cube mesh interconnect topology example for OAMs with tray-to- tray communication. In one embodiment, this topology enables efficient communication between OAMs within a tray and between trays. The OAMs are arranged in a grid-like structure, with each 0AM having multiple connections to other OAMs. The connections between OAMs are represented by lines of different colors, indicating different communication channels. This topology combines the benefits of fully connected and cube mesh topologies, offering a balance between performance, scalability, and complexity.

[0385] FIG. 35 (i.e., FIG. 35A and 35B) illustrates a 2D torus interconnect topology example for the disclosed systems, according to an embodiment of the disclosure. In one embodiment, this topology is used for connecting multiple nodes in a grid-like structure. The nodes are arranged in rows and columns, with each node connected to its neighbors. The connections between nodes are represented by lines, forming a torus shape. This topology can be used to enable parallel computing and high-performance computing applications.

[0386] RAS (Reliability, Availability, and Serviceability)

[0387] In some respects, the A-HBM can include an A-HBM tray that comprises an LLM inference server which includes various types of devices. If any of these components fails, the entire system can stop working, which has a serious adverse effect on the user's information service experience. To avoid this, RAS is implemented as part of the A-HBM Tray, as described below.

[0388] Hardware RAS features

[0389] The A-HBM tray can include redundancy of components, such as Dual PowerSupplies to ensure the server has redundant power supplies to maintain operation if one fails.Moreover, A-HBM tray can have Network Interface Cards (NICs) that are redundant for network connectivity to maintain communication even if one card malfunctions. The A-HBM tray can have Mirrored Storage (e.g., Redundant Array of Independent Disks (RAID)) to protect against data loss due to drive failures. The A-HBM tray can include Error Correction that uses hardware features like ECC memory to detect and correct errors in data transfers, preventing system crashes due to memory faults.

[0390] The A-HBM tray can have hot-swappable components like storage drives or network cards that are hot-swappable. This allows for replacement without powering down the entire server, minimizing downtime for maintenance. The A-HBM tray can have event logging for debuggability using always-on dependable components which can be used for system debugging even when A-HBM tray is dead. All the components in the A-HBM tray can be able to record event log based on the configuration, so that this log can be used for debugging, and to be used in the failure prediction. System administrators can be able to use serial console or IPMI-like out-of-band channel to access this event log. Types of events to record and granularity of information to store are configurable and reduce the time to analyze root cause of failure.

[0391] Software RAS features

[0392] The A-HBM tray can have an Operating System with built-in RAS functionalities like filesystem journaling and process management for enhanced stability. The A-HBM tray can have error handling and recovery mechanisms within the inference software to handle potential issues like model errors or resource limitations. This can include retry logic, fallback mechanisms, or error logging for further analysis. The A-HBM tray can have monitoring and logging tools to track server health, resource utilization, and communicationperformance. It can use logging to record errors, warnings, and system events for troubleshooting and identifying potential issues proactively. The A-HBM tray can provide load balancing techniques to distribute inference requests evenly across multiple servers in the cluster. This prevents overloading individual servers and ensures efficient resource utilization. The A-HBM tray can have feature high availability frameworks to manage server clusters, automate failover processes, and ensure continuous operation.

[0393] Level 5: Rack

[0394] The disclosed systems for Al inference are compatible with various standard rack configurations (e.g., Open Rack V3 and the like). It can therefore be seamlessly integrated into diverse data center environments without requiring specific alterations. The A- HBM can be universally compatible with standard rack systems for model serving, ensuring low latency by allowing the system to be integrated into data centers. The A-HBM can include cooling systems like Air-Assisted Liquid Cooling (AALC) or Facility Water Cooling (FWC). The disclosed systems can remain functional with air cooling by employing power throttling, accommodating users who prioritize reduced operational costs over performance. The A-HBM integrates with existing network architectures. Each tray can be configured to directly connect to the top-of-rack switch, simplifying networking. Additionally, A-HBM's design demands no special maintenance, further reducing total cost of ownership and enhancing operational efficiency.

[0395] Software Stack

[0396] Al Framework

[0397] The hardware and software ecosystem for machine learning is different from the ecosystem for traditional programmable processors. For some processors such as CPU andGPU, a programmer directly writes their program for a target hardware and can try to run the program and debug it directly on the target processor. On the other hand, in machine learning,a user defines their model architecture to be run on a training hardware, which may be different from an inference hardware. The training infrastructure serializes the output of the training into its own format, and runs the trained model. As a result, users of the model can try to run and debug the model on the training hardware but not on the inference hardware. This creates a large gap between the training hardware and the inference hardware in terms of functionalities and performance optimality. Also, because training often takes long, modifying the source program, i.e., the model, to make it compatible with or performant on an inference hardware can be difficult. As a result, the AHBM inference infrastructure runs a given model as is by supporting a model serialized into a serialization format from a training infrastructure.

[0398] Compiler

[0399] Hardware Abstraction

[0400] In one embodiment, the disclosed systems provide an approach to addressing the challenges of programming massively-parallel in-memory computing systems. As the ecosystem evolves around programming models and abstract hardware models, systems can hide and abstract out differences in hardware implementation from users. This can be the case for ML applications, where time-to-market is critical, and there may be limited dedicated authoring or programming frameworks for inference.

[0401] The disclosed systems address this challenge by leveraging a compiler that can effectively hide the deficiencies of programmability in accelerators. Accelerators, designed to improve hardware efficiency, often sacrifice programmability. The compiler plays a role in handling non-traditional techniques such as auto-parallelization and locality management with scratchpad memory. Additionally, the compiler conceals the unique characteristics of the accelerator's hardware implementation.

[0402] To address this, some embodiments may address many hardware-specific problems one at a time by structuring compiler passes so that one pass address only one hardware-specific problem at a time. This results in gradual lowering of hardware models from the perspective of programmer’s view. Thus, the system lowers compiler passes and hardware abstract models at the same time in parallel, improving debuggability and testability of a compiler.

[0403] Overall Architecture

[0404] In one embodiment, the disclosed systems can use a front-end compiler in translating fragmented and non-standard representations of ML models into a unified intermediate representation (IR). This enables the middle-end and back-end compilers to target a single IR, regardless of how external IRs evolve with advancements in ML. By decoupling the front-end compiler from the evolving ML landscape, the middle-end and back-end compilers are shielded from constant updates, ensuring stability and reducing development overhead.

[0405] In another embodiment, the hardware abstraction model employed by the frontend compiler is based on a simple machine model that assumes a unified and infinite memory space consumed by a scalar processor. This abstraction layer provides a layer of indirection between the ML model representations and the underlying hardware, allowing for efficient compilation and execution of ML models on diverse hardware platforms.

[0406] In one embodiment, the disclosed systems can use a middle-end compiler in optimizing the mapping of ML models to a high-level system architecture. The middle-end compiler exploits model-specific information to determine the optimal execution strategy, including auto-parallelizing the model based on memory requirements and compute availability.

[0407] In another embodiment, the middle-end compiler can identify the communication patterns required for data movement between hardware components, such as compute nodes, storage nodes, and move nodes. The compiler assigns tasks to these hardware components to optimize data movement and minimize latency.

[0408] In one embodiment, the middle-end compiler performs various optimizations to align and tile operations with abstract hardware requirements. For example, it tiles operations to match the number of compute units available on the hardware platform. The middle-end compiler also aligns and pads data to meet specific requirements for storage and interconnects. In some embodiments, based on these tiling strategies, synchronization granularity can be determined to hide communication overhead behind computation. This enables efficient execution of ML models on diverse hardware platforms.

[0409] In one embodiment, the disclosed systems can use a back-end compiler for translating the intermediate IR generated by the middle-end compiler into a target instruction set architecture (ISA). This process, known as lowering, enables the compiled code to be executed on a specific hardware platform. In one embodiment, the back-end compiler can perform local optimizations that are tailored to the target hardware. For example, it can schedule instructions to optimize execution based on the availability of arithmetic logic units (ALUs) within a given hardware block. By doing so, the back-end compiler can further improve the performance and efficiency of ML models on diverse hardware platform.

[0410] Control Flow Handling

[0411] Unlike traditional ML models like CNN, modern model architectures like LLM starts to have a lot of control-flow related operations like early exit, which may not supportive to massively-parallel architectures. To address this occasional need while providing full functional coverage, the disclosed systems can include a CPU backend compiler for addressing functionality that is not compatible with NPU / PE IPs.

[0412] Dynamic Shape Handling

[0413] LLMs can present a challenge for in-memory computing due to their tensor dimensions that are not known at compile-time but determined at run-time. Unlike some ML models, such as convolutional CNNs, LLMs require dynamic memory allocation and management. In one embodiment, to address this challenge, a compiler or programmer must tile or chunk unknown tensor dimensions into a series of fixed-sized inference calls. This approach enables the separation of computation into two phases: static computation and dynamic computation. By doing so, it is possible to support complex dynamic shape problems while maintaining hardware efficiency and optimality of execution time. However, in one embodiment, this approach requires significant CPU bandwidth to manage the tiling and chunking process. The disclosed systems provide efficient execution of LLMs on inmemory computing architectures via compiler techniques and hardware optimizations to mitigate the impact of dynamic tensor dimensions on performance.

[0414] In particular, FIG. 36 depicts static and dynamic operations for the disclosed systems, in accordance with embodiments of the disclosure. The diagram shows a software architecture for machine learning applications. It includes at least the following components: application code, library, and flattened view. The Application Code is written in C or Python and interacts with the Library. The Library has two parts: control code and dataflow code. The control code manages data flow and instructions, while the dataflow code defines how data is processed. The flattened view shows a PE array with multiple layers of code, including application code, control code, and dataflow code. This architecture enables efficient processing of machine learning tasks by optimizing data flow and control.

[0415] Op Coverage

[0416] In one embodiment, the disclosed systems can include inference hardware designed to provide functional completeness and fallback capabilities, ensuring that a usercan run a ML model on the hardware without requiring retraining. To achieve this, the hardware should have the capability to execute all types of operations (ops), and the software should be able to lower a given model to the hardware's ISA. In another embodiment, some reduced instruction set computer (RISC) processors can decompose various programs into its ISA. Therefore, the compiler system is structured to support a wide range of ops, including those that are not supported by NPU cores on CPUs. In one embodiment, the disclosed systems can use custom techniques to handle cases where a model contains custom operations that are not supported by the NPU cores. To address this issue, an infrastructure is provided to enable users to decompose compound ops into a set of RISC ops. In one embodiment, the compiler system can be configured to perform fusion. In particular, a series of finer-grained ops can be more efficiently executed when fused into a single hardware instruction. To achieve this, the compiler includes a pass for well-known, hardware-specific fusion patterns.

[0417] Debuggability

[0418] In some aspects, debugging a ML compiler poses challenges. For example, auto-parallelization, a component of ML compilers, can be difficult debug due to its complex and distributed nature. Furthermore, ML models can be challenging to debug as they may lack a clear expected behavior that can be compared against hardware execution results. Further, additional challenges arise from the design of ML models themselves. Specifically, some ML models are designed to "forget" minor differences or generalize diverse patterns into a common feature set, which can effectively hide system bugs. For example, a ReLU operation may clip values in a way that masks underlying issues. Another challenge is introduced by the need to truncate high-precision operations into low-precision operations to run an ML model on a given hardware platform. This makes it impractical to rely on reference executions on CPUs as test vectors. In one embodiment, the reordering ofoperations by an ML compiler to address limited SRAM or register file capacity further complicates the development of test infrastructures. The non-associative nature of floatingpoint operations makes it challenging to build reliable test vectors. To address these challenges, in one embodiment, the disclosed systems can provide compiler passes at a finegrained level as possible so that each pass can handle one challenge at time through finegrained A / B testing. Also for those passes which are mathematically identical but numerically different, the disclosed systems can use symbolic validation strategies to reduce deployment time. Moreover, the disclosed systems can provide fine-grained compiler passes that can provide intermediate compilation results and take user-annotated compilation results along with an authoring framework.

[0419] FIG. 37 shows a low-level API to empower users for the disclosed systems, according to an embodiment of the disclosure. The diagram illustrates a low-level API architecture. In one embodiment, this API enables users to interact with a processor and its ISA. The API provides an interface for programs to access the processor's resources. The user compiler can enable optimizing the performance of the processor.

[0420] Programming Model

[0421] The programming model is an abstraction of A-HBM architecture that provides a shared view between applications and possible implementations of A-HBM hardware. This section describes the main concepts of A-HBM compute and memory models.

[0422] Kernel and Thread Hierarchy

[0423] The disclosed systems can use the concept of thread, thread block and kernel grid. Further, the disclosure outlines the A-HBM thread hierarchy by comparing it with other programming models below. In some programing models such as CUD A, kernels are subdivided into thread blocks. Each block is executed by CUDA SM (streamingmultiprocessor). It cannot be migrated to other A-HBMs but multiple blocks can be assigned to A-HBM.

[0424] In particular, FIG. 38 illustrates a kernel and thread hierarchy for the disclosed systems, in accordance with embodiments of the disclosure. In the A-HBM architecture, threads can be executed by A-HBM PE. Thread blocks can be assigned to A-HBM cube, which contains multiple PEs. Kernel grid, which contains the entire CUDA blocks for the operation, can be mapped to A-HBM SiP. Each thread has their own unique ID, which can be used to identify the region of data it needs to process. A-HBM defines built-in 3D variables for threads and blocks: Axis 0: HBM Index, Axis 1 : PE Index, and Axis 2: Channel Group Index.

[0425] Programmers can calculate the range of data assigned to each kernel by using these variables. While PEs can access any HBM memory region in A-HBM SiP, performance-wise, it may be more efficient to access the region of memory a PE has a direct access. Axis 2 information can be utilized for this purpose. This is described in the Kernel ID assignment section below.

[0426] Memory Hierarchy

[0427] Various memories are exposed by A-HBM Architecture. In particular, FIG. 39 provides a memory hierarchy diagram for the disclosed systems, according to an embodiment of the disclosure. The diagram illustrates a memory hierarchy for the AHBM system. In one embodiment, the memory hierarchy consists of multiple levels, including the following. Scratchpad Memory: A small, fast memory that provides temporary storage for data. TCM: a private memory for each PE that provides low-latency access to data. Instruction Cache: a cache that stores frequently accessed instructions. Global Memory: a large, shared memory that provides access to data for all processing elements. The diagram also shows the relationship between these components and the PEs, as well as the HBM groups.

[0428] In one embodiment, a PE in an A-HBM system includes a private memory region, referred to as TCM private memory. This memory can be exclusive to each PE and can be used to temporarily store input, intermediate, output tensors or as a buffer for VO operations with HBM. The values stored in this region are not accessible to other PEs. In one embodiment, the A-HBM system also includes a shared memory region, referred to as scratchpad memory. This memory region is shared among multiple PEs and provides higher bandwidth than HBM. Each PE can acquire exclusive write permission on a specific area of scratchpad memory by communicating with other PEs in the A-HBM package via message queues.

[0429] In one embodiment, an instruction cache is provided as a read-only memory region that caches kernel instructions. In an embodiment, the A-HBM system includes multiple HBM groups, each comprising a group of HBM pseudo channels that can be directly accessed by a dedicated PE. While each PE can access HBM groups belonging to other PEs, minimum latency is achieved when accessing its corresponding HBM group. In one embodiment, a global memory address space is provided as a linear address space that users can access. This global memory internally includes multiple HBM groups. When accessed sequentially, requests are evenly distributed across HBM groups to maximize bandwidth utilization.

[0430] In one embodiment, the memory accesses in the A-HBM system are physical, and may require explicit paging at the kernel level when processing large datasets that exceed the available physical memory space. In another embodiment, each PE) in the A-HBM system can access a dedicated group of HBM channels with minimal interconnect overheads. To achieve high PE utilization, the disclosed systems can align data across HBM groups. In one embodiment, compilers in the software stack automatically align model data to optimizeperformance. Additionally, programmers can manually align the layout of data using an HBM group allocator to fine-tune performance.

[0431] Kernel ID Assignments

[0432] In one embodiment, the disclosed systems can assign a kernel identifier (kernel ID) to a program or thread executing a function, enabling individual programs to differentiate themselves and access specific data or perform calculations based on their assigned identifier. The kernel ID allows developers to implement conditional logic or data manipulation techniques based on the block's position within a grid. In an embodiment, the kernel ID can be used within the kernel to determine which portion of the data should be processed by the program. For example, in one-dimensional (ID) computations, the kernel ID can determine the starting index of the data slice that will be processed by the program. In more complex scenarios, such as two-dimensional (2D) and three-dimensional (3D) grids, the kernel ID can be mapped to coordinates and used to manage work across different dimensions.

[0433] HBM Channel Bandwidth in PE

[0434] An example of how kernel IDs are utilized to process matrix multiplication operations in a distributed computing environment is now disclosed for a scenario where a GPT 175B model is being processed. The model consists of 96 layers, and for each layer, the weight matrices for attention heads are divided across multiple HBM devices. In one aspect, assume that 96 attention heads are distributed across 96 HBMs, which are further distributed across 6 cards. In this scenario, kernel IDs play a role in determining which portion of the data each PE should process.

[0435] For instance, in one embodiment, each PE can use its kernel ID to identify the specific HBM and data slice it is responsible for processing. By doing so, PEs can efficiently perform matrix multiplication operations on their assigned data slices without conflicts or redundant computations.

[0436] In one example, a matrix multiplication scenario is considered with specific configurations: 12,288 rows, 12,288 columns, 4 channels per PE, 96 HBM devices, 16 PEs per HBM device, a block size of 128 rows per PE, and a resulting block size of 768 columns per PE and 192 columns per channel. In another aspect, splitting the weight matrix by rows between HBM devices leads to each HBM device processing a (128 x 12,288) matrix. Since each HBM device has multiple PEs, each PE will process a (128 x 192) slice. Furthermore, data placement for columns can be divided between channels to achieve maximum bandwidth utilization.

[0437] FIG. 40 describes matrix split between HBMs / PEs / Channels for the disclosed systems, in accordance with embodiments of the disclosure. The diagram illustrates a data placement strategy for the AHBM system. In one embodiment, a large matrix (W) is divided into multiple blocks and distributed across multiple HBM devices. The HBM devices are organized into channels, with each channel having multiple banks. Each bank is further divided into rows, and each row is processed by a specific PE. The PEs are connected to the HBM devices through a hierarchical structure, enabling efficient data transfer. This data placement strategy aims to maximize parallelism and minimize data transfer latency.

[0438] As shown in FIG. 40, the disclosed systems can process the full matrix in parallel by 96 HBMs, where the HBMs process the 128 X 12,288 matrix. That is, HBM0 will process the first 128 rows, HBM1 will process the next 128 rows and so on.

[0439] PID assignments

[0440] In one embodiment, process ID (PID) assignments are provided by the disclosed systems to ensure that each HBM device knows which slice of data it is operating on and which portion of that data needs to be processed by which PE. To achieve maximum bandwidth utilization, pseudo channels handle different columns, allowing data to be loaded and stored simultaneously. In one embodiment, three PID assignment techniques can include:HBM-PID, PE-PID, and Channel-PID. These strategies enable efficient distribution of data across multiple HBM devices, PEs, and channels. FIG. 41 shows PID assignments for oneHBM.

[0441] The diagram illustrates a memory allocation and processing structure for a HBM system. In this structure, HBM-0 has 12,288 rows and is divided into 16 PEs, with each PE handling 768 columns. Each PE is further divided into 4 channels (CH0-CH3), with each channel having 192 columns. The diagram shows three types of Process IDs (PIDs): HBM- PIM PID, PE-PID, and CH-PID. These PIDs are used to identify and manage the different components and their interactions within the HBM system.

[0442] The programming model can be as shown in FIG. 42. The first section assigns PIDs to identify which HBM device, PE, and channel are operating on. The next section calculates the starting row and column positions for the blocks to be processed. Finally, it computes the row and column offsets for each block. This code can be used in distributed computing scenarios, such as training large language models, where data parallelism across multiple HBM devices, PEs, and channels is employed. Proper PID assignments and offset calculations ensure efficient utilization of hardware resources.

[0443] In one embodiment, PID information can be utilized to determine which portion of a large matrix each HBM device, PE, and channel can process. Specifically, the row range (row start - row end) for each HBM device depends on its HBM PID, as the data is split by rows between HBM devices. In one embodiment, to maximize bandwidth utilization, PEs can be assigned different sets of columns to process. The column range for each PE is determined using its PE PID and channel PID. Careful alignment of data across channels is crucial to ensure that each channel handles a distinct set of columns.

[0444] In another embodiment, an alternative implementation involves incorporating a memory management unit (MMU) within each PE. This MMU can translate local addressesused by user programs into corresponding physical addresses on the A-HBM device. By doing so, channel information can be abstracted from user programs, making them portable across different generations of A-HBM devices.

[0445] Kernels for Inference

[0446] GEMM

[0447] The disclosed systems can implement GEMM to multiply two input matrices A of dimensions m x k and B of dimension k x n together to get an output matrix C of dimension m x n. The naive matrix multiplication algorithm can be implemented by traversing matrices element by element and multiplying them individually and has the time complexity of O(n3). The disclosed systems can be configured to use any of multiple optimizations to bring down this time complexity to as low as about 2.37, some being galactic algorithms. The disclosed systems can be configured to use, for example, algorithms like Strassen and Winograd in building the core of GEMM implementations in notable BLAS (Basic Linear Algebra Subprograms) libraries.

[0448] GEMM Modeling for HBM-CM

[0449] In one embodiment, the disclosed system can use a matrix multiplication operation for a high-bandwidth memory-cache (HBM-CM) using a 2D block tiling methodology. This approach partitions a large matrix into smaller blocks of Mtile-by-Ntile independent matrix products. In one embodiment, each block product is computed by accumulating Mtile-by-Ntile-by-Ktile matrix products. This process enables efficient computation of large-scale matrix multiplications by breaking them down into smaller, more manageable blocks.

[0450] Figure 43 shows an example of a tiled Matrix Multiplication, in accordance with embodiments of the disclosure, while FIG. 44 illustrates GEMM modeling with blocks and tiles of the disclosed systems, according to an embodiment of the disclosure. The GEMMtechnique can be performed for a matrix multiplication function by implementing it as a visual basic macro function. Based on the dimensions of the A & B matrix,

[0451] Tile dimensions (TM, TN) are calculated as the minimum of 32 or full matrix row or column divided by the minimum tile size of 4.

[0452] TM = Min (Floor ((A ROW + MIN TILE - 1) / MIN TILE) * MIN TILE, 32)

[0453] TN = Min (Floor ((B COL + MIN TILE - 1) / MIN TILE) * MIN TILE, 32)

[0454] Block dimensions are calculated as follows.

[0455] BM = Min (Max (A ROW, TM), 128)

[0456] BK = Min (A COL, 128)

[0457] BN = Min (Max (B COL, TN), 128)

[0458] The number of blocks in the overall grid is the product of the number of BM blocks and BN blocks. Each of these thread block can be executed independently in separate PU.

[0459] num bm blocks = RoundUp(C_ROW / BM, 0)

[0460] num bn blocks = RoundUp(C_COL / BN, 0)

[0461] num thread blocks in grid = num bm blocks * num bn blocks

[0462] Each output block matrix multiplication involves fetching multiple Tiles from block in A and block in B and the total number of tiles to fetch per block are calculated as follows.

[0463] num_tm_tiles_per_block = RoundUp(BM / TM, 0)

[0464] num_tn_tiles_per_block = RoundUp(BN / TN, 0)

[0465] num_tiles_per_block = num_tm_tiles_per_block * num_tn_tiles_per_block

[0466] The total flops per grid are calculated from the flops per tile and the total number of tiles fetched across the full matrix.

[0467] total_flops_per_tile = mul_per_tile + add_per_tile

[0468] total _flops_per_output_block = num_tiles_per_block * num_loops_per_block * total _flops_per_tile

[0469] total _flops_per_grid = num thread blocks in grid * total fl op s_p er output b lock

[0470] The 2D block tiling assumes the block from A and B are prefetched from the HBM to SRAM prior to kick-starting the matrix multiplication operation. But, this could be extended to fetching tiles of the block needed for the next cycle in parallel with the current batch of execution. Overall, we first calculate the total amount of data to read and writeback per block.

[0471] GMEM_to_SMEM_per_block_A = BM * BK * dt / 1024 # read

[0472] GMEM_to_SMEM_per_block_B = BK * BN * dt / 1024 # read

[0473] GMEM_to_SMEM_per_block_C = BM * BN * dt / 1024 # write

[0474] The total memory operation needed is calculated for the full matrix as follows.

[0475] total_mem_op_per_output_block = (GMEM_to_SMEM_per_block_A + GMEM_to_SMEM_per_block_B) * num_loops_per_block + GMEM_to_SMEM_per_block_C

[0476] total_mem_op_per_grid = total_mem_op_per_output_block * num thread blocks in grid

[0477] The overall compute latency and memory latency in nanoseconds is calculated based on the above calculation and the advertised TFLOP of the CM and the memory bandwidth between the HBM and SRAM.

[0478] T MATH US = ((total_gflops_per_grid / 1000) / tflops) * 1000 * 1000

[0479] T MEM US = ((total_mem_op_per_grid_mb / 1024) / bw) * 1000 * 1000

[0480] The resulting overall latency is the maximum between the compute and the memory latency:

[0481] Max (T MATH US, T MEM US)

[0482] Flash Attention

[0483] The self-attention block in a large language model can be decomposed into 3 computations as follows:

[0484] X = QKT (1)

[0485] A = softmax(X) (2)

[0486] O = AV (3)

[0487] where Q, K, V are matrices of dimension N (sequence length) x d (head dimension). In one embodiment, the disclosed systems can use an attention implementation writing the result of each computation to HBM and then loading the written data back for the next computation. This approach can lead to a significant number of HBM accesses, which can impact performance. In one embodiment, optimized fused implementations of the selfattention stage, such as Flash Attention vl and Flash Attention v2 can be used to reduce the number of HBM accesses. These implementations avoid storing the result at the end of each computation and instead retain the data in SRAM or TCM for use in subsequent computations. In one embodiment, by retaining data in SRAM / TCM, the disclosed systems can use Flash Attention vl and Flash Attention v2 to reduce the number of HBM accesses required for self-attention computations. This optimization leads to improved performance and efficiency in attention-based neural network computations.

[0488] KV Cache Placement

[0489] In one embodiment, a key-value (K, V) cache for a user is divided into multiple blocks, as described in the page attention paper. These blocks are spread across multiple HBM devices to optimize performance. In another embodiment, the block size of the K, V cache is aligned with the execution loop block size, which can be set to a specific value such as 128. This alignment enables efficient computation and data access. In one embodiment, around-robin algorithm is used to distribute the K, V cache blocks across 16 HBM banks within the HBM device. This distribution strategy ensures that the cache blocks are evenly spread across the available HBM banks, maximizing parallelism and bandwidth utilization.

[0490] Parallel Execution

[0491] In one embodiment, the disclosed systems can operate using parallizable algorithms. For example, some versions of FlashAttention (e.g., provides parallelism across the batch dimension and the number of heads dimension. Specifically, the disclosed systems can use it to enable: concurrent calculation of user scores in different HBM devices; and concurrent processing of multiple attention heads in a multi-head attention (MHA) mechanism across different HBM devices. However, some algorithms may not support parallelization across the sequence length dimension. This limitation may exist due to dependencies between intermediate variables and previous outer loop executions, which make independent parallelization challenging.

[0492] FIG. 45 depicts flash attention use case with the disclosed systems, in accordance with embodiments of the disclosure. The algorithm describes a FlashAttention v2 for N = 4096 & d = 128, for a forward pass for a machine learning model. In one embodiment, the algorithm takes in matrices Q, K, and V as input. The algorithm divides the input matrices into blocks and processes them in parallel. The algorithm uses on-chip SRAM to store intermediate results. The algorithm computes attention weights and updates the output matrix O. The algorithm also computes the logsumexp value L. The input and output represent the following. For the input: Q: A matrix representing a query or a question being asked. K: A matrix representing a knowledge base or a set of facts. V: A matrix representing values or importance associated with the knowledge base. For the output: O: A matrix representing the output or response to the query. L: A value representing the relevance or importance of the output.I l l

[0493] In other aspects, with other version of the algorithm such as Flash Attention v2, these challenges may no longer be the case. In the prefill stage, the self-attention calculation per user can be parallelized along the sequence length by partitioning Q along rows across multiple (e.g., 16) HBM banks. The K, V could be brought to scratchpad memory or fetched from peer HBM banks so that every PE loop over full K, V data. There may be no dependency that spans between iterations of the outer loop, the dependency may be restricted to the inner loop iterations only.

[0494] FIG. 46 shows an interaction diagram with simplified self-attention calculation for pre-fill stage for the disclosed systems, in accordance with embodiments of the disclosure. The diagram shows a loop optimization technique for a processing element array. It has two nested loops: Loop [i++] and Loop [j++]. The inner loop is optimized based on data availability in SRAM. If data is not in SRAM, an optimization is performed. This technique aims to improve performance by reducing memory access latency.

[0495] In some examples, for the auto-generation stage, The Q dimension can be 1 x d per user, the outer loop may be reduced to a single iteration, therefore a single PE can be used to do the score calculation per batch element (user) while the batch score calculation could be parallelized across all the PE. In case the batch size is large, then the disclosed systems can coalesce multiple user entries together and process them concurrently on each PE. For example, if the batch size is 256, then we can coalesce 16 batch elements (users) per PE.

[0496] All Reduce

[0497] In an embodiment, the disclosed systems can use an all-reduce collective communication operation in training and inference steps of LLM computations. Specifically, during an example algorithm inference such as GPT-3 inference, all-reduce operations occur twice: once after the completion of multi -head attention (MH A) and once after the completion of feed-forward neural network (FNN) computations. In one aspect, the internalall-reduce mechanism in an A-HBM system -in-package, the high-level flow of an all-reduce operation invoked by a user is first addressed. When a user calls an all-reduce function from a deep learning framework such as PyTorch, or when it is internally invoked through a model, the compiler generates kernels that can be executed on GPUs. In one embodiment, two types of kernels are produced by the compiler and executed concurrently: Worker A-HBM kernels; and Master A-HBM kernels. These kernels can be configured to enable efficient execution of all-reduce operations within the A-HBM system. First, the difference between worker and master A-HBM can be described. For convenience, the A-HBM can also be referred to as a node herein.

[0498] As shown in FIG. 47, the disclosed systems can include a master node in the System in Package that has access to the UAL and can communicate with other cards, leading to separation of functionality between the worker nodes and the master node. As noted, FIG. 47 shows an A-HBM SiP local reduction mechanism for the disclosed systems, in accordance with embodiments of the disclosure. In one embodiment, the A-HBM system consists of multiple worker A-HBM nodes and a master A-HBM node. The worker A-HBM nodes are arranged in a grid-like structure, with each node having multiple connections to its neighbors. The master A-HBM node is connected to the worker nodes through UAL interfaces. In one embodiment, the master node has access to the UAL and can communicate with other cards. This separation of functionality between the worker nodes and the master node enables efficient data processing and reduction.

[0499] In other aspects, there can be three tasks that need to be performed during allreduce: (1) local reduction from worker to master node, (2) card-to-card reduction, and (3) local broadcast from master to worker nodes. The worker A-HBM kernel can encapsulate (1) and (3) while the master A-HBM kernel can perform task (2) concurrently.

[0500] FIG. 48 illustrates a local broadcast mechanism for the disclosed systems, according to an embodiment of the disclosure. Taking a closer look at the figures, the number on the top left of each A-HBM represents a single dimension process ID while the coordinates represent a second dimension process ID (PID). The second coordinate PID can be used in this document. FIG. 48 can show some aspects of task (1). In the example shown, for a local reduction from worker to master node, the operation can be initiated from the very leftmost nodes (PID[x] = 0) and perform reduction at each step until the final result reaches the master node, (3, 1). The longest path of reduction takes 5 hops from: (0, 3) - (1, 3) - (2,3) - (3, 3) - (3, 2) - (3, 1).

[0501] FIG. 48 represents task (3). The local broadcast from master to worker node is the opposite process of task (1). After the card-to-card reduction process, the master node can contain the globally all-reduced data and can start broadcasting the data to all the worker nodes. The technique can finish when the leftmost nodes receive the data. The pseudocode for the worker A-HBM kernel that encapsulates the above two processes is described in FIG. 49.

[0502] As noted, FIG. 49 depicts a worker A-HBM kernel pseudocode for the disclosed systems, in accordance with embodiments of the disclosure. The algorithm describes a worker A-HBM kernel mechanism for parallel data processing. It starts by initializing a "finished" flag to False and kick-starting reduction from PID[x] == 0. The algorithm then enters a while loop that continues until all nodes have received all-reduced data. Inside the loop, it checks the PID value and performs reduction or data transfer operations accordingly. If the PID value is 0, 1, or 2, it reduces data and sends it to the next node or stores it. If the PID value is 3, it reduces data and sends it to multiple nodes. The algorithm uses conditional statements to handle different PID values and ensure efficient dataprocessing. Once all nodes have received all-reduced data, the "finished" flag is set to True, and the algorithm terminates.

[0503] FIGs. 47 and 48 and the associated pseudocode of FIG. 49 defines the process for one way of sending data as a whole to the system. For example, if each worker node contains 24KB, all of the 24KB data will be transferred through the link between the nodes at once. However, this may be an inefficient use of the links and computation in the nodes. Only a couple links and nodes will be utilized at each time, and there may be minimal parallelism available. To resolve this issue and relieve some overhead, a pipelined process can be implemented which is illustrated in FIG. 50.

[0504] In particular, FIG. 50 describes a pipelined local reduce mechanism for the disclosed systems, in accordance with embodiments of the disclosure. In one embodiment, during local reduction, one row of an A-HBM SiP is utilized as an example. The pipelined process divides data in each node into tiles and sends each tile to the next node. By sending tiled data concurrently in each stage, the disclosed systems use parallelism to efficiently utilize link bandwidth and A-HBM compute resources. In an embodiment, this pipelining process also occurs during local broadcast in an equivalent manner. This efficient pipelining can enable concurrent processing of multiple tiles across different nodes.

[0505] In one embodiment, when partial tiles are received in the master node at each time period, card-to-card reduction can be implemented immediately without waiting for all tiles to arrive. If the master node were to wait for all tiles to arrive, the parallelism achieved through pipelining may not be effective.

[0506] In one embodiment, a master A-HBM kernel describes the tiled card-to-card mechanism for efficient reduction. The pseudocode for this kernel is provided in FIG. 51.

[0507] In particular, FIG. 51 provides a master A-HBM kernel pseudocode for the disclosed systems, in accordance with embodiments of the disclosure. The algorithmdescribes a master A-HBM kernel mechanism for parallel data processing. It starts by initializing a "tiles received" counter to 0 and setting the number of tiles to 4. The algorithm then enters a while loop that continues until all tiles have been received. Inside the loop, it receives tiles from specific PIDs, reduces data, and sends it to other cards or nodes. The algorithm uses conditional statements to handle different cases and ensure efficient data processing. Once all tiles have been received, the "tiles received" counter is incremented, and the algorithm terminates.

[0508] In one embodiment, to achieve parallelism in the card-to-card reduction process, each card in the A-HBM system can identify which card receives a tile based on a predetermined equation, such as the one shown in line #7 of FIG. 51. If a card is designated to receive a tile, it performs reduction on all received tiles to produce a global all-reduced tile. This global tile cab then be sent to all other cards while simultaneously initiating local broadcast.

[0509] In one embodiment, if a card is designated to send a tile, it transmits its local all-reduced tile to the receiving card and then receives the global all-reduced tile. Upon receiving the global tile, local broadcast is initiated.

[0510] In one respect, to facilitate understanding of this process, consider an example involving two cards. The flow comprises 7 stages, each explained below. First tile is received for both cards

[0511] Card 0 will receive tile from Card 1 (tile index = 0, card # = 2; 0 % 2 = 0)

[0512] Card 0 will perform global reduction, send to Card 1 while kick-starting local broadcast; Concurrently, 2nd tile is received and this tile is sent from Card 0 to Card 1;

[0513] Card 1 receives global all-reduced 1st tile, and kick-starts local broadcast. Card 1 also received 2nd tile, performs global reduction and kick-starts local broadcast and sends it to Card 0;

[0514] Card 0 receives global all-reduced 2nd tile, and kick-starts local broadcast. Card 0 also received 3rd tile, performs global reduction and kick-starts local broadcast and sends it to Card 1 ;

[0515] Card 1 receives global all-reduced 3rd tile, and kick-starts local broadcast. Card 1 also received 4th tile, performs global reduction and kick-starts local broadcast and sends it to Card 0;

[0516] Card 1 receives global all-reduced 4th tile, and kick-starts local broadcast.

[0517] In one embodiment, when a broadcast process is initiated in a master A-HBM kernel, worker A-HBM kernels receive globally all-reduced tile data from the master and continue the local broadcast process. In an embodiment, a key component of A-HBM's allreduce mechanism is a pipelined process that introduces additional parallelism compared to conventional methods. By leveraging this parallelism, node-to-node and card-to-card link bandwidth can be utilized more efficiently, enabling concurrent computation in each A-HBM device.

[0518] FIG. 52 shows an operational flow associated with the operations of the disclosed systems, in accordance with embodiments of the disclosure. At block 5202, the disclosed systems can receive, at a management processor, a request to execute a kernel for matrix multiplication using at least a first PE of a near memory compute package. At block 5204, the disclosed systems can obtain, by the management processor, input data and the kernel from at least one of a stacked memory device or a shared memory of the near memory compute package. At block 5206, the disclosed systems can execute, by the PE, an operation associated with the kernel using the input data to obtain a result. At block 5208, the disclosed systems can store the result in a second memory device. Although FIG. 52 shows example blocks of a process for accelerating compute, in some implementations, the process may include additional blocks, fewer blocks, different blocks, or differently arranged blocks than those depicted in FIG. 52. Additionally, or alternatively, two or more of the blocks of the process may be performed in parallel.

[0519] LLM architecture

[0520] The disclosed systems can be configured to enable efficient LLM architectures using one or more A-HBM units / devices configured in a AHBM SIP architecture. LLM inference includes a prefill (summarization) phase and a decode (generation) phases. In the prefill (summarization) phase, the LLM processes the input tokens to compute the intermediate states (keys and values), which are used to generate the first new token. Each new token that is processed by the disclosed systems may be dependent on all the previous tokens processed by the systems. Since the full extent of the input is characterized and known at the system level, the processing of input tokens can be performed as a matrixmatrix operation that is highly parallelized. In the decode (generation) phase, the LLM generates output tokens autoregressively one at a time, until a stopping criteria is met. Each sequential output token needs to know all the previous iterations’ output states (keys and values). This is primarily a matrix-vector operation that underutilizes the GPU compute ability compared to the prefill phase. The speed at which the data (weights, keys, values, and activations) is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens. In other words, this is a memory-bound operation.

[0521] Application Parameters

[0522] Model Size

[0523] The disclosed systems can be configured to enable LLMs having different sizes by using A-HBM trays and / or A-HBM SiP cards that feature AHBM devices / units configured in different sizes (e.g., 2x2, 4x4, etc.). The size of a LLM can be determined by the number of parameters it contains. This size plays a a role as it often has a direct correlation with the model's capabilities. Recent advancements in LLMs, have led to the introduction of models exceeding 1 trillion parameters, which enables them to possess more capabilities and exhibit better quality. This improvement is attributed to the incorporation oftechniques such as mixture of experts and multimodal LLM. These advances have significantly improved the performance and versatility of LLMs, allowing them to excel in a wide range of applications.

[0524] Sequence Length

[0525] The disclosed systems can be configured to enable LLM models with different sequence lengths by including A-HBMs featuring stacks of HBM memory with different sizes, and / or by adding PEs and other functional units in different size formats, as variously described herein. Sequence length can include a parameter in LLM models that determines the maximum number of tokens (such as words, subwords, or characters) that can be processed or generated by the model in a single operation. The choice of tokenization method used also impacts this count. This parameter plays a role in defining the model's capacity to handle context, which directly affects its ability to produce coherent and meaningful long- form text. In other words, sequence length determines how much contextual information the model can take into account while generating text. By increasing the sequence length, the model can capture long-range dependencies within the text, resulting in more accurate and relevant outputs. For instance, in a conversation, a model with a longer sequence length can recall previous conversations and respond accordingly, providing a better user experience.

[0526] Inference Speed

[0527] The disclosed systems are configured to improve inference speed by including A-HBM units / devices that have near memory compute that reduced data movement during computations (e.g., GEMM computations) needed for inference. LLM inference performance plays a role as it directly affects the user experience. When interacting with these models, users expect swift responses, making inference speed a vital aspect to measure. There are three key metrics used to evaluate the performance of LLMs during inference:

[0528] Time to First Token (TTFT)

[0529] The disclosed systems are configured to reduce TTFT, again, by including A-HBM units / devices that have near memory compute that reduced data movement during computations (e.g., GEMM computations) needed for inference. This metric measures the time taken for the model to produce the first output token after receiving a user's input or query. This metric is related to the prefill performance of LLM inference system. For example, the average web page response time under a threshold time (e.g., 200 milliseconds) gives users the feeling of an instant response, leading to a positive experience and better engagement with the content. However, a web response time ranging between 200 milliseconds and 1 second is still considered acceptable, as users are unlikely to notice the delay. A lower TTFT indicates a more responsive model, which is particularly important in real-time interactions, such as conversational Al systems. This metric becomes less significant in offline workloads, where immediate responses are not necessary. By reducing TTFT, the disclosed systems improve system responsiveness.

[0530] Time per Output Token (TPOT)

[0531] The disclosed systems are configured to reduce TPOT, again, by including A- HBM units / devices that have near memory compute that reduced data movement during computations (e.g., GEMM computations) needed for inference. This metric represents the time needed to generate each subsequent output token for every user querying the system. TPOT reflects how users perceive the model's responsiveness. This metric is related to the decode performance of LLM inference system. For instance, if the TPOT is 10 milliseconds per token, the model can produce approximately 100 tokens per second and 6000 tokens per minute. It is known that the average reading speed for adults for non-fiction is around 238 words per minute and fiction is around 260 words per minute. That is, 10 milliseconds of TPOT is 23-25 time faster than human reading speed. A lower TPOT translates to a morerapid generation of output tokens, resulting in a smoother and more engaging user experience. By reducing TPOT, the disclosed systems improve system responsiveness.

[0532] Latency

[0533] The disclosed systems are configured to reduce latency by a variety of means, including, by offering near memory compute using A-HBM units / devices that have HBM memory near PEs and using efficient interconnects that can move data between areas of the system (e.g., cards, trays, servers, hosts, etc.). Latency can refer to the total time required for the model to generate a complete response for a user. By combining the TTFT and TPOT metrics, the overall latency can be determined as Latency = TTFT + (TPOT x number of tokens to be generated).

[0534] Modality

[0535] The disclosed systems are configured to support multimodal operations, since the A-HBM systems can be configured to store different types of data in matrix format (e.g., video, audio, text) and can perform computations on the different data types using similar operations (e.g., GEMM). Multimodality refers to the ability of an Al model to process, integrate, and generate multiple forms of data or modalities, such as text (written language, including words, sentences, and paragraphs), images (visual data, like pictures, diagrams, or charts), audio (sound waves, including speech, music, or other sounds), videos (multimedia content combining audio and video). Multimodal models can handle various types of input and output formats, enabling them to interact with users more naturally and effectively.

[0536] By supporting multiple modalities, the disclosed systems can permit user engagement with LLMs through preferred communication channels, making interactions more intuitive and accessible. For instance, the disclosed systems are configured to support a multimodal chat bot to respond to voice commands, display images, or even generate videosto convey information. Multimodal LLMs can adapt to different scenarios and applications, such as visual question answering, image captioning, speech-to-text, cross-modal translation.

[0537] System Metrics

[0538] In one embodiment, the disclosed systems are configured to have support various metrics and performance targets using architectural designs such A-HBM SiP sizes, HBM memory type and stacking size, PE number and configuration, tray configurations, and / or the like. The disclosed systems can support a throughput of prefill and decode, with example targets of approximately 8000 tokens / s and 200 tokens / s, respectively. In an embodiment, disclosed systems can support a power feature including a target thermal design power (TDP) beyond a threshold for different form factors, for example, approximately 1200W for certain card types such as those that are in accordance with 0AM (Open Compute Project (OCP) accelerator module) 3.0, 1000W for 0AM 2.0, or 600W for PCIe cards. In another embodiment, the disclosed systems can support a memory device that is in accordance with a predetermined protocol. For example, the disclosed systems can support a memory that is HBM4E compliant and that meets associated capacity and bandwidth thresholds; for example, the memory can have a capacity of approximately 48GB at 12 stack configurations and 64 GB at 16 stack configurations, and a bandwidth of approximately 2.5 TB / s. In one embodiment, the disclosed systems can support form factors including card, tray, and rack with specific targets. In example aspects, the card can have various example form factors, which can include an 0AM 3.0 card, an 0AM 2.0 (102 mm x 170 mm), and / or a PCIe full height, half length (FHFL) card (111.15 mm x 312.0 mm).

[0539] The tray can include an Open Accelerator Infrastructure (OAI) UniversalBaseboard (UBB) 3.0 based system or an OAI UBB 2.0 based system. The rack can include an OCP Rack 3.0 system. In one embodiment, the cooling system of the system can meet specific requirements for liquid cooling, including power requirements. In some respects, thedisclosed systems can be configured to operate with liquid cooling when a threshold power for liquid cooling exceeds a first amount for the base dies and cards. In one embodiment, air cooling can be used when the base die power is less than the threshold power of the base die and / or the card power is less than the threshold power of the card. Liquid cooling may be used otherwise. In an embodiment, for example, a card with an Al and HPC (High- Performance Computing) system such as a DGX Hl 00 system can have a threshold power of 700 W and a card with an OALUBB has a threshold power of about 600 W. In one embodiment, the scalability of the system is characterized by several metrics. The die-to-die bandwidth of the interconnect, with a corresponding latency and a maximum number of nodes. In one embodiment, the card-to-card bandwidth of the interconnect can be a predetermined amount, for example, approximately 448 GB / s, with a corresponding latency and a maximum number of nodes. In one embodiment, the tray-to-tray bandwidth of the interconnect, with a corresponding latency and a maximum number of nodes that utilizes Ultra Ethernet.

[0540] Power

[0541] In one embodiment, the disclosed systems can comply with a base board specification, such as OAI UBB r3.0 vl.0, which can be used to support 0AM systems with power consumption greater than a threshold, such as 1000 W.

[0542] Form Factor

[0543] In one embodiment, the disclosed systems can include an A-HBM card form factor can be based on the OCP accelerator module (0AM), while an A-HBM tray form factor can employ a baseboard that can based on a standard, such as OOP's UBB. In another embodiment, an A-HBM card may require a predetermined power budget, for example, of approximately 1,200 W. However, legacy PCIe cards have limitations in terms of maximum power budget, typically around 600W. Thus, in some respects, OOP's Open AcceleratorInfrastructure (OCI) standards can be used, such as the OCI-OAM (Open Accelerator Infrastructure - OCP Accelerator Module) and OCI-UBB. In some respects, some versions of these standards (e.g., OCI-OAM revision 2.0 version 1.0 and OCI-UBB revision 2.0 version 1.0) may not meet the required power budget. In another embodiment, for example, the maximum power budget for OCI-OAM is 1,000 W and for OCI-UBB it is 12,000 W which supports eight OAMs. Therefore in one embodiment, the disclosed systems can use and / or update related specifications to support OAMs with a maximum power consumption greater than 1,000 W.

[0544] Scalability

[0545] The disclosed systems can feature an architecture that is scalable, as shown for example, in connection with FIGs. 29-32 above. The disclosed systems can include an A- HBM tray that can provide enough A-HBM card slots, considering the LLM parameter size and the number of attention heads. Total bandwidth of the on-board interconnect can be determined by taking into account the communication topology between A-HBM cards and peak traffic bandwidth among them. The communication topology among A-HBM Cards can be modular and flexible on account of the rapidly evolving nature of LLMs. Provisioning reconfigurability of topology is one possible option, as there is no single 'one-size-fits-alf option. The disclosed systems can be configured to operate with various types of interconnect topologies from die-to-die level to tray-to-tray level.

[0546] RAS

[0547] The disclosed systems can include an A-HBM Tray as shown in connection with several figures above, including, but not limited to FIGs. 29-32 that can serve as anLLM inference server which includes various types of devices. RAS (Reliability,Availability, and Serviceability) are implemented at least at the A-HBM tray level. Moreover,the disclosed systems can include RAS parameters and configurations can be used for each level of systems such as A-HBM die level, SiP level, card level, tray level, and the rack level.

[0548] System Architecture

[0549] In some aspects, the disclosed systems including A-HBM can provide significantly higher memory bandwidth with respect to other memory systems, for example, by including more HBM stacks, resulting in improved performance. In various respects, such an A-HBM system can provides at least the advantages below for LLM inference:

[0550] A-HBM-based inference system can include a number of cards with tiles (e.g., at least 6 cards with 16 A-HBM tiles) that can achieve higher power efficiency (e.g., approximately 5x improvement in power efficiency) in measurement units of tokens / s per watt, as compared to alternative systems for certain use cases, for example, in a GPT-3 175B use case. This enhancement in energy efficiency allows a datacenter to achieve more performance in power-constrained operational conditions common for datacenter servers.

[0551] The disclosed systems can include A-HBMs that accommodate diverse LLM types by supporting various data types (e.g., including, but not limited to INT4 / 8 / 16, FP8 / 16, and BF16 data types) and allowing for the use of PEs that feature general cores as well as tensor cores (see for example, at least FIG. 13 and related discussion). Accordingly, such a versatile approach can ensure that the disclosed systems include A-HBM embodiments that remain compatible with emerging LLM architectures and methodologies.

[0552] In some aspects, the disclosed systems include A-HBMs can be scalable at least in terms of compatibility with Al use cases having different model sizes, sequence lengths, batch sizes, and number of users. For example, the system can support models exceeding 1 trillion parameters long context of 1 million sequence length. In other aspects, the disclosed systems can use interconnects using UCIe 1.1, UAL 1.1, and Ultra Ethernet to make the solution scalable.

[0553] System Architecture Overview

[0554] In some respects, unlike other GPU architectures, the disclosed systems can include A-HBM that can use PIM technology, thereby reducing the need for a separate compute die by instead integrating the compute units into memory. For example, the disclosed systems can include A-HBM compute units that can be directly placed on the buffer die of each HBM package. Such an architecture can allow more memory to be installed per SiP and can enable relatively low latency, and power-efficient memory accesses from compute units. The disclosed systems including A-HBM can maximize relative compute power by using the maximum bandwidth in certain compute phases, such as in the decode phase. Disclosed herein includes the overall architecture of an A-HBM system, configurations of an A-HBM SiP, and other aspects of the A-HBM hardware architecture and LLM inference kernels.

[0555] In some respects, the disclosed systems can include a tray that can serve as an LLM refill / decode inference accelerator. One embodiment of the tray configuration can include a predetermined number (e.g., 6 or 8) of 0AM cards, with one or more A-HBM SiP on each 0AM card, a network switch card with Ethernet or UALink, and a CPU with DDR memory.

[0556] The form factor of the tray can include an 0AM or PCIe card. In some respects, the OAMs / PCIe cards can be connected via any suitable interconnect and can be configured to connect using a network cable. The network cable type within the tray can include Octal Small Form-Factor Pluggable (OSFP). The tray can support LLM inference for determined prefill and decode thresholds; for example, for GPT3 175B prefill can be at 8000 Token / sec and decode can be at 200 Token / sec. The model size supported by the tray can have a predetermined amount based on the choice of components (EG, number and capability of A- HBM SiPs), such as 2 trillion parameters with a single tray, which uses a predeterminedamount of memory (e.g., 4.5 TB of memory), and can support more parameters (e.g., more than 2 trillion parameter) models with multi-tray scale-out.

[0557] The disclosed systems can support various parallelism types by the tray, including tensor parallelism, pipeline parallelism, data parallelism, and mixed-type parallelism. The power consumption of the A-HBM die can be a predetermined threshold, for example, approximately 70W, while the power consumption of a SiP can be a predetermined threshold based in part on the composition of the SiP (EG, number of A-HBM units). For example, the usage can be approximately 70 W for 16 A-HBM dies and 10 W for 8 dies (e.g., about 1200 W) per A-HBM SiP. Therefore, in this example, about six 0AM cards on a tray may consume around 1200 W x 6 or 7200 W of power.

[0558] As an example, the HBM memory type used in one embodiment can include HBM4E with DRAM core dies. Further, the example can include a DRAM core dies that can have 32 Gbit dies for a 12 stack architecture, a bandwidth of about 2TB / s at 8 Gbps, and 2.5TB / s at 10 Gbps. The HBM can feature 32 channels, and 64 pseudochannels. The total number of HBMs used in one embodiment can be 96 (e.g., 16 x 6). Thus, the bandwidth use equals 2 Tbps x 16 x 6 equals 192 Tbps.

[0559] The disclosed systems can include an A-HBM SiP that can have a predetermined power consumption, for example of about 1200 W, which can be configurable and based on the composition of the SiP. The form factor of the A-HBM SiP can include an 0AM card or a PCIe card, with predetermined dimensions, for example, of about 52 mm x 68 mm or about 80 mm x 80 mm. The A-HBM SiP uses HBM4E with a predetermined bandwidth, for example, 32 Gbit die in a 12 or 16 layer stack. The A-HBM SiP can also feature a UALink (for example, 224 Gbps / lane) in predetermined lane count (e.g., 16 lane), as well as UALink and PCIe endpoint / root port (EP / RP) combination. The PCIe bandwidth can be predetermined, such as 128GB / s (for Gen7 128Gbps x 8 lane), and the UALinkbandwidth can be predetermined, for example 448GB / s (224 Gbps x 16 lane). The number of I / O chiplets can be configurable based on the topology of the tray. In some embodiments, a given HBM SiP (card) based system can include multiple PEs and HBMs in a mesh / grid architecture and can include a PCIe-to-D2D connected for CPU communications. The system can, in another example, use a number of I / O chiplets (e.g., 7) for an FC (fully connected) type architecture. Additionally or alternatively, the system can include a number of I / O chiplets (e.g., 3) for a mesh type architecture, which can further use UAlink-to-D2D for a mesh network. The A-HBM SiP can have a relatively high performance in comparison with alternative systems, for example, of approximately 90 TFLOPs / die (in this example, 16 dies yield 1440 TFLOPs).

[0560] The base die size can be predetermined, for example approximately 12.5 mm x 16.5 mm when using a silicon interposer. The A-HBM die can feature a predetermined number of cores (for example, 16 PE cores), with an example performance for FP16 / BF16 TFLOP of 5.6 TFLOP at 1 GHz with 16 PEs which can yield 90 TFLOP at 1 GHz.

[0561] The M-CPU can include any suitable processor type, including but not limited to, an ARM-based processor (or similar) core that manages (e.g., configured, handles communications, coordinates, etc.) the PE cores. The SRAM density can include shared memory density. For example, the shared memory density can be about 10 MB, and a PE's TCM density can be 2MB / PE which for 16 PEs can be about 32 MB. The disclosed systems can use a Network-on-chip FlexNOC (or similar) with a mesh network, and the NOC network speed can be in an example, about 1024 Gbps at 1 GHz or about 512 Gbps at 2GHz. In some examples, the NIC can support any suitable connection such as AXI, AXI-Stream, and the mesh network node-to-node bandwidth can be about 128 GB / s.

[0562] In some aspects, the disclosed systems can include a memory management unit(MMU) (such as that shown in diagram 502 of FIG. 5) can facilitate the PE’s access to external domains such as HBM, shared-SRAM, and other resources.

[0563] The disclosed systems can include a trace / debug feature that, in an example, can include Embedded Trace Macrocell (ETMv4.5) for instruction and data trace. In another example, disclosed systems can include performance monitors extension support for software profiling and performance debugging based on, for example, a Performance Monitoring Unit version 3 (PMUv3) architecture. The disclosed systems can include an interface such as a Cross Trigger Interface (CTI) for multiprocessor debugging. Optional modules can support external devices debugging. For example, disclosed systems can include an interface for integrating a logic analyzer for advanced debug capability and signal observability. This can further be configured to obtain virtual trace info logging per PE level and M-CPU level to monitor kernel execution time / power / resource utilization.

[0564] The PEs can have a predetermined die size (e.g., 80mm2or 5 mm2) and a predetermined amount of memory (e.g., about 2 MB of memory). The PE can use a coprocessor such as an ARM-based coprocessor interface (or similar) core, which can include I-Cache, Instruction Tightly Coupled Memory (ITCM), or Data Tightly Coupled Memory (DTCM). The PE can support various data types, including FP16, BF16, FP8, INT32, INTI 6, INT8, and INT4 for tensor operations and FP32 for reduction. The PE can have an example performance of about 5.6 TFLOPs at 1 GHz. It can include, in some examples, 512 parallel math engines for EXP, FP16 DIV (for Softmax / Norm), SWiGL, GELU, sqrt, tanh, and the like.

[0565] Thermal Aspects

[0566] In order to manage thermals of the semiconductor device and package, the disclosed systems can place a clock scaling mechanism to throttle power consumption. Thedisclosed systems can include a mechanism to counteract performance degradation as the operating clock and HBM bandwidth is scaled down.

[0567] The disclosed systems can, for example, include a database that includes a comparison of performance metrics for different power consumption levels. The comparison includes various HBM bandwidth values and their associated power consumption. The disclosed systems can use a token / s / W ratio over a given core GPU (e.g., an H100) under different regimes of operation (e.g., for decode and prefill), including the performance gain at each power level. The disclosed systems can be configured to use the database to throttle the power to increase and / or decrease memory bandwidth and / or performance accordingly, for example to meet a predetermined thermal condition.

[0568] In one embodiment, the disclosed systems can use clock scaling to reduce power consumption to a level compatible with air cooling. In other aspects, the disclosed systems can be configured to determine the crossing point between air cooling and liquid cooling for a thermal-efficient packaging technology.

[0569] Power Aspects

[0570] In some respects, the disclosed systems can have a predetermined PE power budget (e.g., about 22 W for 16 PEs) and a power usage (e.g., about 4 TFLOPs / W at FP16). The disclosed systems can use a predetermined number of UCIe modules per side for both data and command signal tunneling assuming a protocol overhead of a predetermined amount such as 30 percent. In some aspects, the disclosed systems can use an efficient stream protocol is employed and the data burst transaction size is consistently large, 2 UCIe modules can be sufficient.

[0571] Performance and Power

[0572] In one embodiment, the performance of the disclosed systems can be influenced by at least two factors. The available area in the buffer die can be one such factor. This refersto the physical space allocated within the buffer die for the implementation of PIM. In one embodiment, the power budget can be another factor. This represents the maximum amount of power that can be consumed by PIM while maintaining optimal performance and efficiency.

[0573] In one embodiment, the disclosed systems can use a GEMM engine that utilizes a predetermined power, for example, of approximately 1.5 TFLOPs per mm2and 4 TFLOPs / W in some embodiments. In one embodiment, the disclosed systems can use a MATH engine that performs basic math, logs, exponentiations, trigonometric functions (sine, cosine, etc.). In one embodiment, the MATH engine can be configured to use more space (e.g., approximately 3.3 times more space) than GEMM engine per mm2. In an embodiment, the performance of a MATH engine may be determined by number of arithmetic logic units (ALUs) per HBM channel. In this scenario, certain constraints on area may be present. For example, the available area can be fixed. For example, about 80 mm2of area can be used for PIM within the buffer die. To achieve this, about 20 mm2can be taken from the default custom buffer base size and an additional 60 mm2will be added to the base die, effectively enlarging it. In terms of the power budget, a predetermined amount of total power (e.g., 35 W) can be allocated for the base die (e.g., 20 W) and reserved for PIM operations. This ensures that the memory system operates within safe power limits while delivering desired performance. For example, given the compute area (e.g., of 60 mm2) for PEs, the disclosed systems can configure MATH and GEMM engine varying the priority.

[0574] For the analysis, 80 mm2of compute module space may be used, which may provide MATH performance of 8 TFLOPS and GEMM performance of 87.23 TFLOPs. HBM4 memory that provides 2 TB / s of throughput can be used. It requires 70W of power perA-HBM. The TDP of an A-HBM SiP that includes 16 A-HBMs then becomes 1120 W.

[0575] Any of the storage devices disclosed herein may be implemented entirely or partially with, and / or used in connection with, a server chassis, server rack, dataroom, datacenter, edge datacenter, mobile edge datacenter, and / or any combinations thereof.

[0576] Any of the functionality described herein, including any of the host functionality, device functionally, and / or the like described above with respect to the embodiments disclosed herein may be implemented with hardware, software, or any combination thereof including. For example, they may be implemented with combinational logic, sequential logic, one or more timers, counters, registers, state machines, volatile memories such as DRAM and / or static random access memory (SRAM), nonvolatile memory and / or any combination thereof, CPLDs, FPGAs, ASICs, CPUs including complex instruction set computer (CISC) processors such as x86 processors and / or reduced instruction set computer (RISC) processors such as ARM processors, graphics processing units (GPUs), neural processing units (NPUs), and / or the like, executing instructions stored in any type of memory. In some embodiments, one or more components may be implemented as a system-on-chip (SOC).

[0577] The embodiments illustrated described herein, are example operations and / or components. In some embodiments, some operations and / or components may be omitted and / or other operations and / or components may be included. Moreover, in some embodiments, the temporal and / or spatial order of the operations and / or components may be varied. Although some components and / or operations may be illustrated as individual components, in some embodiments, some components and / or operations shown separately may be integrated into single components and / or operations, and / or some components and / or operations shown as single components and / or operations may be implemented with multiple components and / or operations.

[0578] Certain embodiments may be implemented in one or a combination of hardware, firmware, and software. Other embodiments may also be implemented as instructions storedon a computer-readable storage device, which may be read and executed by at least one processor to perform the operations described herein. A computer-readable storage device may include any non-transitory memory mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a computer-readable storage device may include read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, and other storage devices and media.

[0579] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. The terms “computing device”, “user device”, “communication station”, “station”, “handheld device”, “mobile device”, “wireless device” and “user equipment” (UE) as used herein refers to a wireless communication device such as a cellular telephone, smartphone, tablet, netbook, wireless terminal, laptop computer, a femtocell, High Data Rate (HDR) subscriber station, access point, printer, point of sale device, access terminal, or other personal communication system (PCS) device. The device may be either mobile or stationary.

[0580] As used within this document, the term “communicate” is intended to include transmitting, or receiving, or both transmitting and receiving. This may be particularly useful in claims when describing the organization of data that is being transmitted by one device and received by another, but only the functionality of one of those devices is required to infringe the claim. Similarly, the bidirectional exchange of data between two devices (both devices transmit and receive during the exchange) may be described as ‘communicating’, when only the functionality of one of those devices is being claimed. The term “communicating” as used herein with respect to a wireless communication signal includes transmitting the wireless communication signal and / or receiving the wireless communication signal. For example, a wireless communication unit, which is capable of communicating a wireless communicationsignal, may include a wireless transmitter to transmit the wireless communication signal to at least one other wireless communication unit, and / or a wireless communication receiver to receive the wireless communication signal from at least one other wireless communication unit.

[0581] Some embodiments may be used in conjunction with various devices and systems, for example, a Personal Computer (PC), a desktop computer, a mobile computer, a laptop computer, a notebook computer, a tablet computer, a server computer, a handheld computer, a handheld device, a Personal Digital Assistant (PDA) device, a handheld PDA device, an on-board device, an off-board device, a hybrid device, a vehicular device, a non- vehicular device, a mobile or portable device, a consumer device, a non-mobile or non-portable device, a wireless communication station, a wireless communication device, a wireless Access Point (AP), a wired or wireless router, a wired or wireless modem, a video device, an audio device, an audio-video (A / V) device, a wired or wireless network, a wireless area network, a Wireless Video Area Network (WVAN), a Local Area Network (LAN), a Wireless LAN (WLAN), a Personal Area Network (PAN), a Wireless PAN (WPAN), and the like.

[0582] Some embodiments may be used in conjunction with one way and / or two-way radio communication systems, cellular radio-telephone communication systems, a mobile phone, a cellular telephone, a wireless telephone, a Personal Communication Systems (PCS) device, a PDA device which incorporates a wireless communication device, a mobile or portable Global Positioning System (GPS) device, a device which incorporates a GPS receiver or transceiver or chip, a device which incorporates an RFID element or chip, a Multiple Input Multiple Output (MIMO) transceiver or device, a Single Input Multiple Output (SIMO) transceiver or device, a Multiple Input Single Output (MISO) transceiver or device, a device having one or more internal antennas and / or external antennas, Digital Video Broadcast (DVB)devices or systems, multi-standard radio devices or systems, a wired or wireless handheld device, e.g., a Smartphone, a Wireless Application Protocol (WAP) device, or the like.

[0583] Some embodiments may be used in conjunction with one or more types of wireless communication signals and / or systems following one or more wireless communication protocols, for example, Radio Frequency (RF), Infrared (IR), Frequency -Division Multiplexing (FDM), Orthogonal FDM (OFDM), Time-Division Multiplexing (TDM), Time-Division Multiple Access (TDMA), Extended TDMA (E-TDMA), General Packet Radio Service (GPRS), extended GPRS, Code-Division Multiple Access (CDMA), Wideband CDMA (WCDMA), CDMA 2000, single-carrier CDMA, multi-carrier CDMA, Multi-Carrier Modulation (MDM), Discrete Multi-Tone (DMT), Bluetooth™, Global Positioning System (GPS), Wi-Fi, Wi-Max, ZigBee™, Ultra-Wideband (UWB), Global System for Mobile communication (GSM), 2G, 2.5G, 3G, 3.5G, 4G, Fifth Generation (5G) mobile networks, 3 GPP, Long Term Evolution (LTE), LTE advanced, Enhanced Data rates for GSM Evolution (EDGE), or the like. Other embodiments may be used in various other devices, systems, and / or networks.

[0584] Although an example processing system has been described above, embodiments of the subject matter and the functional operations described herein can be implemented in other types of digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0585] Embodiments of the subject matter and the operations described herein can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described herein can be implemented as one or more computer programs, i.e., one or more components of computerprogram instructions, encoded on computer storage medium for execution by, or to control the operation of, information / data processing apparatus. Alternatively, or in addition, the program instructions can be encoded on an artificially-generated propagated signal, for example a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information / data for transmission to suitable receiver apparatus for execution by an information / data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (for example multiple CDs, disks, or other storage devices).

[0586] The operations described herein can be implemented as operations performed by an information / data processing apparatus on information / data stored on one or more computer- readable storage devices or received from other sources.

[0587] The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, for example an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, for example code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus andexecution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0588] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a component, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or information / data (for example one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (for example files that store one or more components, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0589] The processes and logic flows described herein can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input information / data and generating output. Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and information / data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive information / data from or transfer information / data to, or both, one or more mass storage devices for storing data, for example magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Devices suitable for storing computer program instructions and information / data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, for example EPROM, EEPROM, and flash memory devices; magnetic disks, for example internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0590] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, for example a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information / data to the user and a keyboard and a pointing device, for example a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, for example visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’ s client device in response to requests received from the web browser.

[0591] Embodiments of the subject matter described herein can be implemented in a computing system that includes a back-end component, for example as an information / data server, or that includes a middleware component, for example an application server, or that includes a front-end component, for example a client computer having a graphical user interface or a web browser through which a user can interact with an embodiment of the subject matter described herein, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form ormedium of digital information / data communication, for example a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (for example the Internet), and peer-to-peer networks (for example ad hoc peer-to-peer networks).

[0592] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits information / data (for example an HTML page) to a client device (for example for purposes of displaying information / data to and receiving user input from a user interacting with the client device). Information / data generated at the client device (for example a result of the user interaction) can be received from the client device at the server.

[0593] While this specification contains many specific embodiment details, these should not be construed as limitations on the scope of any embodiment or of what may be claimed, but rather as descriptions of features specific to particular embodiments. Certain features that are described herein in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0594] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0595] Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0596] Many modifications and other embodiments of the disclosure set forth herein will come to mind to one skilled in the art to which these embodiments pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the embodiments are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

CLAIMSWhat is claimed is:

1. A system comprising: a semiconductor system comprising: a first near memory compute package comprising at least a first stacked memory device that is communicatively coupled to at least a first processing element (PE) disposed in a layer of the first near memory compute package; a second near memory compute package comprising at least a second stacked memory device that is communicatively coupled to at least a second PE disposed in the layer, wherein the second near memory compute package is communicatively coupled to the first near memory compute package via an interconnect, and wherein at least one of the first PE and the second PE are configured to accelerate at least one matrix multiplication operation using first data stored on the first stacked memory device; and a network interface configured to connect the semiconductor system to a switch over a network connector and transmit information based on the matrix multiplication operation.

2. The system of claim 1, wherein the network connector comprises at least one optical connector of a first type that transmits at least a portion of the information at a first rate.

3. The system of claim 1, wherein the system further comprises a processor that is connected to the network interface and to the semiconductor system for managing the semiconductor system.

4. The system of claim 1, wherein the system further comprises a storage device connected to the processor for storing at least a portion of the information.

5. The system of claim 1, wherein the semiconductor system further comprises at least one a stacked memory controller configured to manage a transfer of second data between at least one of the first stacked memory device and the first PE.

6. The system of claim 1, wherein the semiconductor system further comprises a silicon interposer that communicatively connects the first near memory compute package and the second near memory compute package.

7. The system of claim 1, wherein the interconnect comprises a die-to-die interconnect configured to operate according to an interconnect protocol.

8. The system of claim 1, wherein the first near memory compute package is further connected to at least one of a first connector configured to operate with a first protocol or a second connector configured to operate with a second protocol, wherein the first protocol and the second protocol are different.

9. The system of claim 8, wherein the first connector is configured to connect the first near memory compute package of the semiconductor system to a host over a third connector configured to operate with the first protocol.

10. The system of claim 8, the second connector is configured to connect the second near memory compute package of the semiconductor system to a third near memory compute package of another semiconductor system over a third connector that is configured to operate with the second protocol.

11. The system of claim 1, wherein the first near memory compute package comprises a testing device configured to test at least one of the first PE or the first stacked memory device using a test pattern.

12. The system of claim 1, wherein the semiconductor system further comprises a management processor configured to modify a clock speed of at least one of the first PE based on a temperature associated with at least one portion of the first near memory compute package.

13. A semiconductor device, comprising: a first stacked memory device for storing first data associated with at least one matrix multiplication operation; and a first PE communicatively coupled to the first stacked memory over a channel of a network on chip (NOC) interconnect and via a first stacked memory device controller, wherein the first stacked memory device controller is configured to manage transfer of first data between the first stacked memory device and the first PE, and wherein the first PE is configured to execute the matrix multiplication operation using the first data.

14. The semiconductor device of claim 13, further comprising a shared memory configured to store a portion of the first data.

15. The semiconductor device of claim 13, further comprising a management processor configured to manage the execution of the matrix multiplication operation by configuring at least one parameter associated with the first PE.

16. The semiconductor device of claim 13, wherein the first PE comprises a scheduler that schedules operations associated with the one matrix multiplication operation using one or more of a tightly coupled memory, a functional unit, a tensor processing element, and a math engine.

17. A method for accelerating computations, comprising: receiving, at a management processor, a request to execute a kernel for matrix multiplication using at least a PE of a near memory compute package, obtaining, by the management processor, input data and the kernel from at least one of a stacked memory device or a shared memory of the near memory compute package, executing, by the PE, an operation associated with the kernel using the input data to obtain a result; and storing the result in a second memory device.

18. The method of claim 17, wherein the second memory device comprises tightly coupled memory (TCM) and wherein the method further comprises flushing the TCM.

19. The method of claim 17, wherein the method further comprises assigning an identifier to the first PE, and the execution of the operation is based on the identifier.

20. The method of claim 17, wherein, the method further comprises executing at least a first portion of the operation on one or more first channels associated with the PE.

Images