Transparent intra- and inter-node communication using a bridging network interface device for AI accelerator systems

Abstract

To provide an AI system that enables fast and efficient communication between multi-node accelerator systems and server systems, does not require explicit software activation, and provides a guaranteed delivery scheme. [Solution] An AI system uses transparent bridging to transfer data between AI accelerator devices over an intra-node or inter-node network. The invention configures a source device to transfer data assuming that the transfer protocol guarantees delivery of the data to the destination device. The receiver does not need to know the identity of the sender that transferred the data or when the data was transferred. Transparent bridging technology can be implemented using input / output (IO) streaming devices connected to switches or CPU sockets on both the source and destination devices.

Description

[Background technology]

[0001] Traditional network interface cards (NICs) that enable Ethernet connectivity face significant challenges when scaling across distributed AI accelerators. This challenge is particularly pronounced in environments involving multiple nodes. RoCEv2 (commonly known as "RDMA over Converged Ethernet"), a solution combined with RDMA-IB ("InfiniBand") fabrics, can facilitate multi-node GPU accelerator communication.

[0002] However, RDMA over Converged Ethernet combined with InfiniBand presents various limitations. Such solutions often require complex shared address spaces to support one-sided communications, making deployment and management more complex. Furthermore, both software and hardware fabric solutions face limitations when trying to meet the low-latency demands required for generative AI (GenAI) inference. These limitations hinder the ability to fully utilize the performance potential of modern accelerators.

[0003] Other conventional techniques include the use of Peripheral Component Interface Express (PCIe) topology. However, PCIe fabric topology presents scalability limitations. While PCIe can support intra-switch communication, PCIe is limited by the limited number of PCIe lanes provided by CPU sockets and PCIe switches. Therefore, PCIe adds complexity to achieving efficient multi-CPU socket peer-to-peer (P2P) connectivity between nodes.

[0004] Some PCIe switch vendors offer synthetic fabric models that enable cross-switch x16 link communication using custom firmware. Unfortunately, such synthetic fabric models remain highly specialized and have not yet seen widespread adoption.

[0005] From the above, it can be seen that techniques for scaling across distributed accelerators are highly desirable. Summary of the Invention

[0006] The present invention generally relates to integrated circuit (IC) devices and artificial intelligence (AI) systems. More particularly, the present invention relates to methods and device structures for accelerating the computational workload of neural network models (e.g., transformer models, convolutional neural network models, etc.). These methods and structures can be used in machine / deep learning applications such as natural language processing (NLP) and computer vision (CV). As an example, the present invention has been applied to AI accelerator devices and chiplet devices configured within PCIe cards.

[0007] In one example, the technique includes a transparent bridging configuration that uses software-assigned task IDs for synchronization of any data transfer across gang boundaries. In one example, the technique configures a sender that does not need to explicitly know that the data transfer has reached the receiver (i.e., the source device assumes that the protocol used for data transmission guarantees delivery of the data to the destination device). In one example, the NoC channel is assumed to be lossless, and any external links are assumed to have consistency / retry capabilities. Thus, the transmission of data from the sender signifies task completion from the sender's point of view (POV). In one example, the receiver does not need to know the identity of the sender that transferred the data or when the data was transferred. In one example, the receiver only needs to prevent reading and / or using buffer assignments for such data until the data arrives. That is, the receiver only needs a barrier to stop or obey downstream execution contingent on the data transfer to the receiver. In one example, both the sender and receiver are agnostic to the other, and the only entity aware of their relationship is the compiler configured on the host central processing unit. Thus, the SW-assigned task ID serves as a means of establishing a producer-consumer relationship for long-distance data transfer. This configuration can be extended to a multi-host configuration, such as a multi-node server system, where the compiler can run on multiple host processing units and configure multiple senders and receivers.

[0008] In one example, the present invention provides a system for PCIe peer-to-peer (P2P) writes between different sets of accelerators using transparent bridging via an input / output (IO) streaming device, which may be configured as a transparent network interface card (NIC) device. In this example, the IO streaming device is shown as exposing the base address registers (BARs) of a neighboring card and representing them as mirrored BARs. However, the system is not limited to communication between two neighboring cards. In one example, the system may be configured to provide communication between nodes (e.g., servers) for tensor parallelism, pipeline parallelism, etc. In one example, the system provides a distribution of the number of mirrored BARs exposed per card determined by the bandwidth and latency requirements of the workload. Such requirements differ depending on whether the workload includes tensor-level parallelism or pipeline parallelism. To meet the bandwidth and latency requirements for tensor-level parallelism, additional IO streaming devices per node can be integrated using this technique for scalability and high-performance AI workloads.

[0009] By using transparent bridging, the present invention can enable fast and efficient communication across multi-node accelerator and server systems. No explicit software enablement for transparent bridging is required, and delivery of data using transparent bridging can be guaranteed through the application of specified communication protocols for scale-up and scale-out. [Brief explanation of the drawings]

[0010] [Figure 1A] 1 is a simplified block diagram of an AI accelerator device according to an example of the present invention; FIG. [Figure 1B] 1 is a simplified block diagram of an AI accelerator device according to an example of the present invention; FIG. [Figure 2A] FIG. 1 is a simplified block diagram illustrating a 16-slice chiplet device according to an example of the present invention. [Figure 2B] FIG. 1 is a simplified block diagram illustrating a 16-slice chiplet device according to an example of the present invention. [Figure 3A] 1 is a simplified block diagram illustrating a slice device according to an example of the present invention; [Figure 3B] 1 is a simplified block diagram illustrating a slice device according to an example of the present invention; [Figure 4] FIG. 2 is a simplified block diagram illustrating an in-memory computing (IMC) module according to an example of the present invention. [Figure 5A] 3 is a simplified block flow diagram illustrating the numeric format of data processed in a slicing device according to an example of the present invention. [Figure 5B] Schematic diagram showing an example of a number format. [Figure 6A] A simplified block diagram of the transformer architecture. [Figure 6B] FIG. 1 is a simplified block diagram illustrating the autoregressive token processing method in a transformer decoder architecture. [Figure 7] FIG. 1 is a simplified block flow diagram illustrating a mapping process between a transformer and an AI accelerator device according to an example of the present invention. [Figure 8A] 1 is a schematic diagram illustrating a transformer calculation unit according to an embodiment of the present invention; [Figure 8B] 1 is a schematic diagram illustrating a transformer calculation unit according to an embodiment of the present invention; [Figure 9] FIG. 1 is a simplified block diagram illustrating an AI accelerator hardware-software system according to an example of the present invention. [Figure 10] 1 is a simplified block diagram illustrating a server system according to an example of the present invention; [Figure 11] 1 is a simplified block diagram illustrating a multi-node server system according to an example of the present invention; [Figure 12]1 is a simplified block diagram illustrating a portion of a server system according to an example of the present invention. [Figure 13A] 1 is a simplified block diagram illustrating a server system using transparent bridging with synthetic fabric switch connection capabilities, according to an example of the present invention; [Figure 13B] 1 is a simplified block diagram illustrating a server system using transparent bridging for intra-node switch connection capabilities and inter-node connection capabilities according to an example of the present invention; [Figure 13C] 1 is a simplified block diagram illustrating a server system using transparent bridging for intra-node socket connection capabilities and inter-node connection capabilities according to an example of the present invention; [Figure 14] 1 is a simplified block diagram illustrating a multi-node server system using transparent bridging for scale-up and scale-out according to an example of the present invention; [Figure 15] 1 is a simplified block diagram illustrating an input / output (IO) streaming device according to an example of the present invention. [Figure 16] FIG. 2 is a simplified block diagram illustrating an IO streaming data format according to an example of the present invention. [Figure 17A] 1A-1C are simplified block diagrams illustrating how data is transmitted and received using transparent bridging, respectively, according to an example of the present invention. [Figure 17B] 1A-1C are simplified block diagrams illustrating how data is transmitted and received using transparent bridging, respectively, according to an example of the present invention. [Figure 18A] 2 is a simplified block flow diagram illustrating a method for transmitting data in a server system having an accelerator device according to an example of the present invention. [Figure 18B] 1 is a simplified block flow diagram illustrating a method for transmitting data in a server system having an accelerator device using transparent bridging according to an example of the present invention. [Figure 18C] 1 is a simplified block flow diagram illustrating a method for transmitting data in a server system having an accelerator device using buffered transparent bridging according to an example of the present invention. [Figure 18D] 1 is a simplified block flow diagram illustrating a method for transmitting data in a server system having accelerator devices, each having a data path controller and one or more subsystems, according to an example of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present invention relates generally to integrated circuit (IC) devices and artificial intelligence (AI) systems. More particularly, the present invention relates to methods and device structures for accelerating the computational workload of neural network models (e.g., transformer models, convolutional neural network models, etc.). The methods and structures can be used in machine / deep learning applications such as image recognition and processing, and elsewhere. By way of example only, the present invention has been applied to AI accelerator apparatus and chiplet devices configured to perform high-throughput convolution operations.

[0012] Currently, most NLP models are based on transformer models, such as the Bidirectional Encoder Representations from Transformers (BERT) model, the BERT-Large model, and Generative Pre-trained Transformer (GPT) models such as GPT-2 and GPT-3. However, these transformers have very high computational and memory requirements. According to one example, the present invention provides an apparatus using a chiplet device configured to accelerate transformer operations for AI applications. An example of an AI accelerator apparatus is shown in Figures 1A and 1B.

[0013] FIG. 1A illustrates a simplified AI accelerator apparatus 101 having two chiplet devices 110. The illustrated chiplet devices 110 are connected to one another by one or more die-to-die (D2D) interconnects 120. Each chiplet device 110 is also connected to a memory interface 130 (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic RAM (SDRAM), etc.). The apparatus 101 also includes a substrate member 140, which provides mechanical support for the chiplet devices 110 configured on a surface region of the substrate member 140. The substrate may include an interposer, such as a silicon interposer, a glass interposer, or an organic interposer. The chiplets may be connected to one or more interposers, which may be configured to enable communication between the chiplets and other components (e.g., act as a bridge or conduit to enable electrical signals to pass between internal and external elements).

[0014] 1B shows a simplified AI accelerator apparatus 102 having eight chiplet devices 110 arranged in two groups of four chiplets on a substrate member 140. Each chiplet device 110 in a group is connected to the other chiplet devices by one or more D2D interconnects 120. The apparatus 102 also shows a DRAM memory interface 130 connected to each of the chiplet devices 110. The DRAM memory interface 130 may be connected to one or more memory modules, represented by a "Mem" block.

[0015] The illustrated AI accelerator devices 101 and 102 are implemented in a peripheral component interconnect express (PCIe) card form factor. However, the AI ​​accelerator devices may be configured in other form factors. These PCIe card form factors may be configured in various dimensions (e.g., full height-full length (FHFL), half height-half length (HHHL), etc.) and mechanical sizes (e.g., 1×, 2×, 4×, 16×, etc.). In one example, one or more substrate members 140, each having one or more chiplets, are connected to the PCIe card. One skilled in the art will recognize other variations, modifications, and alternatives to these elements and configurations of the AI ​​accelerator devices.

[0016] An embodiment of the AI ​​accelerator device can implement several techniques to improve performance (e.g., computational efficiency) in various AI applications. The AI ​​accelerator device can include digital in-memory computing (DIMC) for integrating compute functions and memory fabric. Mapper, numerical computation, and sparsity-related algorithms can be optimized within the compute fabric. And, the use of chiplets and interconnects organized on an organic interposer can provide modularity and scalability.

[0017] According to one example, the present invention implements chiplets with in-memory computing (IMC) capabilities that can be used to accelerate the computations required by transformer workloads. Computations for training these models can include running a scaled dot-product attention function to determine a probability distribution associated with an expected outcome in a particular AI application. When training an NLP model, the expected outcome can include predicting a subsequent word, determining the meaning of a word in context, translating into another language, etc.

[0018] A chiplet architecture may include multiple slice devices (or slices) controlled by a central processing unit (CPU) to perform transformer operations in parallel, with each slice being a modular IC device capable of processing a portion of the operations. The slices may be divided into one or more tiles / gangs (i.e., subsets) of slices, with a CPU connected to each of the slices within the tile. The tile CPUs may be configured to perform transformer operations in parallel using each of the slices within the tile. A global CPU may be connected to each of these tile CPUs and configured to use the tile CPUs to perform transformer operations in parallel using all of the slices in one or more chiplets. Further details of chiplets are described with reference to FIGS. 2A-5B, and transformers are described with reference to FIGS. 6-9.

[0019] 2A is a simplified block diagram illustrating an example of the configuration of a 16-slice chiplet device 201. In this example, the chiplet 201 includes four tile devices 210, each of which includes four slice devices 220, a CPU 221, and a hardware dispatch (HW-DS) device 222. In a specific example, these tiles 210 are arranged symmetrically. As described above, the CPU 221 of one tile 210 can coordinate operations performed by all slices within that tile. The HW-DS 222 is connected to the CPU 221 and can be configured to coordinate control of the slices 220 within the tile 210 (e.g., determine which slice within the tile processes the target portion of a transformer operation). In a specific example, the CPU 221 can be a reduced instruction set computer (RISC)-CPU or the like. Additionally, CPU 221 can be connected to a dispatch engine, which is configured to coordinate control of CPU 221 (eg, determine which portions of a transformer operation are processed by which CPU).

[0020] The CPUs 221 of each tile 210 may be connected to a global CPU via a global CPU interface 230 (e.g., a bus, connector, socket, etc.). This global CPU may be configured to coordinate processing of all chiplet devices in an AI accelerator device, such as device 101 and device 102 shown in FIGS. 1A and 1B, respectively. In one example, the global CPU may use the HW-DS 222 of each tile to cause its associated CPUs 221 to perform various portions of transformer operations between slices within the tile. The global CPU may also be a RISC processor, etc. The chiplet 201 also includes a D2D interconnect 240 and a memory interface 250, both of which are connected to each of the CPUs 221 in each tile. In one example, the D2D interconnect 240 may be configured with single-ended signaling. The memory interface 250 may include one or more memory buses connected to one or more memory devices (e.g., DRAM, SRAM, SDRAM, etc.).

[0021] Additionally, chiplet 201 includes a PCIe interface / bus 260 connected to each of the CPUs 221 in each tile. The PCIe interface 260 may be configured to communicate with a server or other communication system. In the case of a multiple chiplet device, the main bus device is connected to the PCIe bus 260 of each chiplet device using a master chiplet device (e.g., the main bus device is also connected to the master chiplet device). This master chiplet device is interconnected with the chiplet devices using at least the D2D interconnect 240. The master chiplet device and the main bus device may be configured to overlay a substrate member (e.g., the same substrate as the chiplets or a separate substrate). An apparatus integrating one or more chiplets may also be connected to a power supply (e.g., configured on-chip, configured within the system, or connected externally) and may be configured and operate as a server, network switch, or host system using the main bus device. The server apparatus may be one of multiple server apparatuses configured in a server farm in a data center or other similar configuration.

[0022] In a specific example, an AI accelerator device configured for GPT-3 can incorporate eight chiplets (similar to device 102 in FIG. 1B). The chiplets can be configured with a D2D 16x16 Gb / s interconnect, a 32-bit LPDDR5 6.4 Gb / s memory module, and a 16-lane PCIe-Gen5-PHY-NRZ 32 Gb / s / lane interface. LPDDR5 (16x16 GB) can provide the capacity, bandwidth, and low power required for large-scale NLP models such as quantized GPT-3. Of course, other variations, modifications, and options are possible.

[0023] FIG. 2B is a simplified block diagram illustrating an example configuration of a 16-slice chiplet device 202. Similar to chiplet 201, chiplet 202 includes four gangs 210 (or tiles), each of which includes four slice devices 220 and a CPU 221. The CPU 221 of each illustrated gang / tile 210 is connected to each of the slices 220 and to each other with the CPUs 221 of other gangs / tiles 210. In one example, tiles / gangs function as neural cores, and slices function as computational cores. With this multi-core configuration, the chiplet device can be configured to receive and execute several operations in parallel. Each CPU 221 is also connected to a global CPU interface 230, a D2D interconnect 240, a memory interface 250, and a PCIe interface 260. As described with respect to FIG. 2A, the global CPU interface 230 connects to a global CPU that controls all of the CPUs 221 in each gang 210 .

[0024] 3A is a simplified block diagram of an example chiplet slice device 301. In the example of a 16-slice chiplet, slice device 301 includes a compute core 310 with four computation paths 312, each of which includes, connected to one another, an input buffer (IB) device 320, a digital in-memory computing (DIMC) device 330, an output buffer (OB) device 340, and a single instruction, multiple data (SIMD) device 350. Each of the paths 312 is connected to a slice crossbar / controller 360, which is controlled by the tile CPU to coordinate the operations performed on each path 312.

[0025] In one example, a DIMC is connected to a clock and configured within one or more portions of each of a plurality of slices of a chiplet, enabling high throughput of one or more matrix operations provided in the DIMC, where the high throughput is characterized by 512 multiply-accumulate per clock cycle. In a specific example, the clock connected to the DIMC is a second clock derived from a first clock (e.g., a chiplet clock generator, an AI accelerator device clock generator, etc.) configured to output a clock signal on the order of 0.5 GHz to 4 GHz, where the second clock may be configured with an output rate approximately half that of the first clock. The DIMC may be configured to support block-structured sparsity (e.g., to impose structural constraints on the weight patterns of neural networks such as transformers).

[0026] In one example, SIMD device 350 is a SIMD processor connected to the output of the DIMC. SIMD 350 may be configured to process one or more nonlinear operations and one or more linear operations in a vector process. SIMD 350 may be a programmable vector unit or the like. SIMD 350 may also include one or more random access memory (RAM) modules, such as a data RAM module, an instruction RAM module, etc.

[0027] In one example, slice controller 360 is connected to all blocks in each computation path 312 and includes a control / status register (CSR) 362 connected to each computation path. Slice controller 360 is connected to memory banks (GM) 370 and data reshape engine (DRE) 380. Slice controller 360 may be configured to provide data from memory banks 370 to each block in computation paths 312 and coordinate these computation paths 312 through processor interface (PIF) 364. In a specific example, PIF 364 is connected to SIMD 350 in each computation path 312.

[0028] Further details of the compute core 310 are shown in Figure 3B. A simplified block diagram of the slice device 302 includes an Input Buffer 320, a DIMC Matrix-Vector Unit 330, an Output Buffer 340, a Network-on-Chip (NoC) device 342, and a SIMD Vector Unit 350. The DIMC unit 330 includes multiple in-memory computing (IMC) modules 332 configured to compute scaled dot-product attention functions on input data to determine probability distributions requiring high-throughput matrix multiply-accumulate operations.

[0029] Additionally, these IMC modules 332 may be connected to a Block Floating Point Alignment module 334 and a Partial Products Reduction module 336 for further processing before outputting the DIMC results to an output buffer 540. In one example, the input buffer 320 receives input data (e.g., data vectors) from a memory bank 370 (shown in FIG. 3A ) and sends the data to the IMC modules 332. The IMC modules 332 may also receive instructions from the memory bank 370.

[0030] Further to the above details, the SIMD 350 may be configured as an element-wise vector unit. The SIMD 350 may include an arithmetic unit 352 (e.g., add, subtract, multiply, max, etc.), a look-up table (LUT) 354, and a state machine (SM) module 356 configured to receive one or more outputs from the output buffer 340.

[0031] The NoC device 342 is connected to an output buffer 340 configured in a feed-forward loop via shortcut connections 344. The NoC device 342 is also connected to each of the slices and configured for multicast and unicast processes. More specifically, the NoC device 342 can be configured to connect all of the slices and all of the tiles, connect multicast input activations to all of the slices / tiles, and collect sub-operations to be unicast for special distributed accumulations.

[0032] In the example of the eight-chiplet AI accelerator device mentioned above, the input buffer may have a capacity of 64 KB with 16 banks, and the output buffer may have a capacity of 128 KB with 16 banks. The DIMC may be an 8-bit block with dimensions 64x64 (eight 64x64 IMC modules), and the NoC may have a size of 512 bits. The arithmetic blocks in the SIMD may be configured for 8-bit and 32-bit integer (int) and unsigned integer (uint) operations. These slice components may vary depending on which transformer the AI ​​accelerator device supports.

[0033] 4 is a simplified block diagram of an example IMC module 400. The illustrated module 400 includes one or more operation tree blocks 410 configured to perform expected operations on input data from one or more read / write blocks 420. Each of the read / write blocks 420 includes one or more first memory selection units 422 (denoted "W"), one or more second memory selection units 424 (denoted "I"), an activation multiplexer 426, and an operator unit 428. The first memory selection units 422 provide inputs to the operator units 428, while the second memory selection units 424 control the activation multiplexers 426 connected to the operator units 428. For a multiply-accumulate operation, the operator units 428 are multiplier units, and the operation tree block 410 is a multiplier adder tree block (i.e., Σx.w).

[0034] As shown in the enlarged portion 401 of the figure, each of the memory selection units 422, 424 includes a memory cell 430 (e.g., an SRAM cell) and a selection multiplexer 432. Each of the memory selection units 422, 424 is connected to a read / write controller 440, which is also connected to a memory bank / driver block 442. In one example, the read / write controller 440 can be configured with column write drivers and column read sense amplifiers (Column Write Drivers & Column Read Sense Amplifiers), and the memory bank / driver block 432 can be configured with sequential row select drivers (Memory Bank & Sequential Row Select Drivers).

[0035] An input activation controller 450 may be connected to the activation multiplexer 426 of each read / write block 420. The input activation controller 450 may include precision & sparsity aware input activation registers and drivers. The arithmetic unit 428 receives the output of the first memory selection unit 422 and the output of block 450 through the activation multiplexer 426, which is controlled by the output of the second memory selection unit 424. The output of the arithmetic unit 428 is then provided to the arithmetic tree block 410.

[0036] The input activation block 450 is also connected to a clock source / generator (CLK-GEN) 460. As described above, the clock generator 460 can generate a second clock derived from a first clock configured to output a clock signal on the order of 0.5 GHz to 4 GHz, where the second clock can be configured to have an output rate approximately half that of the first clock. The clock generator 460 is connected to one or more sign- and precision-aware accumulators (S&PAA) 470 configured to receive the output of the arithmetic tree block 410. In one example, the accumulator 470 is configured to receive the output of two arithmetic tree blocks 410. Example output readings of the IMC are shown in Figures 13A-13C.

[0037] Referring back to the example of an 8-chiplet AI accelerator device, the memory cells may be dual-bank 2x6T-SRAM cells, and the selection multiplexer may be an 8T bank selection multiplexer. In this case, the memory bank / driver block 442 includes a dual-bank SRAM bank. Also, the read / write controller may include a 64-byte write driver and a 64-byte read sense amplifier. Those skilled in the art will recognize other variations, modifications, and options for these IMC module components and their configurations.

[0038] Figure 5A is a simplified block flow diagram illustrating an example of numeric formatting of data processed in a slice. Diagram 501 shows a loop with data formatting from GM / Input Buffer 510, IMC 520, Output Buffer 530, SIMD 540, and NoC 550, which feeds back to GM / Input Buffer 510. IMC block 520 represents a multiply-accumulate (Σx,w) operation. Additionally, data formatting by IMC 521 also flows to Output Buffer 530. In this example, numeric formats include integer (int), floating-point (float), and block-float (bfloat) of varying lengths.

[0039] Figure 5B is a simplified diagram of example numeric formats, including the particular format shown in Figure 5A. Block floating-point values ​​can be used to address certain performance barriers. Transformer training is typically performed in floating point, i.e., 32-bit float or 16-bit float, and inference is typically performed in 8-bit integers ("int8"). In block floating point, a single exponent is shared across a group of mantissa significant values ​​(see the shaded blocks in the int8 vector at the bottom of Figure 5B). This contrasts with floating point, where each mantissa has a separate exponent (see the 32-bit float and 16-bit float formats at the top of Figure 5A). Using block-floating-point numeric formats during training can offer fixed-point efficiency without the problems of integer arithmetic, and may also allow the use of smaller mantissas, such as 4-bit integers ("int4"), while retaining precision. Furthermore, by using block-floating-point formats (e.g., for activations, weights, etc.) and sparsity, inference of trained models can be accelerated for better performance. Those of ordinary skill in the art will recognize other derivations, variations, and options for these numeric formats used to process Transformer workloads.

[0040] Figure 6A shows a simplified Transformer architecture 601. A typical Transformer can be described as having an encoder stack configured with a decoder stack, each of which may have one or more layers. Within the Encoder layer 610, a Self-Attention layer 612 encodes input data and determines context information, feeding the encoded data to a Feed Forward Neural Network 616. The Encoder layer 610 processes the input sequence from bottom to top and converts the output into a set of attention vectors K and V. The Decoder layer 620 also includes a corresponding Self-Attention layer 622 and Feed Forward Neural Network 626, and may further include an Encoder-Decoder Attention layer 624, which uses attention vectors from the Encoder stack to assist the decoder in further context processing. The decoder stack outputs a floating-point vector (as described in FIG. 5B), which is fed into a linear and softmax (LINEAR+SOFTMAX) layer 630 to project the output into a final expected result (e.g., an expected word prediction, interpretation, or translation). The linear layer is a fully connected neural network that projects the decoder output vector into a larger vector (i.e., a Logits vector) containing scores associated with all potential results (e.g., all potential words), and the softmax layer converts the scores into probabilities. Based on the probability output, the predicted word meaning can be selected based on the highest probability or by other derived criteria depending on the application.

[0041] An important class of Transformer models includes those based solely on decoder stacks (e.g., Transformer language models such as GPT-2, GPT-3, etc.), which pose particular challenges for inference. Figure 6B shows an example of an autoregressive token processing method, using a decoder stack to process one token at a time from segment 602 ("the quick brown fox jumps over the lazy dog"). This decoder stack includes multiple DECODER layers 620, each with a Self-Attention layer 622 and a Feed Forward layer neural network 626. Here, the input token "fox" is generated from its previous tokens "brown," "quick," and "the," as well as the starting token " <s>" is processed after ".

[0042] In one example, a query (Q), key (K), and value (V) vector is created for each input token by multiplying each token by the Q, K, and V weight matrices, respectively. The processing method for each token includes computing a dot product of the current token's Q vector with the K vector of each token in the segment to determine each token's percentage score relative to the current token, as shown in the self-attention layer 622. In this example, the method includes a masked self-attention process in which future tokens are scored as zero, i.e., score values ​​are calculated only for the current token and all previous tokens. Each token's value vector is then multiplied by its respective score value and all summed, resulting in the current token's output vector. The output vector for a given token is then passed to the next decoder layer 620 for further processing (indicated by the shaded vector following the dotted arrow). Once the Q, K, and V vectors are created for each token, the Q, K, and V matrices containing these vectors grow dynamically with each additional token (as illustrated by the right diagram of the self-attention layer 622).

[0043] The transformer is based on four parameters: sequence length (S) (i.e., number of tokens), number of attention heads (A), number of layers (L), and embedding length (H). Variations of these parameters are used to build practically all current transformer-based models. Embodiments of the present invention can be configured for all similar model types.

[0044] Transformers start out untrained and are pre-trained by exposing them to a desired dataset for a desired learning application. Transformer-based language models are exposed to large amounts of text (e.g., Wikipedia) to train them for language processing functions such as predicting the next word in a text sequence or translating text into another language. This training process involves converting text (e.g., words or parts of words) into token IDs, evaluating the context of the tokens with a self-attention layer, and predicting the results with a feedforward neural network.

[0045] The self-attention process involves (1) determining query (Q), key (K), and value (V) vectors for each word embedding in the input sentence, (2) calculating a score for each word in the input sentence from the dot product of Q and K against the target word, (3) dividing that score by the square root of the dimension of K, (4) running the result through a softmax operation to normalize the score, (5) multiplying each V by the softmax score, and (6) summing the weighted V vectors to generate the output. Notably, the value matrix V becomes the weight matrix for the matrix multiplication with the softmax attention matrix, and in terms of block-floating-point values, this requires a column-block converter for V, as explained below. Column-wise blocking of V is more complex in decoder conversion architectures, where the V matrix grows one row at a time for each additional token input. In the case of column-wise blocking, this requires requantizing the last matrix tile to block-floating point for each additional row of V.

[0046] Many things affect the performance of the transformer architecture. The softmax function tends to be the critical path of the transformer layer (and has been difficult to accelerate in hardware). The requirement to overlap computational processing, SIMD processing, and NoC transfer also impacts performance. Furthermore, the efficiency of NoC, SIMD, and memory bandwidth utilization is equally important.

[0047] 7 is a simplified block flow diagram illustrating the mapping process between a transformer and an example AI accelerator device. The illustrated transformer 701 includes multiple transformer layers 710, each having an attention layer 702. In this example, there are 16 attention heads 720 (e.g., BERT Large) that compute attention functions, as described above. These 16 attention heads are mapped to 16 slices 730 of an AI accelerator device 703 (similar to device 201 and device 202) via a global CPU 732 that communicates with a tile CPU 734.

[0048] According to one example, the present invention relates to processing transformer workloads in a transformer computing device. In certain applications, it is desirable to improve the handling of large data sizes. For example, transformer-based modeling networks typically involve a huge number of elements (e.g., weights, activations, etc.) that cannot all be stored in on-chip memory. Therefore, accessing these elements requires frequent transfers from a memory storage device (e.g., DDR), which can make processing these elements memory-bound due to the long latency of such memory operations. In addition, quantizing data into a particular format can pose challenges when the target matrix data is characterized by redundant quantization, potential precision loss, and shrinking dimensionality changes due to inefficient memory / cache transfers.

[0049] FIG. 8A is a simplified diagram of a transformer processing unit 801 (or matrix multiplication processing unit) according to one embodiment of the present invention. The illustrated device may be configured similarly to the slice device 301 illustrated in FIG. 3A. Common reference numerals between these figures refer to the same elements as previously described. Meanwhile, the device 801 includes a cache memory device 830 connected to a crossbar (Slice XBar) 360 and a memory device (GM) 370. The cache memory device 830 may include at least a first cache device 832 and a second cache device 834. The cache memory device 830 may also include additional cache devices.

[0050] The apparatus 801 also includes a crossbar converter (XBar Conv) device 810 connected to the crossbar 360, an input buffer (IB) device 320, and a weight buffer (WB) device 820 connected to the computing device 330. The converter device 810 may receive data directly from the output buffer (OB) device 340 or from the memory device 370 or the cache memory device 830 via the crossbar device 360. The converter device 810 may then convert the data from the first format to the second format by determining a mantissa value and an exponent value from the data in the first format. These mantissa and exponent are then stored in a blocking configuration in a designated memory location (e.g., the memory device 370, the cache memory device 830, etc.). In a specific example, the first format may be a floating-point (FP) format, while the second format may be a block-floating-point (BFP) format. Additionally, the crossbar device 360 ​​may transmit the converted data to the IB device 320 and / or the WB device 820 for processing by the computing device 330.

[0051] In one example, the WB device 820 can be configured as one buffer device together with the IB device 320. Also, the crossbar converter device 810 can be configured together or separately within each computation path 312. Alternatively, the crossbar converter device 810 can be configured within the crossbar device 360 ​​and connected to each computation path 312.

[0052] 8B is a simplified diagram of a transformer processing unit 802 (or matrix multiplication processing unit) according to one example of the present invention. The illustrated unit 802 may be configured similarly to the slice device 302 illustrated in FIG. 3B. However, the unit 802 includes a weight buffer (WB) device 820 connected to an in-memory computing (IMC) module 332. Similar to the input buffer (IB) device 320, the WB device 820 is also connected to a network-on-chip (NOC) device 342 and a memory device (indicated by an input from "GM"). As mentioned above, the WB device 820 may be configured together with the IB device 320.

[0053] The apparatus includes at least a data path having an IB device, a computing device connected to the IB device, an OB device connected to the computing device, and a SIMD device connected to the OB device. One or more of these data paths, and each of the components therein, are connected to a crossbar device, which is connected to at least a memory device. Furthermore, a crossbar converter device may be configured within the crossbar device or within each data path connecting the crossbar device and the OB device. In a specific example, the transformer computing unit may be configured as a low-precision, high-accuracy system for generative large language models (LLMs) by supporting BFP numerical methods and storage. The apparatus may also be configured within a chiplet device and / or an AI accelerator device. Depending on the embodiment, the apparatus may include any of the elements and configurations described above.

[0054] 9 is a simplified block diagram of an AI accelerator hardware-software system 900 according to one embodiment of the present invention. The Host Computing Device 910 shown in the system 900 includes at least a Host Runtime 912 running a Compiler Stack 920, a Workload Preprocessor 930, and an Execute Stack 940. This hardware-software system can be configured to perform ML and language model computing using a dataflow architecture and in-memory computing units to process neural network model workloads (e.g., tensor operations, etc.). This configurable system embodiment allows for the selection of computing throughput, latency, energy consumption, and functional accuracy.

[0055] Compiler stack 920 includes at least a Handles layer 922 and an Instruction Set Architecture Graph (ISA graph) layer 924. Host runtime 912 can use Handles layer 922 to determine references to resources for a neural network model workload, and host runtime 912 can use ISA graph layer 924 to translate an operation graph representing the target neural network model workload in machine code.

[0056] The workload preprocessor 930 may be configured to determine a plurality of workload parameters using the translated operation graph from the ISA graph layer 924. The host runtime 912 may then use the compiler stack 920 to issue the commands and instructions of the workload parameters to the execution stack 940, which sends these commands to the target hardware. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to the configuration of the host computing device 910 and associated software systems.

[0057] In one example, the target hardware includes an AI accelerator apparatus 950 with multiple chiplet devices 960 connected to a CPU 962, which may include a global CPU and multiple local CPUs. The chiplet CPUs 962 are connected to multiple matrix compute apparatuses 970 via respective crossbar devices 972. Each of the crossbar devices 972 is connected to at least a compute device 974 (e.g., a DIMC device) and a single-input multiple-data (SIMD) device 976. In one example, compiler commands are sent to the AI ​​accelerator apparatus 950 and can be used to program the CPU 962 (or multiple CPUs) and elements of the matrix compute apparatus 970 connected via the crossbar device 972. The AI ​​accelerator apparatus 950, chiplet devices 960, and matrix compute apparatus 970 can be configured similarly to any of the previous examples.

[0058] Although matrix operation unit 970 is configured within chiplet device 960 within AI accelerator unit 950 in this example, host computing device 910 may also be configured to send compiler commands to an independent chiplet device having a matrix operation unit or to a server system having multiple AI accelerator units. For example, a server system may include multiple AI accelerator PCIe card devices connected to multiple switches, each connected to one or more server CPUs. Those skilled in the art will recognize other variations, modifications, and options for this workload transfer configuration.

[0059] FIG. 10 is a simplified block diagram of a server system according to an example embodiment of the present invention. The illustrated server system 1000 includes multiple processor devices (CPUs) 1010, each connected to one or more memory devices 1020 and a network interface controller (NIC) device 1030. In one example, the memory devices 1020 may include hard disk drives (HDDs) or solid-state drives (SSDs), such as E1.S-SSDs. Here, each of the processor devices 1010 is connected to three memory devices 1020 (shown as S0-S2). Each processor device 1010 may also be connected to one or more processor devices in a multiprocessor configuration. In a specific example, the processors in the multiprocessor configuration may be connected using a point-to-point processor interconnect, such as an Ultra Path Interconnect (UPI). In FIG. 10, system 1000 includes four multiprocessors, each of which has a first processor device 1010 connected to a second processor device (CPU) 1012 .

[0060] The system 1000 also includes multiple switch devices 1040 connected to the processor devices 1010 and 1012. These switch devices 1040 may be configured for various form factors (standards), such as PCIe (Peripheral Component Interface Express). Each switch device 1040 is connected to each of the other switch devices (e.g., using PCIe cables). In the specific example, the specific connections between the switches 1040 may be configured for pipeline traffic or host traffic. In FIG. 10, the system 1000 includes four switch devices 1040 (shown as Sw0-Sw3) connected to the processor devices 1010 and 1012, such that the second processor device 1012 is connected to a different switch device 1040 than the first processor device 1010.

[0061] Here, a first processor device 1010 of a first multiprocessor is connected to a first switch device 1040, and a second processor device 1012 of the first multiprocessor is connected to a second switch device 1040. Similarly, a first processor device 1010 of a second multiprocessor is connected to a first switch device 1040, and a second processor device 1012 of the second multiprocessor is connected to a second switch device 1040. The third and fourth multiprocessors have a similar configuration using third and fourth switch devices 1040. Although system 1000 shows this pairwise connection configuration between the first and second processor devices 1010, 1012 and switch device 1040, the connection configuration can be scaled to a larger subset of switch devices 1040 with additional processor devices in the multiprocessor.

[0062] Additionally, each switch device 1040 is connected to one or more processing unit (PU) devices 1050, which may include GPU configurations, TPU configurations, etc. These PU devices (PU CARDs) 1050 may include the AI ​​accelerator device configurations described above, which may include various form factors such as PCIe. In the case of a PCIe card configuration, these PU devices 1050 may be configured similarly to the AI ​​accelerator devices 101 and 102 of FIGS. 1A and 1B. In FIG. 10, the system 1000 includes four PU devices (labeled PU0-PU3). Those skilled in the art will recognize other variations, modifications, and options for this server system configuration.

[0063] FIG. 11 is a simplified block diagram of a multi-node server system according to one embodiment of the present invention. The illustrated multi-node server system 1100 includes at least two server systems 1000 (see FIG. 10) configured as interconnected server nodes. Only switch devices 1040 (labeled Sw0-Sw3) are shown within each server system 1000 to highlight example connections between the switch devices both within the node and between two nodes. In this example, a first switch device 1040 in a first node is connected to a fourth switch device 1040 in a second node, and the fourth switch device 1040 in the first node is connected to the first switch device 1040 in the second node. Depending on the application, the system 1100 may include one or more additional server nodes, and the connection configuration between the switches within the nodes may vary. Alternatively, the nodes may be connected using a NIC in each node system 1000, such as a pipelined Ethernet connection. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to this multi-node server system configuration.

[0064] FIG. 12 is a simplified block diagram of a portion of a server system according to an example embodiment of the present invention. The illustrated server system 1200 includes a switch device 1210 connected to multiple PU card devices 1220. Similar to the server system 1000, the system 1200 includes four PU devices 1220 (labeled C0-C3) in a card form factor (e.g., PCIe cards). The PU devices 1220 here are configured similarly to the AI ​​accelerator apparatus 102 of FIG. 1B, with eight chiplet devices 1240 stacked on an interposer 1230 in two groups of four interconnected chiplets 1240. Each of the chiplet devices 1240 also includes a connection interface 1242, such as a PCIe interface. Each group of chiplets 1240 is further connected to eight memory devices 1250 (e.g., DRAM). However, the specific number and configuration of the chiplet devices in the AI ​​accelerator apparatus may vary and may include any of the configurations described above.

[0065] The server system 1200 also includes details of various interconnects between chiplet devices 1240 within the same PU device 1220 and between different PU devices 1220. As shown in the expanded views of the first and second PU devices “C1” and “C2,” the switch device 1210 is connected by connection paths 1212 to connection interfaces 1242 of one of the chiplet devices 1240 of the first chiplet group in each of the PU devices 1220. In particular examples, these connection paths 1212 may include printed circuit board (PCB) paths, cables, etc. In both PU devices “C1” and “C2,” another chiplet device 1240 of the first chiplet group is connected by connection paths 1222 to another chiplet device 1240 of the second chiplet group via their connection interfaces 1242. In particular examples, these connection paths 1222 may also include PCB paths, cables, etc.

[0066] 12 further illustrates that the remaining chiplet devices 1240 that are not connected to the switch via connection paths 1212 or between chiplet groups via connection paths 1222 are connected across the PU device 1220 via their connection interfaces 1242 using bridge connection paths 1232. More specifically, each of the remaining two chiplet devices 1240 in each group is connected to a chiplet device 1240 of another chiplet group in the other PU device 1220. The server system 1200 may include additional connections via connection paths 1212 (switch-to-chiplet), connection paths 1222 (group-to-group), and connection paths 1232 (card-to-card) that may be included to connect to other PU devices 1220 or if the configuration of chiplet devices 1240 in the AI ​​accelerator apparatus is different. Those skilled in the art will recognize other variations, modifications, and alternatives.

[0067] The present invention also provides a method and server system configuration that uses transparent bridging to enable communication between multiple central processing unit (CPU) sockets and server nodes. By way of example only, the transparent bridging method and configuration are applied to push-based communications using Ethernet connection capabilities. Those of ordinary skill in the art will recognize other variations, modifications, and alternatives to the application of these transparent bridging methods and configurations.

[0068] Traditional network interface cards (NICs) that enable Ethernet connectivity are difficult to scale with accelerators distributed across nodes in a multi-node accelerator system (e.g., a multi-node GPU accelerator). Network fabric configurations such as remote direct memory access (RDMA) and RDMA over Converged Ethernet (RoCE) can be used with multi-node accelerator systems to transfer data more quickly and efficiently, but these network fabrics may require complex shared address space configurations (e.g., for one-sided communication). Traditional software and hardware implementations of such network fabric configurations may be constrained to operate at the low latency required for specific target applications (e.g., generative AI inference applications). Furthermore, the use of such network fabric configurations can encounter various implementation challenges. For example, peripheral component interconnect express (PCIe) fabric topologies without custom firmware are typically limited to scale within the PCIe switch and the CPU sockets provided for the PCIe lanes, resulting in limited multi-CPU socket peer-to-peer (P2P) connectivity.

[0069] By using transparent bridging, the present invention can enable fast and efficient communication between multi-node accelerator systems and server systems. No explicit software enablement for transparent bridging is required, and delivery of data using transparent bridging can be guaranteed through the application of a specified communication protocol (e.g., an Ethernet-based protocol) for scale-up and scale-out. Further details of this transparent bridging application are described in the following figure.

[0070] 13A is a simplified block diagram illustrating a server system using transparent bridging with synthetic fabric switch connection capabilities according to one embodiment of the present invention. The illustrated server system 1301 may include multiple CPU devices 1310. In a specific example, the multiple CPUs 1310 may be configured as one or more multiprocessors connected to each other using a point-to-point processor interconnect such as an Ultra Path Interconnect (UPI). Here, the CPUs 1310 are configured in pairs, designated "CPU1" and "CPU2."

[0071] Each of the CPU devices 1310 is connected to a switch device 1320. Here, the switches 1320 connected to the dual-core multiprocessor CPUs 1310 are represented as "Switch1" and "Switch2." These switch devices 1320 can be configured for various form factors, such as PCIe (peripheral component interconnect express). In one example, the switches 1320 configured with each multiprocessor are also connected to each other using a composite fabric configuration 1322 (e.g., PCIe fabric, Ethernet fabric, etc.). While the system 1300 is shown using pairs of CPUs 1310 and switches 1320, this coupling configuration can be scaled to larger subsets of CPUs 1310 and switch devices 1320.

[0072] Each switch device 1320 is also connected to one or more processing unit (PU) devices 1330, which may include GPU configurations, TPU configurations, etc. These PU devices 1330 may include the AI ​​accelerator apparatus configurations described above, which may include various form factors, such as PCIe cards. In the case of a PCIe card configuration, the PU devices 1330 may be configured similarly to the AI ​​accelerator apparatuses 101 and 102 of FIGS. 1A and 1B. In FIG. 13A, system 1301 includes four PU devices (labeled PU1-PU4) in a paired configuration connected by bridge connections 1332 (see FIG. 12). Here, a composite fabric configuration 1322 enables communication between PU devices 1330 associated with different switches.

[0073] Each switch device 1320 is also connected to at least one input / output (IO) streaming device (IO CARD) 1340, which may be configured in the same form factor as the PU device (PU CARD) 1330. These IO streaming devices 1340 may be configured to implement transparent bridging to facilitate P2P communication between the PU devices 1330 connected to different multiprocessors. Here, the IO streaming device 1340 connected to "Switch2" of the left multiprocessor is configured to communicate with the IO streaming device 1340 connected to "Switch1" of the right multiprocessor. Additionally, the IO streaming devices 1340 connected to other switch devices 1320 of each multiprocessor may be configured to communicate with the IO streaming devices 1340 of another multiprocessor.

[0074] In one example, these IO streaming devices 1340 can be configured to transparently carry data using transaction layer packets (TLPs) for memory functions (e.g., PCIe MemWr64) and completion packets (e.g., TLP prefixes). The IO streaming devices 1340 can also replicate subsequent parallelism stages (e.g., pipeline parallelism, tensor parallelism, etc.) for the memory base address register (BAR) space used by the CPU 1310. In a specific example, the IO streaming devices 1340 are configured for PCIe-P2P communication using an Ethernet fabric.

[0075] In one example, system 1301 may be configured such that one IO streaming device configured in one of the multiprocessors and connected to one of the switches in the composite fabrication configuration manages communications with IO streaming devices configured in other multiprocessors. Each multiprocessor with a CPU 1310 and its associated switch 1320, PU device 1330, and IO streaming device 1340 may be configured as a separate server node in a multi-node server system. Examples of multi-node server systems using transparent bridging are shown in Figures 13B and 13C.

[0076] FIG. 13B is a simplified block diagram of a server system using transparent bridging for intra-node and inter-node connection capabilities according to one embodiment of the present invention. The illustrated system 1302 is similar to the system 1301 of FIG. 13A, except that each multiprocessor is configured within a separate server node device 1350, and an IO streaming device 1340 is configured for P2P communication between the server nodes 1350 (i.e., inter-node P2P communication). Also, the switch devices 1320 connected to each multiprocessor are not connected to each other using a synthetic fabric configuration. Instead, each switch device 1320 under each multiprocessor further includes an intra-node IO streaming device 1342 configured for P2P communication between the switches 1320 within the server nodes 1350 (i.e., intra-node P2P communication) instead of using a CPU-UPI interface.

[0077] In this example, an intra-node IO streaming device 1342 enables P2P communication between PU devices (PU CARDs) 1330 connected to two switches 1320 in a server node 1350 connected to two different CPU sockets. These intra-node devices 1342 can be configured for low latency and high bandwidth communication. In a server rack configuration, this intra-node connectivity capability can also use a loopback method instead of going to the top of the rack for a low latency interface from a top-of-rack (ToR) switch, etc.

[0078] 13C is a simplified block diagram of a server system using transparent bridging for intra-node socket connection capabilities and inter-node connection capabilities according to one example of the present invention. The illustrated system 1303 is similar to system 1302 of FIG. 13B, except that there is no switch device 1320 in each server node 1352. Instead, each CPU 1310 (configured in a separate CPU socket) is directly connected to a PU device (PU CARD) 1330, an inter-node IO streaming device (IO CARD) 1340, and an intra-node IO streaming device (INTRA NODE IO CARD) 1342.

[0079] In this example, the intra-node IO streaming device 1342 enables P2P communication between PU devices (PU CARDs) 1330 connected to different CPU sockets instead of using the CPU-UPI interface. Similar to the configuration of Figure 13B, this server configuration can also use a loopback method. Those skilled in the art will recognize other variations, modifications, and options for these server configurations that use transparent bridging.

[0080] In one example, the present invention provides an AI system using transparent bridging. The system includes a first server device having at least a first host CPU connected to a first PCIe switch, which is also connected to multiple first AI accelerator devices / apparatuses. The system also includes an input network interface, an intra-network interface, and an output network interface. These interfaces can be configured to enable communication between the first server device and a second server device. The second server device also includes at least a second PCIe switch connected to a second host CPU, multiple second AI accelerator devices / apparatuses, an output network interface, and an intra-network interface.

[0081] In one example, the input networking interface is configured to receive information using a task ID from a second server device configured with an Ethernet protocol. Each of the first and second host CPUs may be configured to operate a compiler in parallel with and independently of operating the intra-network interface to transfer information using the task ID. Each of the first and second host CPUs may also be configured to be asynchronous with the input network interface, the intra-network interface, or the output network interface. The second server device may be configured to wait (e.g., remain idle or in a previous operating state) or process instructions related to one or more task IDs. Alternatively, the second server device may be configured to process a most recently received set of data until receiving a next set of data (e.g., from another accelerator device).

[0082] In one example, the intra-network interface is configured to communicate from one or more of the plurality of first AI accelerator devices to one or more of the plurality of second AI accelerator devices, such that the communication uses one or more task IDs without communicating directly or indirectly from the second host central processing unit. The intra-network interface may also facilitate communication between one or more of the plurality of first AI accelerator devices and one or more of the plurality of second AI accelerator devices. Furthermore, communication from the intra-network interface may be provided without including instructions from either the first host CPU or the second host CPU. This communication may also be transparent to either the plurality of first AI accelerator devices or the second AI accelerator device.

[0083] In one example, the output network interface is configured to send information using one or more task IDs to a second server device using an Ethernet protocol. The output network interface and the input network interface may each include a PCIe endpoint device, a first input / output (IO) bridge device, a second IO bridge device, and an engine configured to communicate using the communication protocol. In one example, each of the input network interface and the output network interface may be configured as an IO network interface, and the configuration and method for communication may also be applied from the second server device to the first server device.

[0084] The system can be configured as a multi-node AI server system and can include an inter-node network interface that can be configured similarly to an intra-network interface but applied to communication between server nodes. Each system can also be configured so that multiple AI accelerator devices are connected to sockets of a host CPU, and the inter-node network interface can facilitate communication between the CPU sockets. These and other examples are described in the following figures.

[0085] 14 is a simplified block diagram of a multi-node server system using a transparent bridge for scale-up and scale-out according to an example of the present invention. The illustrated system 1400 includes a switch device (Switch) 1410 (e.g., a ToR Ethernet switch) connected to multiple server node devices (Server) 1420 (numbered 1 through N), each connected to a first IO streaming device (I / O-1 Card) 1430 and a second IO streaming device (I / O-2 Card) 1432. In one example, the first IO streaming device 1430 may be configured to receive data from the switch 1410, and the second IO streaming device 1432 may be configured to send data to the switch 1410. Alternatively, both IO streaming devices 1430 and 1432 may be configured to receive and send data. Depending on bandwidth and latency requirements, each IO streaming device may be configured as a transmitter, a receiver, or a transceiver.

[0086] In one example, these IO streaming devices 1430, 1432 are configured to implement transparent bridging to scale up and scale out the network of server nodes 1420 in the multi-node server system 1400. The system 1400 can be configured as a lossless network or a lossy network. Also, each of the nodes 1420 can be configured similarly to the server node configurations described above, in which case the IO streaming devices 1430, 1432 can be configured to communicate between switches within the node 1420 similarly using transparent bridging. Those of ordinary skill in the art will recognize other variations, modifications, and options for scaling up and out in a multi-node server system.

[0087] FIG. 15 is a simplified block diagram of an IO streaming device according to one example of the present invention. The illustrated device 1500 includes an endpoint (EP) device 1510 connected to a communications engine device 1530 via a first bridge device 1520 and a second bridge device 1522, which may be configured as a transmitter bridge path and a receiver bridge path, respectively. The first bridge device 1520 and the second bridge device 1522 are configured to communicate between the EP device 1510 and the communications engine device 1530 using transparent bridging. The EP device 1510 is configured for communication using a specified interface standard, such as PCIe. In one example, the EP device 1510 may be connected to another EP device (e.g., another IO streaming device, an AI accelerator PU, etc.), a switch, or a root complex. Depending on the application, the EP device 1510 and communications engine 1530 can be configured for various interconnect technologies (eg, PCIe, Ethernet, etc.).

[0088] The communication engine device 1530 is configured to communicate with other devices (e.g., in a server system) using one or more communication protocols, such as a transmission control protocol / internet protocol (TCP / IP), a die-to-die (D2D) interface communication protocol, an Ethernet communication protocol, or a layer 2 (L2) communication protocol. In one example, the engine device 1530 may include a TCP / IP offload engine (TOE) configured for a lossy network. The communication protocol may include a guaranteed delivery scheme (i.e., no packet loss), which may include a retry buffer and a congestion control scheme using pause packets. And, the D2D interface communication protocol may include D2D logic configured for a lossless network with porting support (e.g., from an application-specific integrated circuit [ASIC] to a field-programmable gate array [FPGA]).

[0089] The first bridge device 1520 is configured as a bridge from the EP device 1510 to the communications engine device 1530, while the second bridge device 1522 is configured as a bridge from the communications engine device 1530 to the EP device 1510. In one example, the first bridge device 1520 may be configured to manage a network communication flow control system for IO streaming devices, such as in a PCIe credit-based flow control system. The first bridge device 1520 may facilitate memory write functionality by sending data / completion packets to the EP device 1510. These packets may include transaction layer packets (TLPs), data link layer packets (DLLPs), etc. In one example, the completion packets follow the in-order flow of data through the first bridge device 1520. The first bridge device 1520 may stream data and control signals to the communications engine device 1530 using a streaming interface, such as an advanced extensible interface (AXI). Similarly, the second bridge device 1522 may receive control signals and data streamed from the communications engine device 1530 using such a streaming interface. Additionally, the second bridge device 1522 may send memory write requests to the EP device 1510 using similar data packets. Using this configuration, the IO streaming device 1500 may connect an endpoint to another endpoint, an endpoint to a root complex, or a root complex to another root complex.

[0090] In a specific example, IO streaming device 1500 includes an FPGA device configured for PCIe communications supporting TLP prefixes and steering tags. EP device 1510 can be a PCIe-EP device, and communication engine 1530 is configured for Ethernet connectivity via a TOR switch. In this case, first and second bridge devices 1520 and 1522 are configured as PCIe-to-Ethernet bridges. Those skilled in the art will recognize other variations, modifications, and alternatives to this IO streaming device configuration.

[0091] 16 is a simplified block diagram of an IO streaming data format according to one embodiment of the present invention. The illustrated data format 1600 may include fields for Preamble 1610, Start Frame Delimiter (SFD) 1620, Destination 1630, Source 1640, Type / Length 1650, Data 1660, and Frame Check Sequence (FCS) 1670. The Data field may be configured to include subfields to enable transparent bridging using data packets, such as Destination Card Number 1662, Packet Header Prefix 1664, and Packet Data 1666. The source 1640 and destination 1630 may be in the same domain (e.g., PCIe domain) or different domains.

[0092] In a specific example, data format 1600 is structured as an Ethernet frame, and data field 1660 incorporates PCIe-TLP information (e.g., TLP header prefix and TLP data) to enable transparent bridging between PCIe switches. Those skilled in the art will recognize other variations, modifications, and alternatives to the data format used for transparent bridging.

[0093] According to one example, the present invention provides a technique for configuration using data transfers across gang boundaries that are synchronized using SW-assigned task IDs. In one example, the technique configures a sender that does not need to explicitly know that the data transfer has reached the receiver (i.e., the source device assumes that the protocol used to transmit the data guarantees delivery of the data to the destination device). In one example, the NoC channel is assumed to be lossless, and any external links are assumed to have consistency / retry capabilities. Therefore, the transmission of data from the source signifies the completion of the task from the sender's POV. In one example, the receiver does not need to know the identity of the sender that transferred the data or when the data was transferred. In one example, the receiver only needs to prevent reading and / or using buffer assignments for such data until the data arrives. As a result, the receiver only needs a barrier to stop or obey downstream execution contingent on the data transfer to the receiver. In one example, both the sender and receiver are agnostic to the other, and the only entity aware of their relationship is a compiler configured on the host central processing unit. Thus, the SW-assigned task ID serves as a means of establishing a producer-consumer relationship for long-distance data transfer. This configuration can be extended to a multi-host configuration, such as a multi-node server system, where the compiler can run on multiple host processing units and configure multiple senders and receivers.

[0094] In one example, the present invention provides a system and method for PCIe peer-to-peer (P2P) writes between different sets of accelerators using transparent bridging via an input / output (IO) streaming device, which may be configured as a transparent NIC device. In this example, the transparent NIC is shown as exposing the BARs (Base Address Registers) of neighboring cards and representing them as mirrored BARs. However, the present system and method is not limited to communication between only two neighboring cards. In one example, the present system and method may be configured to provide communication between nodes (e.g., servers) for tensor parallelism, pipeline parallelism, etc. In one example, the present system and method provides a distribution of the number of mirrored BARs exposed per card determined by the bandwidth and latency requirements of the workload. Such requirements differ depending on whether the workload includes tensor-level parallelism or pipeline parallelism. To meet the bandwidth and latency requirements for tensor-level parallelism, additional IO streaming devices per node can be integrated using the present technology for scalability and high-performance AI workloads.

[0095] FIG. 17A is a simplified block diagram illustrating a method 1701 for transmitting data using transparent bridging according to one example of the present invention. As shown in flow diagram 1720, the method may include step 1722 of processing a transmit operation (e.g., a memory write) using a receive first-in, first-out (Rx FIFO) data structure. This receive FIFO data structure may be implemented in a memory device (e.g., static random access memory [SRAM], dynamic random access memory [DRAM], high-bandwidth memory [HBM], etc.) within the IO streaming device to receive data packets from other devices connected to the same switch or CPU socket. This memory device may be configured or connected within the EP device or bridge device in the transmit data path (see FIG. 15) from the EP device to the communications engine. In a specific example, the FIFO structure may be configured for the aforementioned TLP used to transfer data between PCIe devices.

[0096] In step 1724 (BAR2Card Bridge Map Addr Suffix Extract), the method includes using bridging logic to map a mirrored BAR space (Mirrored Card Bar) 1712 to a device ID (Card ID) 1714 (e.g., a card ID of a PCIe card) using a first lookup table (LUT) 1710. This step may further include extracting an address suffix. Then, in step 1726 (Packet Creation), the method includes creating a data packet that can be transmitted using a communication engine (see FIG. 15) of the IO streaming device. This step may also include data field generation, which may include steps such as integrating TLP information into an Ethernet frame, as shown in FIG. 16. Using the communication engine, these data packets can be transmitted to devices in different domains (e.g., another switch, CPU socket, node, etc.) of the multi-node server system.

[0097] FIG. 17B is a simplified block diagram illustrating a method 1702 for receiving data using transparent bridging according to one embodiment of the present invention. As shown in method 1702, the receive data path operations are reversed compared to the transmit data path operations. In step 1742 (Packet Parser), the method includes parsing a packet received by the IO streaming device in the communications engine. Next, in step 1744 (Card2BAR Bridging Map Addr Suffix Addition), the method includes using bridging logic to map a device ID 1732 (e.g., a card ID) to a real card space 1734 (Real Card BAR) using a second LUT 1730. This step may further include address suffix addition and packet formation. Then, in step 1746, the method includes processing the receive operation in a transmit FIFO (Tx FIFO) data structure, which may be configured within or connected to the EP device of the bridge device in the receive data path (see FIG. 15) from the communications engine to the EP device. FIFO structures can be configured for TLP as described with respect to Figure 17A. Those skilled in the art will recognize other variations, modifications, and alternatives to these datapath methods.

[0098] 18A is a simplified block flow diagram illustrating a method for transmitting data in a server system using an accelerator device according to one example of the present invention. The illustrated method 1801 represents a P2P data transfer between a first accelerator (Accelerator-1 Node-1) 1810 in a first server system node and a second accelerator device (Accelerator-2 Node-1) 1812 in the same server system node. Each of these accelerator devices 1810, 1812 may be configured similarly to the AI ​​accelerator device, PU device, etc. described above. The data transfer includes multiple memory writes (Memory Write) from the first accelerator device 1810 to the second accelerator device 1812. The final Memory Write also includes a prefix (+Prefix) that causes the second accelerator device 1812 to begin processing the transferred data. In a specific example, P2P data flow is performed between PCIe connections (eg, via a shared PCIe switch using the PCIe fabric default mode).

[0099] In one example, the P2P flow may follow a push-based mechanism in which a source accelerator device (e.g., a first accelerator device 1810) writes to a preprogrammed BAR address of a destination accelerator device (e.g., a second accelerator device 1812). As described above, each accelerator device is configured to process an operation graph representing a neural network model workload. In one example, the graph of the destination accelerator device (see FIG. 9 ) provided by the compiler assumes that data from the source accelerator device is written to a predefined location. In a specific example, when the destination accelerator device receives a task ID flag through a prefix (e.g., a PCIe-TLP prefix) or an information unit (e.g., a PCIe TLP Dword), the destination accelerator device compares the received task ID with a preprogrammed task ID and proceeds with execution if the task IDs match. The method may include a double-buffering process in which data movement is scheduled between the source accelerator device and empty memory so that the destination accelerator device does not wait for the data to be operated on.

[0100] In one example, the present invention provides a method for transferring data between a pair of AI accelerator devices within a server node (i.e., intra-node data transfer), where the node includes at least a pair of processing units, each connected to a PCIe switch, which is connected to multiple AI accelerator devices and an intra-node source network interface card (NIC) device.

[0101] 18B is a simplified block flow diagram illustrating a method for transmitting data in a server system having accelerator devices using transparent bridging according to one example of the present invention. The illustrated method 1802 illustrates a data transfer from an accelerator device (Accelerator-1 Node-1) 1810 of a first server system node to an accelerator device (Accelerator-1 Node-2) 1820 of a second server system node using an IO streaming device (I / O Streaming Card Node-1) 1830 of the first node and an IO streaming device (I / O Streaming Card Node-2) 1832 of the second node. This data transfer process includes transparent pass-through of memory writes and prefixes (e.g., PCIe memory writes and TLP prefixes to Ethernet).

[0102] Here, method 1802 involves multiple memory writes and prefixes from a first accelerator device 1810 to an IO streaming device 1830 within the same first server node. In particular examples, these memory writes are performed across PCIe connections (e.g., a shared PCIe switch or a CPU socket). After each memory write, the IO streaming device 1830 of the first node passes the memory write and prefix to an IO streaming device 1832 of a second node. In particular examples, the memory writes and prefixes are configured as Ethernet frames that are transferred between IO streaming devices of different nodes. Similarly, the IO streaming device 1832 of the second node forwards the memory write and prefix to the accelerator device of the second node. In particular examples, these memory writes are performed across PCIe connections as well.

[0103] In one example, the present invention provides a method for transferring data between a pair of AI accelerator devices (i.e., inter-node data transfer) between at least one pair of nodes, including a first node and a second node, where each node includes at least one pair of processing units, each processing unit connected to a PCIe switch, and the PCIe switch is connected to multiple AI accelerator devices and inter-node source network interface card (NIC) devices.

[0104] FIG. 18C is a simplified block flow diagram illustrating a method for transmitting data in a server system with an accelerator device using buffered transparent bridging according to one embodiment of the present invention. The illustrated method 1803 is similar to method 1802 of FIG. 18B, except that it adds buffering of data transfers using a store-and-forward approach to pack more data packets and / or prefixes to achieve higher bandwidth. In this example, packets are merged in buffer 1840 within IO streaming device 1830 of a first node. The merged packets are then unpacked in buffer 1842 within IO streaming device 1832 of a second node. The IO streaming device can be configured to manage low latency and high bandwidth flows when using buffered transparent bridging. In a specific example, PCIe-TLPs and prefixes are buffered to achieve higher bandwidth over Ethernet IO.

[0105] 18D is a simplified block diagram illustrating a method for transmitting data in a server system having accelerator devices, each including a datapath controller and one or more subsystems, according to one example of the present invention. The illustrated method 1804 represents a data communication flow from a datapath controller 1850 of a first accelerator device (Accelerator-1), through a subsystem 1852 of the first accelerator device, through a subsystem 1862 of a second accelerator device, to a datapath controller 1860 of the second accelerator device. In one example, the first accelerator device 1850 can instruct the subsystem 1852 (e.g., a PCIe subsystem) to perform multiple memory writes from its local memory to or reads from its local memory (e.g., by performing direct memory access [DMA]). This process may involve informing the subsystem of the task ID or completion packet field (+Completion Packet / Task ID) that is sent as part of the last packet.

[0106] Here, the first accelerator device is the source device and the second accelerator device is the destination device. Data flow between the first accelerator device subsystem 1852 and the second accelerator device subsystem 1862 can follow the same P2P process as described above or a transparent bridging process with multiple memory writes and prefixes (+Prefix). The task ID sent from the source device is pre-programmed by the compiler and should match the expected value of the destination device.

[0107] In one example, when the second accelerator device receives the packet as subsystem 1862, it uses a memory write and transfers the data to its local memory using data path controller 1860. The second accelerator device is preprogrammed with task ID information by a compiler to match the received completion packet and task ID. Upon receiving the matching completion packet and task ID, the second accelerator device can execute its associated operation graph (Execution upon receiving matching completion packet and task-ID). Those skilled in the art will recognize other variations, modifications, and alternatives to these data transfer methods.

[0108] While specific embodiments have been fully described above, various modifications, alternative configurations, and equivalents may be used. As an example, the AI ​​accelerator device and chiplet device may include any combination of the elements described above, as well as others outside the scope of this specification. Therefore, the above description and examples should not be construed as limiting the scope of the present invention, which is defined by the claims.< / s>

Claims

1. An AI system, a first server device, the first server system including: a first host central processing unit; a first PCIe switch connected to the first host central processing unit; a plurality of first AI accelerator devices connected to the first PCIe switch; an input network interface configured to receive information using a task ID from a second server device configured with an Ethernet protocol; an intra-network interface configured to communicate from one or more of the plurality of first AI accelerator devices to one or more of the plurality of second AI accelerator devices, the communication using one or more task IDs without communicating directly or indirectly from a second host central processing unit; an output network interface configured to transmit information using one or more task IDs to the second server device using the Ethernet protocol; a second PCIe switch configured to connect the second host central processing unit to the plurality of second AI accelerator devices, the output network interface, and the intra-network interface.

2. The system of claim 1 , wherein the intra-network interface facilitates communication between one or more of the plurality of first AI accelerator devices and one or more of the plurality of second AI accelerator devices.

3. 2. The system of claim 1, wherein each of the first host central processing unit and the second host central processing unit is configured to operate a compiler concurrently and independently of operating the intra-network interface to transfer information using the task ID.

4. The system of claim 1 , wherein the second server device is configured to wait for or process instructions related to the one or more task IDs.

5. The system of claim 1 , wherein each of the first host central processing unit and the second host central processing unit is not synchronized with the input network interface, the intra-network interface, or the output network interface.

6. 2. The system of claim 1, wherein the communication from the intra-network interface is provided without instructions from either the first host central processing unit or the second host central processing unit.

7. The system of claim 1 , wherein the communication is transparent to any of the plurality of first AI accelerator devices and the second AI accelerator device.

8. The output network interface and the input network interface each include: a PCIe endpoint device; a first IO bridge device; a second IO bridge device; and an engine configured to communicate in at least one of a TCP / IP communication protocol, a die-to-die interface communication protocol, or a Layer 2 communication protocol having a guaranteed delivery scheme.

Images