Node with a bridge network interface device for an AI accelerator system

Abstract

AI system comprehensive: a first server device comprehensive: a first host central processing unit; a first PCIe switch coupled with the first host central processing unit; a plurality of first AI accelerator devices coupled with the first PCIe switch; an input network interface configured to receive information using a task ID from a second server device configured with an Ethernet protocol; an intranetwork value interface configured to communicate from one or more of the plurality of first AI accelerator devices to one or more of a plurality of second AI accelerator devices, such that the communication uses one or more task IDs without communicating directly or indirectly from a second host central processing unit; an output network interface configured to transmit information to the second server device using one or more task IDs via the Ethernet protocol; and a second PCIe switch configured to couple the second host central processing unit with the majority of second AI accelerator devices, the output network interface, and the intra-network value interface.

Description

BACKGROUND OF THE INVENTION

[0001] Traditional network interface cards (NICs) that provide Ethernet connectivity face significant challenges when scaling across distributed AI accelerators. These challenges are particularly evident in environments involving multiple nodes. One solution, RoCEv2 (commonly known as "RDMA over Converged Ethernet"), combined with RDMA-IB ("InfiniBand") fabric, can facilitate communication between GPU accelerators with multiple nodes.

[0002] However, RDMA over Converged Ethernet in combination with InfiniBand has several limitations. Such a solution often requires a complex shared address space to support one-way communication, which complicates deployment and management. Furthermore, both software and hardware fabric solutions reach their limits when attempting to meet the low-latency requirements of generative AI (GenAI) inference. This limitation prevents them from fully realizing the performance potential of modern accelerators.

[0003] Other conventional techniques involve the use of PCIe (Peripheral Component Interface Express) topologies. However, PCIe fabric topologies have limitations regarding scalability. While PCIe supports communication within a switch, it is limited by the number of PCIe lanes provided by CPU sockets and PCIe switches. Consequently, PCIe makes it difficult to implement efficient peer-to-peer (P2P) connectivity between multiple CPU sockets across nodes.

[0004] Certain PCIe switch vendors offer synthetic fabric models that enable cross-switch x16 link communication through the use of custom firmware. Unfortunately, such synthetic fabric models remain highly specialized and not yet widely adopted.

[0005] The above demonstrates that techniques for scaling across distributed accelerators are highly desirable. BRIEF SUMMARY OF THE INVENTION

[0006] The present invention relates generally to integrated circuits (ICs) and artificial intelligence (AI) systems. In particular, the present invention relates to methods and device structures for accelerating computational loads of neural network models (e.g., transformer models, convolutional neural network models, etc.). These methods and structures can be used in machine / deep learning applications such as natural language processing (NLP), computer vision (CV), and the like. The invention has been applied, by way of example, to AI accelerator devices and chiplet devices configured in a PCIe card.

[0007] In one example, the techniques presented here include a transparent bridge configuration that utilizes any data transmission crossing a channel boundary, synchronized using a software-assigned task ID. In another example, the techniques configure a sender that does not need to explicitly know (i.e., the source device assumes that the protocol used to send the data guarantees delivery to a destination device) that a data transmission has reached a device. In yet another example, it is assumed that NoC channels are lossless and that all external connections have integrity / retry capabilities. Accordingly, sending data from the sender's perspective signifies the completion of the task. In yet another example, a receiver does not need to know the identity of the sender who transmitted the data, nor when the data was transmitted.In one example, the receiver only needs to prevent reading and / or using a buffer label for such data until the data arrives. Therefore, the receiver only requires a barrier that delays downstream execution or makes it dependent on data transmission to the receiver. In another example, both the sender and receiver are independent of each other; the only instance aware of their relationship is the compiler configured on a host central processing unit. Thus, the task ID assigned by the software serves as the means to establish a producer-consumer relationship for long-distance data transmission. This configuration can be extended to a multi-host configuration, for example, in a multi-node server system where the compiler runs on multiple host processors and can configure multiple senders and receivers.

[0008] In one example, the present invention provides a system for PCIe peer-to-peer (P2P) write operations between different sets of accelerators using a transparent bridge via input / output (IO) streaming devices that can be configured as transparent network interface card (NIC) devices. In this example, the IO streaming device is shown to expose the base address registers (BARs) of adjacent cards and to represent them as mirrored BARs. However, the present system is not limited to communication between only two adjacent cards. In one example, the present system can be configured to enable communication between nodes (e.g., servers) for tensor parallelism, pipeline parallelism, and the like.In one example, the present system envisions a distribution of a number of mirrored BARs exposed per map, determined by the bandwidth and latency requirements of a workload. These requirements vary depending on whether the workload involves tensor parallelism or pipeline parallelism. To meet the bandwidth and latency requirements for tensor parallelism, additional I / O streaming devices can be integrated per node using the techniques presented here for scalability and high-performance AI workloads.

[0009] By using a transparent bridge, the present invention enables fast and efficient communication across multi-node accelerator systems and server systems. Explicit software activation for the transparent bridge is not required, and data transfer using the transparent bridge can be ensured by applying specific communication protocols for scale-up and scale-out. BRIEF DESCRIPTION OF THE DRAWINGS The Fig. Figures 1A-1B are simplified block diagrams illustrating AI accelerator devices according to examples of the present invention. The Fig. Figures 2A-2B are simplified block diagrams illustrating 16-disk chiplet devices according to examples of the present invention. The Fig. Figures 3A-3B are simplified block diagrams illustrating disk devices according to examples of the present invention. Fig. Figure 4 is a simplified block diagram illustrating an in-memory compute module (IMC) according to an example of the present invention. Fig. Figure 5A is a simplified block flow diagram illustrating numerical formats of the data processed in a disk device according to an example of the present invention. Fig. 5B is a simplified diagram illustrating exemplary numerical formats. Fig. 6A is a simplified block diagram of a transformer architecture. Fig. Figure 6B is a simplified block diagram illustrating an autoregressive token processing procedure in a transformer decoder. Fig. Figure 7 is a simplified block flow diagram illustrating a figure process between a transformer and an AI acceleration device according to an example of the present invention. Fig. 8A and Fig. Figure 8B are simplified diagrams illustrating transformer computing devices according to examples of the present invention. Fig. Figure 9 is a simplified block diagram illustrating an AI accelerator hardware-software system according to an example of the present invention. Fig. Figure 10 is a simplified block diagram illustrating a server system according to an example of the present invention. Fig. Figure 11 is a simplified block diagram illustrating a multi-node server system according to an example of the present invention. Fig. Figure 12 is a simplified block diagram illustrating a portion of a server system according to an example of the present invention. Fig. Figure 13A is a simplified block diagram representing a server system that uses a transparent bridge connection with synthetic fabric-switch connectivity according to an example of the present invention. Fig. Figure 13B is a simplified block diagram representing a server system that uses a transparent bridge connection for switch connectivity within an intranode and connectivity between nodes according to an example of the present invention. Fig. Figure 13C is a simplified block diagram illustrating a server system according to an example of the present invention, which uses a transparent bridge connection for intranode socket connectivity and connectivity between nodes. Fig. Figure 14 is a simplified block diagram illustrating a multi-node server system according to an example of the present invention, which uses a transparent bridge connection for scaling upwards and outwards. Fig. Figure 15 is a simplified block diagram illustrating an input / output (IO) streaming device according to an example of the present invention. Fig. Figure 16 is a simplified block diagram illustrating an IO streaming data format according to an example of the present invention. Fig. 17A and Fig. Figure 17B are simplified block diagrams, each illustrating a method for sending and receiving data using a transparent bridge connection according to an example of the present invention. Fig. Figure 18A is a simplified block flow diagram illustrating a method for sending data in a server system with accelerator devices according to an example of the present invention. Fig. Figure 18B is a simplified block flow diagram illustrating a method for sending data in a server system with accelerator devices using a transparent bridge according to an example of the present invention. Fig. Figure 18C is a simplified block flow diagram illustrating a method for sending data in a server system with accelerator devices using a buffered transparent bridge according to an example of the present invention. Fig. Figure 18D is a simplified block diagram illustrating a method for transferring data in a server system with accelerator devices, each comprising a data path controller and one or more subsystems, according to an example of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present invention relates generally to integrated circuit (IC) devices and artificial intelligence (AI) systems. In particular, the present invention relates to methods and device structures for accelerating computational loads of neural network models (e.g., transformer models, convolutional neural network models, etc.). These methods and structures can be used in machine / deep learning applications such as image recognition and processing, as well as in other applications. The invention has been applied, by way of example, to AI accelerator devices and chiplet devices configured for performing high-throughput convolution operations.

[0011] Currently, most NLP models are based on the transformer model, such as the Bidirectional Encoder Representations from Transformer (BERT) models, the BERT Large model, and generative pretrained transformer (GPT) models like GPT-2 and GPT-3, etc. However, these transformers have very high computational and memory requirements. According to one example, the present invention provides a device that uses chiplet devices configured to accelerate transformer computations for AI applications. Examples of the AI ​​accelerator device are described in the Fig. 1A and Fig. 1B is shown.

[0012] Fig. Figure 1A shows a simplified AI accelerator device 101 with two chiplet devices 110. As shown, the chiplet devices 110 are coupled to each other by one or more die-to-die (D2D) connections 120. Each chiplet device 110 is also coupled to a memory interface 130 (e.g., static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous dynamic RAM (SDRAM), or the like). The device 101 also includes a substrate element 140, which provides mechanical support to the chiplet devices 110 configured on a surface area of ​​the substrate element 140. The substrate may include interposers, such as a silicon interposer, a glass interposer, an organic interposer, or the like.The chiplets can be coupled with one or more interposers, which can be configured to enable communication between the chiplets and other components (e.g., as a bridge or conduit that allows the passage of electrical signals between internal and external elements).

[0013] Fig. Figure 1B shows a simplified AI accelerator device 102 with eight chiplet devices 110 configured in two groups of four chiplets on the substrate element 140. Here, each chiplet device 110 within a group is coupled to other chiplet devices via one or more D2D connections 120. The device 102 also shows a DRAM memory interface 130 coupled to each of the chiplet devices 110. The DRAM memory interface 130 can be coupled to one or more memory modules, represented by the block "Mem".

[0014] As shown, the AI ​​accelerator devices 101 and 102 are implemented in the form of PCIe (Peripheral Component Interconnect Express) cards, but the AI ​​accelerator device can also be configured in other form factors. These PCIe card form factors can be configured in a variety of dimensions (e.g., full height, full length (FHFL); half height, half length (HHHL), etc.) and mechanical sizes (e.g., 1×, 2×, 4×, 16×, etc.). In one example, one or more substrate elements 140, each having one or more chiplets, are coupled to a PCIe card. Experts in this field will recognize other variations, modifications, and alternatives to these elements and configurations of the AI ​​accelerator device.

[0015] Implementations of the AI ​​accelerator device can employ several techniques to improve performance (e.g., computational efficiency) in various AI applications. The AI ​​accelerator device can incorporate digital in-memory computing (DIMC) to integrate computational functions and memory structure. Algorithms for mapping, numerical analysis, and sparse population awareness can be optimized within the computational structure. Furthermore, the use of chiplets and interconnects configured on organic interposers can provide modularity and scalability.

[0016] According to one example, the present invention implements chiplets with in-memory computation (IMC) functionality that can be used to accelerate the computations required by Transformer workloads. The computations for training these models can include the execution of a scaled dot-product attention function to determine a probability distribution associated with a desired outcome in a specific AI application. In the case of training NLP models, the desired outcome can include predicting subsequent words, determining the contextual meaning of words, translating into another language, and so on.

[0017] The chiplet architecture can comprise multiple disk devices (or disks) controlled by a central processing unit (CPU) to perform transformer calculations in parallel. Each disk is a modular IC device capable of handling a portion of these calculations. The multiple disks can be divided into tiles / groups (i.e., subsets) of one or more disks, with a CPU coupled to each disk within the tile. This tile CPU can be configured to perform transformer calculations in parallel across each disk within the tile. A global CPU can be coupled to each of these tile CPUs and configured to perform transformer calculations in parallel across all disks in one or more chiplets using the tile CPUs. Further details regarding the chiplets are provided with reference to the Fig. 2A-5B explains, while Transformer refers to the Fig. 6-9 will be explained.

[0018] Fig. Figure 2A is a simplified block diagram illustrating an example configuration of a 16-disk chiplet device 201. In this case, the chiplet 201 comprises four tile devices 210, each of which includes four disk devices 220, a CPU 221, and a hardware dispatch device (HW DS) 222. In a specific example, these tiles 210 are arranged symmetrically. As explained earlier, the CPU 221 of a tile 210 can coordinate the operations performed by all disks within the tile. The HW DS 222 is coupled to the CPU 221 and can be configured to coordinate the control of the disks 220 in the tile 210 (e.g., to determine which disk in the tile processes a target portion of the transformer computations). In a specific example, the CPU 221 could be a Reduced-Instruction-Set Computer (RISC) CPU or something similar.In addition, the CPU 221 can be coupled with a dispatch engine configured to coordinate the control of the CPU 221 (e.g., to determine which portions of the transformer calculations are processed by each CPU).

[0019] The CPUs 221 of each tile 210 can be coupled to a global CPU via a global CPU interface 230 (e.g., buses, connectors, sockets, etc.). This global CPU can be configured to coordinate the processing of all chiplet devices in an AI accelerator device, such as devices 101 and 102 in the Fig. 1A and Fig. 1B. In one example, a global CPU can use the HW DS 222 of each tile to instruct each associated CPU 221 to perform different parts of the transformer calculations across the disks in the tile. Furthermore, the global CPU can be a RISC processor or similar. The chiplet 201 also includes D2D links 240 and a memory interface 250, both of which are coupled to each of the CPUs 221 in each of the tiles. In one example, the D2D links can be configured with single-ended signaling. The memory interface 250 can include one or more memory buses coupled to one or more memory devices (e.g., DRAM, SRAM, SDRAM, or the like).

[0020] Furthermore, the chiplet 201 includes a PCIe interface / PCIe bus 260 that is coupled to each of the CPUs 221 in each of the tiles. The PCIe interface 260 can be configured to communicate with a server or other communication system. In the case of multiple chiplet devices, a main bus device is coupled to the PCIe bus 260 of each chiplet device via a master chiplet device (e.g., a main bus device that is also coupled to the master chiplet device). This master chiplet device is coupled to each other chiplet device via at least the D2D connections 240. The master chiplet device and the main bus device can be configured to overlay a substrate element (e.g., the same substrate as the chiplets or a separate substrate). A device integrating one or more chiplets can also be coupled to a power source (e.g., a power supply).(configured on the chip, configured in a system, or externally coupled) and can be configured and operated to communicate with the main bus device, a server, a network switch, or a host system. The server device can also be one of multiple server devices configured for a server farm within a data center or other similar configuration.

[0021] In a specific example, an AI accelerator device configured for GPT-3 can contain eight chiplets (similar to device 102 in Fig. 1B). The chiplets can be configured with D2D 16x16 Gb / s interconnects, 32-bit LPDDR5 6.4 Gb / s memory modules, and a 16-lane PCIe Gen 5 PHY NRZ 32 Gb / s / lane interface. LPDDR5 (16 × 16 GB) can provide the required capacity, bandwidth, and low power consumption for large-scale NLP models such as quantized GPT-3. Of course, other variations, modifications, and alternatives are also possible.

[0022] Fig. Figure 2B is a simplified block diagram illustrating an example configuration of a 16-disk chiplet device 202. Similar to chiplet 201, chiplet 202 comprises four groups 210 (or tiles), each of which includes four disk devices 220 and one CPU 221. As shown, the CPU 221 of each group / tile 210 is coupled to each of the disks 220 and to each other CPU 221 of the other groups / tiles 210. In one example, the groups / tiles serve as neural cores and the disks as computation cores. With this multi-core configuration, the chiplet device can be configured to perform multiple computations in parallel. The CPUs 221 are also coupled to a global CPU interface 230, D2D connections 240, a memory interface 250, and a PCIe interface 260. As for Fig. As described in 2A, the global CPU interface 230 is connected to a global CPU that controls all CPUs 221 of each group 210.

[0023] Fig. Figure 3A is a simplified block diagram illustrating an example of a disk device 301 of a chiplet. For the example of a 16-disk chiplet, the disk device 301 comprises a compute core 310 with four compute paths 312, each of which includes an input device (IB) 320, a digital in-memory computation (DIMC) device 330, an output buffer (OB) device 340, and a single-instruction multiple data (SIMD) device 350. Each of these paths 312 is coupled to a disk crossbar / controller 360, which is controlled by the tile CPU to coordinate the computations performed by each path 312.

[0024] In one example, the DIMC is coupled to a clock generator and configured within one or more portions of each of the majority of the chiplet disks to enable high throughput of one or more matrix computations provided in the DIMC, such that the high throughput is characterized by 512 multiplication accumulations per clock cycle. In a specific example, the clock coupled to the DIMC is a second clock derived from a first clock (e.g., chiplet clock generator, AI accelerator device clock generator, etc.) configured to output a clock signal from approximately 0.5 GHz to 4 GHz; the second clock can be configured with an output rate of about half the rate of the first clock. The DIMC can also be configured to support block-structured sparse occupancy awareness (e.g.,by imposing structural restrictions on weight patterns of a neural network such as a transformer).

[0025] In one example, the SIMD device 350 is a SIMD processor coupled to an output of the DIMC. The SIMD 350 can be configured to apply one or more nonlinear operations and one or more linear operations to a vector process. The SIMD 350 can be a programmable vector unit or similar. The SIMD 350 can also include one or more random-access memory (RAM) modules, such as a data RAM module, an instruction RAM module, and the like.

[0026] In one example, the disk controller 360 is coupled to all blocks of each compute path 312 and also includes a control / status register (CSR) 362 that is coupled to each compute path. The disk controller 360 is also coupled to a memory bank 370 and a data transformation unit (DRE) 380. The disk controller 360 can be configured to forward data from the memory bank 370 to the blocks in each of the compute paths 312 and to coordinate these compute paths 312 via a processor interface (PIF) 364. In a specific example, the PIF 364 is coupled to the SIMD 350 of each compute path 312.

[0027] Further details about the 310 processing core are in Fig. Figure 3B shows the simplified block diagram of the disk device 302, which includes an input buffer 320, a DIMC matrix vector unit 330, an output buffer 340, a network-on-chip (NoC) device 342, and a SIMD vector unit 350. The DIMC unit 330 includes a plurality of in-memory computation (IMC) modules 332 configured to compute a scaled dot-product attention function on input data to determine a probability distribution, requiring high-throughput matrix multiplication and accumulation operations.

[0028] These IMC modules 332 can also be coupled with a block floating-point alignment module 334 and a subproduct reduction module 336 to further process the DIMC results before outputting them to the output buffer 540. In one example, the input buffer 320 receives input data (e.g., data vectors) from memory bank 370 (shown in Fig. 3A) and sends the data to the IMC modules 332. The IMC modules 332 can also receive instructions from memory bank 370.

[0029] In addition to the details described above, the SIMD 350 can be configured as an element-wise vector unit. The SIMD 350 can include a computation unit 352 (e.g., addition, subtraction, multiplication, maximum value, etc.), a lookup table (LUT) 354, and a state machine module (SM) 356 configured to receive one or more outputs from the output buffer 340.

[0030] The NoC device 342 is coupled via a short-circuit connection 344 to the output buffer 340, which is configured in a feedforward loop. Furthermore, the NoC device 342 is coupled to each of the disks and configured for multicast and unicast processes. Specifically, the NoC device 342 can be configured to connect all disks and all tiles, send multicast input activations to all disks / tiles, and collect the partial calculations to be sent unicast for a specially distributed accumulation.

[0031] Considering the previous example of an AI accelerator with eight chiplets, the input buffer can have a capacity of 64 KB with 16 banks, and the output buffer can have a capacity of 128 KB with 16 banks. The DIMC can be an 8-bit block with dimensions of 64x64 (eight 64x64 IMC modules), and the NoC can be 512 bits in size. The computation block in the SIMD can be configured for 8-bit and 32-bit integer (int) and unsigned integer (uint) calculations. These disk components can vary depending on which transformer the AI ​​accelerator will be serving.

[0032] Fig. Figure 4 is a simplified block diagram illustrating an example of an IMC Module 700. As shown, the Module 700 comprises one or more compute tree blocks 410 configured to perform desired calculations on input data from one or more read / write blocks 420. Each of these read / write blocks 420 comprises one or more first memory selectors 422 (also referred to as "W"), one or more second memory selectors 424 (also referred to as "I"), an activation multiplexer 426, and an operator unit 428. The first memory selector 422 provides an input to the operator unit 428, while the second memory selector 424 controls the activation multiplexer 426, which is also coupled to the operator unit 428. In the case of multiplication-accumulation operations, the operator unit 428 is a multiplier unit and the computation tree blocks 410 are multiplier-adder tree blocks (i.e.Σx.w).

[0033] As shown in close-up 401, each of the memory selector units 422, 424 comprises a memory cell 430 (e.g., an SRAM cell or the like) and a selector multiplexer 432. Each of the memory selector units 422, 424 is coupled to a read / write controller 440, which is also coupled to a memory bank / driver block 442. In one example, the read / write controller 440 can be configured with column write drivers and column read amplifiers, while the memory bank / driver block 432 can be configured with sequential row selectors.

[0034] An input controller 450 can be coupled to the activation multiplexer 426 of each of the read / write blocks 420. The input controller 450 can include a precision- and sparsely populated input activation register and driver. The operator unit 428 receives the output of the first memory selector 422 and receives the output of this block 450 via the activation multiplexer 426, which is controlled by the output of the second memory selector 424. The output of the operator unit 428 is then fed into the computation tree block 410.

[0035] The input identifier block 450 is also coupled to a clock source / clock generator 460. As explained earlier, the clock generator 460 can generate a second clock derived from a first clock configured to output a clock signal from approximately 0.5 GHz to 4 GHz; the second clock can be configured to output at approximately half the rate of the first clock. The clock generator 460 is coupled to one or more sign- and precision-aware accumulators 470 configured to receive the output of the computation tree blocks 410. In one example, an accumulator 470 is configured to receive the output of two computation tree blocks 410. Example output values ​​of the IMC are shown in the Fig. 13A-13C shown.

[0036] Referring to the example of the eight-chiplet AI accelerator device, the memory cell can be a dual-bank 2×6T SRAM cell, and the selector multiplexer can be an 8T-bank selector multiplexer. In this case, the memory bank / driver block 442 comprises a dual-bank SRAM bank. Additionally, the read / write controller can include a 64-byte write driver and a 64-byte read / write amplifier. Experts will recognize other variations, modifications, and alternatives to these IMC module components and their configurations.

[0037] Fig. Figure 5A is a simplified block flow diagram illustrating example numeric formats of the data processed in a disk. Diagram 501 shows a loop with the data formats for the GM / Input buffer 510, the IMC 520, the Output buffer 530, the SIMD 540, and the NoC 550, which is fed back to the GM / Input buffer 510. The IMC block 520 shows the multiplication-accumulation (Σx.w) operation. Additionally, the format for the data from the IMC 532 also flows to the Output 530. In this example, the numeric formats include integers (int), floating-point numbers (float), and block floating-point numbers (bfloat) of varying lengths.

[0038] Fig. 5B is a simplified diagram illustrating certain numerical formats, including specific formats found in Fig. 5A are shown. Block floating-point numbers can be used to overcome certain performance barriers. Transformer training is typically done in floating-point numbers, i.e., 32-bit or 16-bit floating-point numbers, and inference is typically done in 8-bit integers (“int8”). With block floating-point numbers, one exponent is shared across a set of mantissa significance values ​​(see the diagonally line-filled blocks of the int8 vectors at the bottom of Figure 5A). Fig. 5B), unlike floating-point numbers, where each mantissa has a separate exponent (see 32-bit and 16-bit floating-point formats at the top of Fig. 5A). The use of block floating-point numeric formats for training can exhibit the efficiency of fixed-point without the problems of integer arithmetic and also allows the use of a smaller mantissa, e.g., a 4-bit integer (“int4”), while maintaining accuracy. Furthermore, by using the block floating-point format (e.g., for activation, weights, etc.) and sparse population awareness, the inference of training models can be accelerated for improved performance. Professionals in this field will recognize other variations, modifications, and alternatives to these numeric formats used for processing transformer workloads.

[0039] Fig. Figure 6A shows a simplified transformer architecture 601. The typical transformer can be described as an encoder stack configured with a decoder stack, each of which may have one or more layers. Within the encoder layers 610, a self-attention layer 612 determines context information during the encoding of input data and feeds the encoded data into a feedforward neural network 616. Encoder layers 610 process an input sequence from bottom to top and convert the output into a set of attention vectors K and V. The decoder layers 620 also include a corresponding self-attention layer 622 and a feedforward neural network 626 and may additionally include an encoder-decoder attention layer 624, which uses the attention vectors from the encoder stack to assist the decoder in further context processing.The decoder stack outputs a vector of floating-point numbers (as in . Fig. (5B explained), which is passed to linear and softmax layers 630 to project the output into a final desired outcome (e.g., desired word prediction, interpretation, or translation). The linear layer is a fully connected neural network that projects the decoder's output into a larger vector (i.e., a logits vector) containing evaluations associated with all potential outcomes (e.g., all potential words), and the softmax layer converts these evaluations into probabilities. Based on the probability output, the projected word meaning can be selected based on the highest probability or other derived criteria, depending on the application.

[0040] An important class of transformer models includes those based solely on the decoder stack (e.g., transformer language models such as GPT-2, GPT-3, etc.), which pose particular challenges for inference. Fig. Figure 6B illustrates an example of an autoregressive token processing procedure that uses a decoder stack to process one token at a time from segment 602 ("the quick brown fox jumps over the lazy dog"). This decoder stack comprises a plurality of decoder layers 620, each containing a self-attention layer 622 and a neural feedforward network 626. Here, the input token "fox" is followed by the previous tokens "brown", "quick", "the", and the start token " <s>"processed".

[0041] In one example, query (Q), key (K), and value (V) vectors are created for each input token by multiplying each token by the weighting matrices Q, K, and V, respectively. The processing procedure for each token involves calculating the dot product of the current token's Q vector and the K vector of each token in the segment to determine the percentage values ​​for each token relative to the current token, as shown in Self-Attention layer 622. In this case, the procedure includes a masked self-attention process where future tokens are weighted as zero, so values ​​are calculated only for the current token and all previous tokens. The value vectors for each token are then multiplied by their respective values, and all are summed, resulting in an output for the current token.The output vector for each token is then passed to the next layer, 620, for further processing (represented by the shaded vectors following the dotted arrow). Since the Q, K, and V vectors are created for each token, the Q, K, and V matrices containing these vectors grow dynamically with each additional token (represented by the graph on the right side of the Self-Attention layer, 622).

[0042] Transformers are based on four parameters: sequence length (S) (i.e., number of tokens), number of attention heads (A), number of layers (L), and embedding length (H). Variations of these parameters are currently used to create virtually all transformer-based models. Embodiments of the present invention can be configured for all similar model types.

[0043] A transformer starts as untrained and is pre-trained by exposure to a desired dataset for a specific learning application. Transformer-based language models are exposed to large amounts of text (e.g., Wikipedia) to train language processing functions such as predicting the next word in a text sequence, translating the text into another language, and so on. This training process involves converting the text (e.g., words or word parts) into token IDs, evaluating the context of the tokens using a self-attention layer, and predicting the outcome using a feedforward neural network.

[0044] The self-attention process involves (1) determining query (Q), key (K), and value vectors (V) for embedding each word in an input, (2) calculating a score from the dot product of Q and K for each word of the input against a target word, (3) dividing the scores by the square root of the dimension of K, (4) passing the result through a softmax operation to normalize the scores, (5) multiplying each V by the softmax score, and (6) summing the weighted V vectors to produce the output. Note that the value matrix V becomes the weighting matrix for matrix multiplication by the softmax attention matrix; in the context of block floating-point numbers, this requires a column block converter for V, as described below.Column-wise blocking of V is more complex in decoder-transformer architectures, where the V matrix grows by one row for each additional token input. Column-wise blocking would require re-quantizing the last matrix tile to block the floating-point numbers for each additional row of V.

[0045] Many factors influence the performance of such transformer architectures. The softmax function is typically the critical path of the transformer layers (and has historically been difficult to accelerate in hardware). Requirements for the overlap of computations, SIMD operations, and NoC transfers also impact performance. Furthermore, the efficiency of NoC, SIMD, and memory bandwidth utilization is also significant.

[0046] Fig. Figure 7 is a simplified block flow diagram illustrating a figure process between a transformer and an exemplary AI accelerator device. As shown, a transformer 701 comprises a plurality of transformer layers 710, each of which has an attention layer 702. In this case, there are 16 attention heads 720 (e.g., BERT Large) that compute the attention function described earlier. These 16 attention heads are mapped, via a global CPU 732 that communicates with the tile CPUs 734, to 16 disks 730 of an AI accelerator device 703 (similar to devices 201 and 202).

[0047] According to one example, the present invention relates to the processing of Transformer workloads in a Transformer computing device. In certain applications, it is desirable to improve the processing of large amounts of data. For example, Transformer-based modeling networks typically comprise a huge number of elements (e.g., weights, activations, etc.), not all of which can be stored in on-chip memory. Accessing these elements therefore requires frequent transfers from a storage device (e.g., DDR), which, due to the high latency of such storage operations, can lead to the processing of these elements becoming memory-bound.Furthermore, quantizing the data into certain formats can present challenges if the target matrix data is characterized by a changing contraction dimension due to redundant quantizations, potential loss of accuracy, and inefficient storage / cache transfers.

[0048] Fig. Figure 8A is a simplified diagram illustrating a transformer calculating device 801 (or matrix multiplication calculating device) according to an example of the present invention. As shown, this device can be constructed similarly to the example disk device 301. Fig. 3A may be configured. All common reference numerals between these figures refer to the same elements as previously described. In contrast, device 801 comprises a cache storage device 830 coupled to the crossbar 360 and the storage device 370. The cache storage device 830 may include at least one first cache device 832 and one second cache device 834. The cache storage device 830 may also include additional cache devices.

[0049] Device 801 also includes a crossbar converter device 810, which is coupled to the crossbar 360, the input buffer (IB) device 320, and a weight buffer (WB) device 820, which is coupled to the compute device 330. The converter device 810 can receive data directly from the output buffer (OB) device 340, or from the storage device 370 or the cache storage device 1030 via the crossbar device 360. The converter device 810 can convert the data from a first format to a second format by determining mantissa values ​​and common exponent values ​​from the data in the first format. These mantissas and common exponents are then stored in a block configuration at a specific memory location (e.g., storage device 370, cache storage device 830, etc.).In a specific example, the first format can be a floating-point (FP) format, while the second format can be a block floating-point (BFP) format. Furthermore, the crossbar device 360 ​​can send the converted data to the IB device 320 and / or the WB device 820 to prepare it for processing by the computing device 330.

[0050] In one example, the WB device 820 can be configured together with the IB device 320 as a buffer device. Furthermore, the crossbar converter device 810 can be configured together or separately within each computation path 312. Alternatively, the crossbar converter device 810 can also be configured within the crossbar device 360 ​​and coupled to each computation path 312.

[0051] Fig. Figure 8B is a simplified diagram illustrating a transformer computing device 802 (or matrix multiplication computing device) according to an example of the present invention. As shown, this device 802 can be constructed similarly to the example disk device 302. Fig. 3B can be configured. In contrast, Device 802 includes the WB Device 820, which is coupled to the In-Memory Compute Modules (IMC) 332. Similar to the IB Device 320, the WB Device 820 is also coupled to the Network-on-Chip (NOC) Device 342 and to a storage device (designated by the input of "GM"). As explained previously, the WB Device 820 can be configured together with the IB Device 320.

[0052] This device comprises at least one data path with an IB device, a compute device coupled to the IB device, an OB device coupled to the compute device, and a SIMD device coupled to the OB device. One or more of these data paths and each of their components are coupled to a crossbar device, which is also coupled to at least one storage device. Furthermore, a crossbar converter device can be configured within the crossbar device or within each data path that couples the crossbar device and the OB device. In a specific example, the transformer compute device can be configured in a low-precision, high-accuracy system for generative large language models (LLMs) with support for BFP numerics and storage.This device can also be configured within a chiplet device and / or an AI accelerator device. Depending on the embodiment, this device can include any of the elements and configurations described above.

[0053] Fig. Figure 9 is a simplified block diagram illustrating an AI accelerator hardware-software system 900 according to an example of the present invention. As shown in System 900, a host computing device 910 comprises a host runtime environment 912 that operates at least one compiler stack 920, a workload preprocessor 930, and an execution stack 940. This hardware-software system can be configured for NIL and language model computations using a dataflow architecture and in-memory computational units for processing neural network model workloads (e.g., tensor operations and the like). Embodiments of this configurable system allow the selection of computational throughput, latency, power consumption, and functional accuracy.

[0054] The compiler stack 920 includes at least one Handles layer 922 and one ISA (Instruction Set Architecture) graph layer 924. The host runtime environment 912 can use the Handles layer 922 to determine references to resources for a neural network model workload, and the host runtime environment 912 can use the ISA graph layer 924 to translate a computation graph representing a target neural network model workload into machine code.

[0055] The workload preprocessor 930 can be configured to determine a plurality of workload parameters using the translated computation graph from the ISA graph layer 924. The host runtime environment 912 can then use the compiler stack 920 to issue commands for the workload parameters and instructions to the execution stack 940, which sends these commands to target hardware. Experts in this field will recognize other variations, modifications, and alternatives for configuring the host computing device 910 and its associated software system.

[0056] In one example, the target hardware comprises an AI accelerator device 950 with a plurality of chiplet devices 960 coupled to a CPU 962, which may include a global CPU and a plurality of local CPUs. The chiplet CPU 962 is coupled via its crossbar devices 972 to a plurality of matrix computation devices 970, each of which is coupled to at least one computation device 974 (e.g., a DIMC device) and one SIMD (Single Input, Multiple Data) device 976. In one example, compiler instructions are sent to the AI ​​accelerator device 950, which can be used to program the CPU 932 (or CPUs) and connected elements of the matrix computation device 970 via the crossbar device 972. The AI ​​Accelerator Device 950, the Chiplet Devices 960 and the Matrix Computing Device 970 can be configured similarly to the examples described above.

[0057] Although the matrix compute device 970 in this example is configured within a chiplet device 960 inside an AI accelerator device 950, the host compute device 910 can also be configured to send compiler instructions to an independent chiplet device containing matrix compute devices or to a server system with multiple AI accelerators. For example, the server system might include multiple AI accelerator PCIe card devices coupled to multiple switches, each of which is coupled to one or more server CPUs. Professionals in this field will recognize other variations, modifications, and alternatives to this workload transmission configuration.

[0058] Fig. Figure 10 is a simplified block diagram illustrating a server system according to an example of the present invention. As shown, the server system 1000 comprises a plurality of processor devices 1010, and each processor device 1010 is coupled to one or more storage devices 1020 and a network interface controller (NIC) 1030. In one example, the storage devices 1020 may include hard disk drives (HDDs) or solid-state drives (SSDs), such as an E1.S SSD, or the like. Here, each processor device 1010 is coupled to three storage devices 1020 (designated S0 to S2). Each processor device 1010 may also be coupled to one or more processor devices in a multiprocessor configuration.In a specific example, the processors in the multiprocessor configuration can be coupled using point-to-point processor connections, such as Ultra Path Interconnect (UPI) or similar technologies. Fig. In 10, the system 1000 comprises four multiprocessors, with each first processor device 1010 coupled to a second processor device 1012.

[0059] The System 1000 also includes multiple Switch Devices 1040, which are coupled to the Processor Devices 1010 and 1012. These Switch Devices 1040 can be configured for various form factors, such as Peripheral Component Interconnect Express (PCIe) or similar. Each Switch Device 1040 is coupled to every other Switch Device (e.g., using PCIe cables or similar). In a specific example, certain connections between the Switch Devices 1040 can be configured for pipeline traffic or host traffic. Fig. 10 The system 1000 comprises four switch devices 1040 (designated Sw0 - Sw3) which are coupled to the processor devices 1010, 1012, such that the second processor device 1012 is coupled to a different switch device 1040 than the first processor device 1010.

[0060] Here, the first processor device 1010 of the first multiprocessor is coupled to the first switch 1040, while the second processor device 1012 of the first multiprocessor is coupled to the second switch 1040. Similarly, the first processor device 1010 of the second multiprocessor is coupled to the first switch 1040, while the second processor device 1012 of the second multiprocessor is coupled to the second switch 1040. The third and fourth multiprocessors have a similar configuration, except for the third and fourth switches 1040. Although System 1000 demonstrates this pair-coupling configuration between the first and second processor units 1010 and 1012 and the switch devices 1040, the coupling configurations can be scaled to larger subsets of switch devices 1040 with multiprocessors containing additional processor units.

[0061] Each Switch Device 1040 is also coupled with one or more Processing Units (PUs) 1050, which may include GPU configurations, TPU configurations, or the like. These PUs 1050 can comprise the AI ​​accelerator device configurations described earlier, which can include various form factors such as PCIe or the like. In the PCIe card configuration, these PUs 1050 can function similarly to the AI ​​accelerator devices 101 and 102 from the Fig. 1A and Fig. 1B can be configured. In Fig. System 10 comprises four PU devices (designated PU0 - PU3). Experts in this field will recognize other variations, modifications, and alternatives to this server system configuration.

[0062] Fig. Figure 11 is a simplified block diagram illustrating a multi-node server system according to an example of the present invention. As shown, the multi-node server system 1100 comprises at least two server systems 1000 (see Figure 11). Fig. 10), which are configured as interconnected server nodes. Only the switch devices 1040 (labeled Sw0 - Sw3) are shown within each server system 1000 to highlight the example connections between the switch devices both within the node and between the two nodes. Here, the first switch device 1040 of the first node is connected to the fourth switch device 1040 of the second node, and the fourth switch device 1040 of the first node is connected to the first switch device 1040 of the second node. Depending on the application, the system 1100 may include one or more additional server nodes, and the connection configuration between the switches in the nodes may vary. Alternatively, the nodes may be connected via the NICs within each node system 1000, for example, via a pipelined Ethernet connection or similar.Experts will recognize other variations, modifications, and alternatives to this configuration of a multi-node server system.

[0063] Fig. Figure 12 is a simplified block diagram illustrating a portion of a server system according to an example of the present invention. As shown, the server system 1200 comprises a switch 1210 coupled to a plurality of PU card devices 1220. Similar to the server system 1000, this system 1200 comprises four PU devices 1220 (designated C0-C3) in a card form factor (e.g., PCIe card or the like). Here, the PU devices 1220 are similar to the AI ​​accelerator device 102 from the Fig. 1B is configured with eight chiplet devices 1240 arranged above an interposer 1230 in two groups of four interconnected chiplets 1240. Each of these chiplet devices 1240 also includes a connection interface 1242, such as a PCIe interface or the like. Furthermore, each group of chiplets 1240 is coupled to eight memory devices 1250 (e.g., DRAM or the like). However, the specific number and configuration of these chiplet devices in the AI ​​acceleration device can vary and include any of the configurations described above.

[0064] The server system 1200 also includes details of various connections between chiplet devices 1240 within the same PU device 1220 and between different PU devices 1220. As shown in the extended representation of the first and second PU devices "C1" and "C2", the switching device 1210 is coupled via connection paths 1212 to the connection interface 1242 of one of the chiplet devices 1240 of the first chiplet group in each PU device 1220. In a specific example, these connection paths 1212 can include printed circuit board (PCB) paths, cables, or the like. For both PU devices "C1" and "C2", another chiplet device 1240 of the first chiplet group is also coupled via its connection interfaces 1242 through connection paths 1222 to another chiplet device 1240 in the second chiplet group.In a specific example, these connection paths 1222 can also include printed circuit board traces, cables or the like.

[0065] Furthermore, it shows Fig. 12, that the remaining chiplet devices 1240, which were not coupled to the switch via connection paths 1212 or coupled across chiplet groups via connection paths 1222, are coupled to the PU devices 1220 via their connection interfaces 1242 using bridge connection paths 1232. More precisely, the two remaining chiplet devices 1240 in each group are coupled to chiplet devices 1240 of different chiplet groups in the other PU device 1220. The server system 1200 can include additional connections via connection paths 1212 (switch-to-chiplet), 1222 (group-to-group), and 1232 (card-to-card) to connect to other PU devices 1220 or, in the case of a different configuration of chiplet devices 1240 in the AI ​​accelerator device. Experts in this field will recognize further variations, modifications, and alternatives.

[0066] The present invention also provides methods and server system configurations that use transparent bridging to enable communication across multiple central processing unit (CPU) sockets and server nodes. By way of example only, the transparent bridging methods and configurations are applied to push-based communication using Ethernet connectivity. Those skilled in the art will recognize other variations, modifications, and alternative applications of these transparent bridging methods and configurations.

[0067] Traditional network interface cards (NICs) that provide Ethernet connectivity struggle to scale with accelerators distributed across nodes within a multi-node accelerator system (e.g., multi-node GPU accelerators and the like). And while network fabric configurations such as Remote Direct Memory Access (RDMA) and RDMA over Converged Ethernet (RoCE) can be used with multi-node accelerator systems to transfer data faster and more efficiently, these network fabrics can also require complex shared address space configurations (e.g., for one-way communication). Traditional software and hardware implementations of such network fabric configurations may also be limited to operating at the lower latency required for certain target applications (e.g., generative AI inference applications).Furthermore, the use of such a network structure configuration can be associated with various implementation challenges. For example, PCIe (Peripheral Component Interconnect Express) structure topologies without custom firmware are typically limited to scaling within PCIe switches and CPU sockets with PCIe lanes, resulting in limited peer-to-peer (P2P) connectivity between multiple CPU sockets.

[0068] By using a transparent bridge connection, the present invention enables fast and efficient communication between multi-node accelerator systems and server systems. Explicit software activation for the transparent bridge connection is not required, and data transmission using the transparent bridge connection can be ensured by applying specific communication protocols (e.g., Ethernet-based protocols) for scale-up and scale-out. Further details regarding these transparent bridge connection applications are described in the following figures.

[0069] Fig. Figure 13A is a simplified block diagram illustrating a server system that uses a transparent bridging synthetic fabric-switch connectivity according to an example of the present invention. As shown, the server system 1301 can comprise a plurality of CPU devices 1310. In a specific example, the plurality of CPUs 1310 can be configured as one or more multiprocessors coupled to each other via point-to-point processor connections such as Ultra Path Interconnect (UPI) and the like. Here, the CPUs 1310 are configured in pairs as “CPU1” and “CPU2”.

[0070] Each of the CPU devices 1310 is also coupled to a switching device 1320. Here, the switches 1320 coupled to the CPUs 1310 of each dual-core multiprocessor are referred to as "Switch1" and "Switch2". These switch devices 1320 can be configured for various form factors, such as Peripheral Component Interconnect Express (PCIe) and the like. In one example, the switches 1320 configured with each multiprocessor are also interconnected via a synthetic fabric configuration 1322 (e.g., PCIe fabrics, Ethernet fabrics, and the like). Although the system 1300 is depicted with pairs of CPUs 1310 and switches 1320, the coupling configurations can also be scaled to larger subsets of CPUs 1310 and switch devices 1320.

[0071] Each Switch Device 1320 is also coupled with one or more Processing Units (PUs) 1330, which may include GPU configurations, TPU configurations, or the like. These PU Devices 1330 can include the previously described AI accelerator device configurations, which can encompass various form factors such as PCIe cards and the like. In the PCIe card configuration, these PU Devices 1330 can function similarly to the AI ​​Accelerator Devices 101 and 102 in the Fig. 1A and Fig. 1B can be configured. In Fig. 13A comprises the system 1000 four PU devices (designated PU1 - PU4), which are configured in pairs and coupled by bridge connections 1332 (see Fig. 12). Here, the synthetic fabric configuration 1322 enables communication between PU devices 1330 on different switches.

[0072] Each Switch Device 1320 is also coupled with at least one Input / Output Streaming (IO) Device 1340, which can be configured in the same way as the PU Devices 1330. These IO Streaming Devices 1340 can be configured to implement transparent bridging to facilitate peer-to-peer communication between PU Devices 1330 coupled with different multiprocessors. Here, the IO Streaming Device 1340 coupled to "Switch2" of the left multiprocessor is configured to communicate with the IO Streaming Device 1340 coupled to "Switch1" of the right multiprocessor. In addition, the IO streaming devices 1340, which are coupled with the other switch device 1320 of each multiprocessor, can be configured to communicate with IO streaming devices 1340 of other multiprocessors.

[0073] In one example, these 1340 IO streaming devices can be configured to transparently transport data using transaction layer packets (TLPs) for memory functions (e.g., PCIe MemWr64) and completion packets (e.g., TLP prefixes). The 1340 IO streaming devices can also duplicate the next level of parallelism (e.g., pipeline parallelism, tensor parallelism, etc.) for memory base address register (BAR) ranges used by the 1310 CPUs. In one specific example, the 1340 IO streaming devices are configured for PCIe peer-to-peer communication using an Ethernet architecture.

[0074] In one example, the System 1301 can also be configured so that an I / O streaming device configured for one of the multiprocessors and coupled to one of the switches in the synthetic fabrication configuration manages communication with I / O streaming devices configured for other multiprocessors. Each multiprocessor with CPUs 1310 and associated switches 1320, PU devices 1330, and I / O streaming devices 1340 can also be configured as a separate server node in a multi-node server system. Examples of multi-node server systems using transparent bridges are described in the Fig. 13B and Fig. 13C shown.

[0075] Fig. Figure 13B is a simplified block diagram illustrating a server system that uses a transparent bridge connection for the connection between switches within an intranode and the connection between nodes according to an example of the present invention. As shown, system 1302 is similar to system 1301 in Figure 13B. Fig. 13A, except that each multiprocessor is configured within a separate server device 1350, and the IO streaming devices 1340 are configured for peer-to-peer communication between server nodes 1350 (i.e., communication between nodes). Furthermore, the switch devices 1320 coupled to each multiprocessor are not coupled to each other via a synthetic fabric configuration. Instead, each switch device 1320 under each multiprocessor additionally includes an intra-node I / O streaming device 1342, which is configured for peer-to-peer communication across the switches 1320 within the server node 1350 (i.e., intra-node P2P communication), rather than using the CPU UPI interface.

[0076] In this example, the intranode I / O streaming devices 1342 enable peer-to-peer communication between the PU devices 1330, which are coupled to the two switches 1320 within the server node 1350, which are coupled to two different CPU sockets. These intranode devices 1342 can be configured for low-latency, high-bandwidth communication. In a server rack configuration, this intranode connectivity can also use loopback methods, instead of going to the top of the rack, to obtain a low-latency interface from top-of-rack (ToR) switches and the like.

[0077] Fig. Figure 13C is a simplified block diagram illustrating a server system that uses a transparent bridge connection for linking sockets within an intranode and for linking nodes according to an example of the present invention. As shown, system 1303 is similar to system 1302 from Figure 13C. Fig. 13B, except that there are no switching devices 1320 in each server node 1352. Instead, each CPU 1310 (which is configured in a separate CPU socket) is directly coupled to the PU devices 1330, the IO streaming devices 1340 between the nodes, and the IO streaming devices 1342 within the intranode.

[0078] In this example, the IO streaming devices 1342 in the intranode enable peer-to-peer communication between PU devices 1330 that are coupled to different CPU sockets, instead of using the CPU UPI interface. Similar to the configuration in Fig. 13B can also use loopback methods in this server configuration. Experts in this field will recognize other variations, modifications, and alternatives to these server configurations that use a transparent bridge connection.

[0079] In one example, the present invention provides an AI system that uses a transparent bridge connection. This system comprises a first server device with at least one first host CPU coupled to a first PCIe switch, which is also coupled to a plurality of first AI accelerator devices. The system further comprises an input network interface, an intranetwork interface, and an output network interface. These interfaces can be configured to enable communication between the first server device and a second server device, which comprises at least one second PCIe switch coupled to a second host CPU, a plurality of second AI accelerator devices, the output network interface, and the intranetwork interface.

[0080] In one example, the input network interface is configured to receive information using a task ID from a second server device configured with an Ethernet protocol. Both the first and second host CPUs can be configured to run a compiler concurrently and independently of the operation of the intranetwork interface to transmit information with the task ID. The first and second host CPUs can also be configured to be unsynchronized with the input network interface, the intranetwork interface, or the output network interface. The second server device can be configured to wait (e.g., in an idle state or while maintaining its previous operating state) or to process an instruction for one or more task IDs.Alternatively, the second server device can be configured to process a previously received data set until it receives the next data set (e.g., from another accelerator device).

[0081] In one example, the intranetwork value interface is configured to communicate from one or more of the first AI accelerator devices to one or more of the second AI accelerator devices, using one or more task IDs without communicating directly or indirectly from a second host central processing unit. The intranetwork value interface can also facilitate communication between one or more of the first AI accelerator devices and one or more of the second AI accelerator devices. Furthermore, communication from the intranetwork value interface can occur without instructions from either the first or second host CPU. This communication can also be transparent to each of the first AI accelerator devices and the second AI accelerator devices.

[0082] In one example, the output network interface is configured to transmit information to the second server device using one or more task IDs and the Ethernet protocol. The output network interface and the input network interface can each include a PCIe endpoint device, a first input / output (I / O) bridge device, a second I / O bridge device, and a device configured to communicate using a communication protocol. In another example, both the input network interface and the output network interface can be configured as I / O network interfaces, and the configurations and procedures for communication can also be applied from the second server device to the first server device.

[0083] The system can be configured as a multi-node AI server system and may also include an inter-node network interface, which can be configured similarly to the intra-network interface but is used for communication between server nodes. Each system can also be configured so that the majority of the AI ​​accelerator devices are coupled to the host CPU socket, and the inter-node network interface can facilitate communication across CPU sockets. These and other examples are described in the following figures.

[0084] Fig. Figure 14 is a simplified block diagram showing a multi-node server system that uses a transparent bridge connection for scaling up and outwards according to an example of the present invention. As shown, the system 1400 comprises a switch 1410 (e.g., a ToR Ethernet switch and the like) coupled to a plurality of server node devices 1420 (numbered from 1 to N), and each of the server nodes 1420 is coupled to a first IO streaming device 1430 and a second IO streaming device 1432. In one example, the first IO streaming device 1430 can be configured to receive data from the switch 1410, and the second IO streaming device 1432 can be configured to transmit data to the switch 1410. Alternatively, both IO streaming devices 1430 and 1432 can be configured to receive and transmit data.Depending on bandwidth and latency requirements, each IO streaming device can be configured as a transmitter, receiver, or transceiver.

[0085] In one example, these IO streaming devices 1430 and 1432 are configured to implement a transparent bridge to scale the network of server nodes 1420 in the multi-node server system 1400. The system 1400 can be configured as a lossless or lossy network. Furthermore, each of the nodes 1420 can be configured similarly to the server node configurations described previously. In this case, the IO streaming devices 1430 and 1432 can also be configured to communicate via switches within the nodes 1420 using a transparent bridge connection. Experts in this field will recognize other variations, modifications, and alternatives for scaling in a multi-node server system.

[0086] Fig. Figure 15 is a simplified block diagram illustrating an I / O streaming device according to an example of the present invention. As shown, the device 1500 comprises an endpoint (EP) device 1510 coupled to a communication device 1530 via a first bridge device 1520 and a second bridge device 1522, each of which can be configured as a sender bridge path and receiver bridge path, respectively. The first and second bridge devices 1520 and 1522 are configured for communication between the EP device 1510 and the communication device 1530 using a transparent bridge connection. The EP device 1510 is configured for communication using a specific interface standard, such as PCIe, and the like. In one example, the EP device 1510 can be connected to another EP device (e.g.,It can be coupled to another IO streaming device, an AI accelerator device, etc., a switch, or a root complex. Depending on the application, the EP device 1510 and the communication engine 1530 can be configured for various connection technologies (e.g., PCIe, Ethernet, etc.).

[0087] The communication device 1530 is configured to communicate with other devices (e.g., within a server system) using one or more communication protocols, such as a Transmission Control Protocol / Internet Protocol (TCP / IP), a die-to-die (D2D) interface communication protocol, an Ethernet communication protocol, a Layer 2 (L2) communication protocol, and the like. For example, the engine device 1530 may include a TCP / IP offload engine (TOE) configured for lossy networks. The communication protocol may include a guaranteed delivery scheme (i.e., no packet loss), which may include a retry buffer and a congestion control scheme with a pause packet. The D2D interface communication protocol may include D2D logic configured for lossless networks with porting support (e.g.,configured from application-specific integrated circuits [ASIC] to field-programmable gate arrays [FPGA].

[0088] The first bridge device 1520 is configured as a bridge from the EP device 1510 to the communication device 1530, while the second bridge device 1522 is configured as a bridge from the communication device 1530 to the EP device 1510. In one example, the first bridge device 1520 can also be configured to manage a network communication flow control system for the IO streaming device, such as in a PCIe credit-based flow control system. The first bridge device 1520 can facilitate memory write operations by sending data / completion packets with the EP device 1510. These packets can include transaction layer packets (TLPs), data link layer packets (DLLPs), and the like. In one example, completion packets follow an ordered data flow through the first bridge device 1520.The first bridge device 1520 can also stream data and control signals to the communication device 1530 using a streaming interface, such as an expandable interface (AXI) or similar. Similarly, the second bridge device 1522 can receive control signals and data streamed from the communication device 1530 using the streaming interface. Furthermore, the second bridge device 1522 can send memory write requests to the EP device 1510 using similar data packets. Using this configuration, the IO streaming device 1500 can connect one endpoint to another, one endpoint to a root complex, or one root complex to another.

[0089] In a specific example, the IO streaming device 1500 includes an FPGA device configured for PCIe communication with support for TLP prefix and steering tag. The EP device 1510 can be a PCIe EP device, and the communication engine 1530 is configured for Ethernet connectivity via a TOR switch. In this case, the first and second bridge devices 1520 and 1522 are configured as PCIe Ethernet bridges. Professionals in this field will recognize other variations, modifications, and alternatives to this IO streaming device configuration.

[0090] Fig. Figure 16 is a simplified block diagram illustrating an I / O streaming data format according to an example of the present invention. As shown, the data format 1600 can include the following fields: Preamble 1610, Start Frame Delimiter (SFD) 1620, Destination 1630, Source 1640, Type / Length 1650, Data 1660, and Frame Check Sequence (FCS) 1670. The data field can be configured to include subfields to allow transparent bridging using data packets, such as the Destination Device Number 1662, the Packet Header Prefix 1664, and the Packet Data 1666. The Source 1640 and the Destination 1630 can be in the same domain (e.g., PCIe domain) or in different domains.

[0091] In a specific example, data format 1600 is configured as an Ethernet frame, and data field 1660 integrates PCIe TLP information (e.g., TLP header prefix and TLP data) to enable transparent bridging across PCIe switches. Professionals in this field will recognize other variations, modifications, and alternatives to the data format used for transparent bridging.

[0092] According to one example, the present invention provides techniques for configuring data transmissions that cross a channel boundary synchronized using a software-assigned task ID. In one example, the present techniques configure a sender that does not need to explicitly know (i.e., the source device assumes that the protocol used to send the data guarantees delivery of the data to a destination device) that a data transmission has been received by a receiver. In one example, NoC channels are assumed to be lossless, and all external connections are assumed to have integrity / retry functions. Accordingly, sending data from the sender's perspective means completing the task from the sender's point of view. In one example, a receiver does not need to know the identity of the sender who transmitted the data, nor does it need to know when the data was transmitted.In one example, the receiver only needs to prevent reading and / or using a buffer label for such data until the data arrives. Therefore, the receiver only requires a barrier that delays downstream execution or makes it dependent on data transmission to the receiver. In another example, both the sender and receiver are independent of each other; the only instance aware of their relationship is the compiler configured on a host central processing unit. Thus, the task ID assigned by the software serves as the means to establish a producer-consumer relationship for long-distance data transmission. This configuration can be extended to a multi-host configuration, for example, in a multi-node server system where the compiler runs on multiple host processors and can configure multiple senders and receivers.

[0093] In one example, the present invention provides a system and method for PCIe peer-to-peer (P2P) write operations between different sets of accelerators using a transparent bridge connection via input / output streaming (IO) devices that can be configured as transparent NIC devices. In this example, the transparent network card is represented such that it exposes the base address registers (BARs) of adjacent cards and represents them as mirrored BARs. However, the present system and method are not limited to communication between only two adjacent cards. In one example, the present system and method can be configured to enable communication between nodes (e.g., servers) for tensor parallelism, pipeline parallelism, and the like.In one example, the present system and procedure provide a distribution of a number of mirrored BARs exposed per map, determined by the bandwidth and latency requirements of a workload. These requirements vary depending on whether the workload involves tensor parallelism or pipeline parallelism. To meet the bandwidth and latency requirements for tensor parallelism, additional I / O streaming devices can be integrated per node using the presented techniques for scalability and high-performance AI workloads.

[0094] Fig. Figure 17A is a simplified block diagram illustrating a method 1701 for transferring data using a transparent bridge according to an example of the present invention. As shown in flowchart 1720, this method may include step 1722 of processing receive operations (e.g., memory writes) using a FIFO (First In, First Out) data structure. This FIFO data structure may be implemented in a storage device (e.g., static random access memory [SRAM], dynamic random access memory [DRAM], high bandwidth memory [HBM], and the like) within the IO streaming device to receive data packets from other devices coupled to the same switch or CPU socket.This storage device can be configured within the EP device or the bridge device on the transmission data path from the EP device to the communication engine or coupled to it (see . Fig. 15) In a specific example, the FIFO structure can be configured for the previously discussed TLPs, which are used to transfer data between PCIe devices.

[0095] In step 1724, the procedure includes the use of bridging logic to map mirror BAR memory areas 1712 to device IDs 1714 (e.g., card ID for PCIe cards and the like) using a first lookup table (LUT) 1710. This step may also include extracting the address suffix. And in step 1726, the procedure includes creating data packets that can be transmitted using the communication facility of the IO streaming device (see Fig. 15) This step may also include the generation of data fields, which may involve steps such as integrating the TLP information into an Ethernet frame, as in Fig. Figure 16 illustrates this. Using the communication engine, these data packets can be sent to devices in different domains (e.g., another switch, CPU socket, node, etc.) of a multi-node server system.

[0096] Fig. Figure 17B is a simplified block diagram illustrating a method 1702 for receiving data using a transparent bridge according to an example of the present invention. As shown in method 1702, the receive data path operation is the reverse of the transmit data path operation. In step 1742, the method includes parsing the packets received from the IO streaming device in the communication engine. Subsequently, in step 1744, the method includes using bridging logic to map device IDs 1732 (e.g., card IDs) to real bar spaces 1734 using a second LUT 1730. This step may additionally include adding address suffixes and packet formation.And in step 1746, the procedure includes the processing of the receive operations in a transmit FIFO data structure, which can be configured or coupled within the EP device of the bridge device on the receive data path from the communication device to the EP device (see . Fig. 15). As for Fig. As explained in section 17A, the FIFO structure can be configured for TLPs. Experts in this field will recognize other variations, modifications, and alternatives to these methods for the transmit and receive data path.

[0097] Fig. Figure 18A is a simplified block flow diagram illustrating a method for sending data in a server system with accelerator devices according to an example of the present invention. The method 1801 shown here represents a peer-to-peer data transfer between a first accelerator 1810 of a first server node and a second accelerator 1812 of the same server node. Each of these accelerator devices 1810, 1812 can be configured similarly to the AI ​​accelerator devices, PU devices, etc., described above. The data transfer includes a plurality of memory writes from the first accelerator device 1810 to the second accelerator device 1812. The last memory write also includes a prefix that causes the second accelerator device 1812 to begin processing the transferred data.In a specific example, the P2P data flow is carried out over PCIe connections (e.g. via a shared PCIe switch using a PCIe Fabric standard mode).

[0098] In one example, the P2P flow can follow a push-based mechanism where a source accelerator device (e.g., the first accelerator device 1810) writes to pre-programmed BAR addresses of the target accelerator device (e.g., the second accelerator device 1812). As explained earlier, each accelerator device is configured to process a computation graph representing a neural network model workload. In one example, the compiler-provided graph of the target accelerator device (see Fig. 9) Assumes that data is written from the source accelerator device to a predefined memory location. In a specific example, when the target accelerator device receives a task ID flag via a prefix (e.g., PCIe TLP prefix, etc.) or a unit of information (e.g., PCIe TLP Dword), it compares the received task ID with a pre-programmed task ID and proceeds with execution if the task IDs match. The procedure can also include a double-buffering process, where the data movement between the source accelerator device and empty memory is scheduled so that the target accelerator device does not have to wait for the data to be computed.

[0099] In one example, the present invention provides a method for transferring data between a pair of AI accelerator devices within a server node (i.e., data transmission within a node). The node comprises at least one pair of processing units, each of which is coupled to a PCIe switch, the PCIe switch being coupled to a plurality of AI accelerator devices and an intranodal source network interface card (NIC) device.

[0100] Fig. Figure 18B is a simplified block flow diagram illustrating a method for transferring data in a server system with accelerator devices using a transparent bridge according to an example of the present invention. The method 1802 shown here represents data transmission from an accelerator device 1810 of a first server node to an accelerator device 1820 of a second server node using an IO streaming device 1830 of the first server node and an IO streaming device 1832 of the second server node. This data transmission process includes transparently forwarding a memory write operation and a prefix (e.g., PCIe memory write operation and TLP prefix to Ethernet).

[0101] Here, procedure 1802 involves multiple memory writes and a prefix from the first accelerator device 1810 to the I / O streaming device 1830 within the same first server node. In a specific example, these memory writes are performed over a PCIe connection (e.g., a shared PCIe switch or CPU socket). After each memory write, the I / O streaming device 1830 of the first node forwards the memory writes and the prefix to the I / O streaming device 1832 of the second node. In a specific example, the memory write and the prefix are configured in an Ethernet frame to be transmitted between the I / O streaming devices in different nodes. Similarly, the I / O streaming device 1832 of the second node transmits the memory writes and the prefix to the accelerator device in the second node.In a specific example, these memory write operations are also performed via a PCIe connection.

[0102] In one example, the present invention provides a method for transferring data between a pair of AI accelerator devices between at least one pair of nodes (i.e., data transmission between nodes), including a first node and a second node. Each node comprises at least one pair of processing units, each of which is coupled to a PCIe switch, the PCIe switch being coupled to a plurality of AI accelerator devices and an internode source network interface card (NIC) device.

[0103] Fig. Figure 18C is a simplified block flow diagram illustrating a method for transferring data in a server system with accelerator devices using a buffered transparent bridge according to an example of the present invention. As shown, Method 1803 is similar to Method 1802 in Fig. 18B, except for the additional buffering of data transmission using a store-and-forward approach to pack more data packets and / or the prefix and achieve higher bandwidth. In this case, the packets are consolidated into a buffer 1840 in the IO streaming device 1830 of the first node. The consolidated packets are then unpacked into a buffer 1842 in the IO streaming device 1832 of the second node. The IO streaming devices can be configured to manage the flow with low latency and high bandwidth in cases where a buffered transparent bridge is used. In a specific example, the PCIe TLPs and the prefix are buffered to achieve higher bandwidth on Ethernet IO.

[0104] Fig. Figure 18D is a simplified block diagram illustrating a method for transferring data in a server system with accelerator devices, each comprising a data path controller and one or more subsystems, according to an example of the present invention. The method 1804 shown here represents a data communication flow from a data path controller 1850 of a first accelerator device, via a subsystem 1852 of the first accelerator device, to a data path controller 1860 of a second accelerator device, via a subsystem 1862 of the second accelerator device. In one example, the first accelerator device 1850 can either perform a plurality of memory writes from its local memory to the subsystem 1852 (e.g., PCIe subsystem) or instruct the subsystem 1852 to perform a memory read from its local memory (e.g., perform a direct memory access [DMA]).This process may involve informing the subsystem about the task ID or the field for the completion package to be transferred as part of the final package.

[0105] Here, the first accelerator device is the source device and the second accelerator device is the destination device. The data flow between subsystem 1852 of the first accelerator device and subsystem 1862 of the second accelerator device can follow a similar peer-to-peer or transparent bridge process with multiple memory writes and prefixes, as described previously. Because the task ID transferred from the source device is pre-programmed by the compiler, it matches the expectations of the destination device.

[0106] In one example, the second accelerator receives the packets as subsystem 1862 and forwards the data to its local memory using memory writes via the data path controller 1860. The second accelerator is pre-programmed by the compiler with task ID information so that it matches the received completion packet and task ID. After receiving the matching completion packet and task ID, the second accelerator can execute its associated computation graph. Experts in this field will recognize other variations, modifications, and alternatives to these data transmission methods.

[0107] Although the above constitutes a complete description of the specific embodiments, various modifications, alternative designs, and equivalents may be used. For example, the AI ​​accelerator device and the chiplet devices may comprise any combination of the elements described above, as well as elements outside the present specification. Therefore, the above description and the figures should not be considered as limiting the scope of the present invention, which is defined by the accompanying claims.< / s>

Claims

[1] AI system encompassing: a first server device encompassing: a first host central processing unit; a first PCIe switch coupled with the first host central processing unit; a plurality of first AI accelerator devices coupled with the first PCIe switch; an input network interface configured to receive information using a task ID from a second server device configured with an Ethernet protocol; an intranetwork value interface configured to communicate from one or more of the plurality of first AI accelerator devices to one or more of a plurality of second AI accelerator devices, such that the communication uses one or more task IDs without communicating directly or indirectly from a second host central processing unit; an output network interface configured to transmit information to the second server device using one or more task IDs via the Ethernet protocol; and a second PCIe switch configured to couple the second host central processing unit with the majority of second AI accelerator devices, the output network interface, and the intra-network value interface. [2] System according to claim 1, wherein the intra-network interface facilitates communication between one or more of the plurality of the first AI accelerator devices with one or more of the plurality of the second AI accelerator devices. [3] System according to claim 1, wherein each of the first host central processing unit and the second host central processing unit is configured to operate a compiler simultaneously with and independently of the operation of the intra-network interface to transfer information with the task ID. [4] System according to claim 1, wherein the second server device is configured to wait for or process an instruction for one or more task IDs. [5] System according to claim 1, wherein the first host central processing unit and the second host central processing unit are not synchronized with the input network interface, the intra-network value interface or the output network interface. [6] System according to claim 1, wherein the communication from the intra-network value interface is provided free of instructions either from the first host central processing unit or the second host central processing unit. [7] System according to claim 1, wherein the communication is transparent for each plurality of the first AI accelerator devices and the second AI accelerator devices. [8] System according to claim 1, wherein the output network interface and the input network interface each comprise: a PCIe endpoint device; a first I / O bridge device; a second I / O bridge device; and a device configured to communicate using at least one of: a TCP / IP communication protocol, a die-to-die interface communication protocol, or a Layer 2 communication protocol with a guaranteed delivery scheme.

Images