Data passing through the gateway

Abstract

A gateway for use in a computing system, for interfacing a host with a subsystem for acting as a working accelerator for the host, the gateway having: an accelerator interface for connecting to the subsystem to enable transfer of batch data between the subsystem and the gateway; a data connection interface for connecting to an external storage to exchange data between the gateway and the storage; a gateway interface for connecting to at least one second gateway; a memory interface connected to a local memory associated with the gateway; and a streaming engine for controlling streaming of batch data entering and leaving the gateway in response to a pre-compiled data exchange synchronization point reached by the subsystem, wherein the streaming of the batch data is selectively via at least one of the accelerator interface, the data connection interface, the gateway interface, and the memory interface.

Description

Technical Field

[0001] The present disclosure relates to a gateway for use in a computer system to interface a host with a subsystem for acting as a work accelerator, and more particularly to the use of the gateway for a scalable system that provides such a work accelerator. Background Art

[0002] In the context of processing data for complex or high-volume applications, a work accelerator may be a subsystem to which the processing of certain data is offloaded from a host system. Such a work accelerator may have dedicated hardware for performing a specific type of processing.

[0003] As an example, a class of computing fields in which such dedicated accelerator subsystems can be used is found in machine intelligence. As is familiar to those skilled in the art of machine intelligence, machine intelligence algorithms are based on iterative updates to a "knowledge model", which can be represented by a graph of multiple interconnected nodes. The implementation of each node involves processing of data, and the interconnection of the graph corresponds to the data to be exchanged between nodes. Typically, at least some of the processing of each node can be performed independently of some or all other nodes in the graph, and therefore large graphs have a great opportunity to use multithreading. Therefore, a working accelerator dedicated to machine intelligence applications can include a large degree of multithreading. A form of parallelism can be achieved by means of a processor, which includes multiple tiles arranged on the same chip (i.e., the same die), each tile including its respective separate corresponding processing unit and memory (including program memory and data memory). Therefore, separate parts of the program code can be run in parallel on different tiles. The blocks are connected together via on-chip interconnections, which enables data exchange between blocks. Such accelerators can be used as subsystems of a host system to perform parallel processing on data sets provided to it.

[0004] Generally speaking, there may be dependencies between program parts running on different blocks. Therefore, a technique is needed to prevent a piece of code on one block from running before the data it depends on and available through another piece of code on another block. There are many possible schemes for achieving this purpose, for example, one of which is described as "BSP (bulk synchronous parallel)", that is, batch synchronous parallelism. According to BSP, each block performs a calculation phase and an exchange phase in alternating cycles. During the calculation phase, each block performs one or more calculation tasks locally on the block, but does not communicate any of its calculation results with any other block. In the exchange phase, each block is allowed to exchange one or more calculation results from the previous calculation phase to one or more other blocks in the group and / or exchange one or more calculation results from the previous calculation phase from one or more other blocks in the group, but the block has not yet entered the next calculation phase. In addition, according to the BSP principle, barrier synchronization is placed at the handover from the calculation phase to the exchange phase, or the handover from the exchange phase to the calculation phase, or the handover between the two. That is: (a) all blocks in the group are required to complete their respective computation phases before any block in the group is allowed to proceed to the next computation phase, or (b) all blocks in the group are required to complete their respective computation phases before any block in the group is allowed to proceed to the next computation phase, or (c) both. In some cases, blocks performing computations may be allowed to communicate with other system resources such as network cards or storage disks, as long as this does not involve communication with other blocks in the group.

[0005] In order to increase the processing power of the system for performing these operations, it is advantageous to provide a system incorporating multiple subsystems for acting as working accelerators. In the above mode, the blocks of each accelerator are configured to process data in parallel during the calculation phase, and then exchange data during the exchange phase. By combining multiple accelerators, the blocks of multiple independent accelerators can process data in parallel, and then exchange data not only between blocks within an accelerator, but also between blocks belonging to different accelerators. In this way, accelerators can be combined to produce greatly improved processing power for performing parallel operations.

[0006] One challenge that arises when providing a scalable accelerator system is how to ensure that data is transmitted between the accelerator and other data sources in the system (such as one or more hosts) so that the data is delivered to the appropriate accelerator for processing at the right time. A first accelerator may need to process a set of data received from a host during a first computation phase. A second accelerator may then process the results of the above processing during a second computation phase. Delivering data between different entities (e.g., a host and different accelerators) to provide an extended accelerator system including cooperating subsystems presents technical challenges. Summary of the invention

[0007] The present invention uses the concept of a gateway that can be used to provide data from external storage to an accelerator and thus interconnect them to expand a subsystem that acts as a working accelerator. The accelerator receives data from the gateway during the exchange phase and computes the data in a subsequent computation phase. In some forms of gateways, the gateway itself is an active processor of data and manages its data flow autonomously. The gateway acts as an intermediary between external storage and one or more accelerators. The gateway has a local memory for temporarily storing data delivered from external storage to the accelerator. This poses a challenge, namely how to ensure that the gateway does not fall behind the accelerator when providing the data that the accelerator needs to process in the upcoming computation phase.

[0008] According to a first aspect, there is provided a gateway for use in a computing system, for interfacing a host with a subsystem for acting as a working accelerator for the host, the gateway having: an accelerator interface for connecting to the subsystem to enable transfer of batch data between the subsystem and the gateway; a data connection interface for connecting to an external storage to exchange data between the gateway and the storage; a gateway interface for connecting to at least one second gateway; a memory interface connected to a local memory associated with the gateway; and a streaming engine for controlling streaming of batch data entering and leaving the gateway in response to a pre-compiled data exchange synchronization point reached by the subsystem, wherein the streaming of the batch data is selectively via at least one of the accelerator interface, the data connection interface, the gateway interface, and the memory interface.

[0009] In some embodiments, one or more of the batches of data are transferred from the local memory to the subsystem in response to a pre-compiled data exchange synchronization point being reached by the subsystem.

[0010] In some embodiments, one or more batches of data are made available at indicated storage locations in the local memory to be pulled from the local memory by the subsystem.

[0011] In some embodiments, the streaming engine is configured to execute instructions to push one or more batches of data from the local memory to the subsystem.

[0012] In some embodiments, the streaming engine is configured to stream one or more batches of data from the at least one second gateway into the local memory prior to a pre-compiled data exchange synchronization point reached by the subsystem.

[0013] In some embodiments, the streaming engine is configured to stream one or more batches of data from the external storage into the local memory prior to a pre-compiled data exchange synchronization point reached by the subsystem.

[0014] In some embodiments, the streaming engine includes a data mover engine and a memory management engine, the data mover engine and the memory management engine are configured to execute instructions from a work descriptor in a coordinated manner, the memory management engine is configured to execute instructions from the work descriptor to stream one or more batches of data into a local memory before a precompiled data exchange synchronization point reached by the subsystem, and the data mover engine is configured to execute instructions from the work descriptor to push one or more batches of data from the local memory to the subsystem in response to a precompiled data exchange synchronization point reached by the subsystem.

[0015] In some embodiments, the gateway is configured to: receive an additional batch or more batches of data in the batch data at the accelerator interface in response to a precompiled data exchange synchronization point reached by the subsystem; and store the additional batch or more batches of data in a local memory, wherein the streaming engine is configured to subsequently stream the additional batch or more batches of data from the local memory to the external storage.

[0016] In some embodiments, the memory management engine is configured to execute instructions from the work descriptor to stream one or more additional batches of data to external storage.

[0017] In some embodiments, the gateway is configured to receive a data packet from a subsystem in response to a pre-compiled data exchange synchronization point being reached by the subsystem.

[0018] In some embodiments, the gateway is configured to: check an address of a data packet to determine whether the data packet is addressed to a resource associated with the gateway; and in response to determining that the data packet is addressed to a resource associated with the gateway, forward the data packet to the resource, wherein the resource associated with the gateway includes one or more of: a second subsystem for acting as a working accelerator of the host; and a memory of the gateway.

[0019] In some embodiments, the gateway is configured to: look up in a routing table a port of a gateway interface corresponding to an address indicated in the data packet; and forward the data packet through the port of the gateway.

[0020] In some embodiments, the address indicates an address of an accelerator associated with a second gateway, wherein the gateway is configured to forward the data packet to the second gateway through the gateway interface.

[0021] In some embodiments, the address is an address in a global address space that encompasses multiple accelerators and gateways in the extended system.

[0022] In some embodiments, the external storage includes at least one of: storage of the host; local storage and network attached storage.

[0023] In some embodiments, the gateway interface includes a plurality of ports, each port connected to a corresponding second gateway of the plurality of gateways.

[0024] In some embodiments, the accelerator and the gateway form part of a common synchronization area and are each configured to generate a synchronization confirmation in response to receiving an indication of a precompiled data synchronization point when a corresponding condition is met.

[0025] According to a second aspect, a system on chip is provided, comprising one or more gateways according to the first aspect and at least one accelerator connected at an accelerator interface.

[0026] According to a third aspect, there is provided a method of operating a gateway according to the first aspect, the method comprising: a streaming engine executing instructions to selectively transmit batch data to and from the gateway via a selected one of the interfaces. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] For a better understanding of the invention and to show how it may be carried into effect, reference will now be made, by way of example, to the accompanying drawings, in which:

[0028] Figure 1 is a schematic block diagram of a processor chip including a plurality of blocks;

[0029] Figure 2 It is a schematic diagram of the Bulk Synchronous Parallel (BSP) computing model;

[0030] Figure 3 This is another schematic diagram of the BSP model;

[0031] Figure 4 is a schematic diagram of a synchronization request / acknowledgement exchange between an accelerator and a gateway;

[0032] Figure 5 is another schematic diagram of a system with multiple processor chips;

[0033] Figure 6 schematically illustrates the program flow involving synchronization with a host;

[0034] Figure 7A system including an accelerator, a gateway, and a host is schematically illustrated;

[0035] Figure 8 is a schematic diagram of the different data paths through the gateway;

[0036] Fig. 9 schematically illustrates the aggregation of accelerators and hosts using a gateway;

[0037] Fig.10 is a schematic diagram of the data flow through the gateway;

[0038] Fig.11 is a schematic diagram of a system including an accelerator, a gateway, and a host;

[0039] Fig.12 is a schematic diagram of a machine including multiple accelerators and gateways;

[0040] Fig.13 is a schematic diagram of a cluster (pod) including multiple machines;

[0041] Fig.14 illustrates example methods of deploying and computing data; and

[0042] Fig.15 is a schematic diagram of the exchange of synchronization requests and acknowledgments between three gateways. DETAILED DESCRIPTION

[0043] Embodiments of the present application relate to a gateway used in a computer system to connect a host to a subsystem that acts as a working accelerator for the host. The gateway has an accelerator interface for connecting to the subsystem so as to enable the transmission of batch data between the subsystem and the gateway. The gateway also has a data connection interface for connecting to an external storage (such as a host or a network attached storage) so as to exchange data between the gateway and the external storage. The gateway also has a gateway interface for connecting to at least one second gateway. The gateway also has a memory interface connected to a local memory associated with the gateway. The gateway also has a streaming engine for controlling the streaming of batch data entering and leaving the gateway in response to a precompiled data exchange synchronization point reached by the subsystem, wherein the streaming of the batch data is selectively via at least one of the accelerator interface, the data connection interface, the gateway interface, and the memory interface.

[0044] For example, the streaming engine can cause data transfer to occur between a gateway and a subsystem, for example, to transfer data to be processed by an accelerator and receive the results. The streaming engine can cause data transfer to occur between a gateway and external storage, for example, to receive data to be processed from a host. The streaming engine can cause data transfer to occur between a gateway and a second gateway. The data transfer between two gateways can be used to transfer data between accelerators or to transfer data to other elements of the system.

[0045] The streaming engine enables data transfer to occur between the gateway and local storage to store or retrieve data to be processed by an accelerator, results to be sent to external storage, or data to be sent to another gateway.

[0046] Providing such a gateway having a streaming engine for streaming batch data between different entities in the system enables expansion of the accelerator system by allowing different elements of the subsystem to communicate batch data to each other at appropriate points in time. For example, communication between two gateways can enable data transfer between different accelerators, so that data arrives at the accelerator at a pre-compiled data exchange synchronization point. Using the pre-compiled data exchange synchronization point to control streaming between these accelerators can ensure that data is delivered to the relevant accelerator at the correct time point expected by the accelerator.

[0047] In an embodiment of the present application, 128 subsystems acting as accelerators are connected together to form a fully expanded artificial intelligence (AI) cluster. Since each subsystem provides 120TF (120 trillion calculations per second), the resulting combination of collaborative subsystems produces a 16PFlop (16 quadrillion calculations per second) system. Therefore, the gateway that enables system expansion greatly improves processing power.

[0048] The following description explains various embodiments of the present application in more detail. The present application relates to a subsystem for acting as a work accelerator for a host system, and a combination of multiple such subsystems. The subsystem acts as an accelerator to perform predetermined processing steps on a data set (work) assigned by a host, where the host is running a process that requires a large amount of data to undergo mainly repeated processing. Each subsystem can be a so-called intelligent processing unit (IPU) or any class of accelerators (XPU). The technology described herein can be used with the IPU described in the earlier U.S. application No. 15 / 885925 (the content of the application is incorporated herein by reference), but can also be applied to any accelerator. As will be described in more detail, several accelerators can be combined to form an accelerator machine or equipment. Several accelerator equipments can be combined in a chassis. Multiple chassis can be organized in groups and can be arranged in a rack. The resulting combination of accelerators can produce a system with a large amount of processing power for performing parallel operations. This is particularly useful for implementing neural network processing in artificial intelligence applications. The principles listed here can also potentially be used to expand beyond a single rack.

[0049] The present application relates to a novel gateway that has many advantages in improving the efficiency of such accelerators. The gateway(s) allow the accelerator to be separated from the one or more host systems that provide the data sets processed by the accelerator. This has several benefits. First, it allows the number of accelerators per host to be configurable by the user, and the number can be increased to exceed the physical capacity of the host. Second, it allows the accelerator I / O to be decoupled from the host, which enables the I / O capacity to scale as a function of the number of accelerators. Third, the separation enables multiple hosts to use a collection of accelerator resources that are allocated and grouped on demand to the host through a well-defined API that supports lifecycle management of these resources and the associated hosts.

[0050] Each accelerator may be a single-chip processor. Figure 1A single-chip processor 2 is shown, i.e., a single wafer, which includes multiple processor blocks 4 and an array 6 of on-chip interconnects 34 that connect between the blocks 4. The chip 2 can be implemented alone on its own single-chip integrated circuit package, or as one of multiple wafers packaged in the same integrated circuit. The on-chip interconnect may also be referred to as a "switch fabric" 34 herein because it enables the blocks 4 to exchange data with each other. Each block 4 is a processing unit capable of executing instructions (code) from a local instruction memory and processing data in a local data memory. The blocks 4 may include corresponding instances of barrel thread processing units 10 and memories 11. For example, for example, the chip 2 may include hundreds of blocks 4, or even more than a thousand. For the sake of completeness, it is also noted that the "array" referred to herein does not necessarily mean any particular number of dimensions or physical layout of the blocks 4.

[0051] Each chip 2 also includes one or more external links 8 that enable the chip 2 to connect to one or more other external processors on different chips (e.g., one or more other instances of the same chip 2). These external links 8 can act as chip-to-chip links for connecting with one or more other instances of the chip 2 on the same IC package or on a card or on different cards. Multiple instances of the chip 2 can be connected together to form a card (as described later) via chip-to-chip links. Fig.12 ). The chip also has a connector 9 that connects the chip to a gateway, which will be described in detail later. Note that not all accelerators need to have a gateway connector 9, but for the purposes of the description herein, at least some accelerators have a gateway connector 9. In one example arrangement, chip 2 receives work assigned by a host from a gateway in the form of input data to be processed by chip 2. Note that reference to a host may alternatively mean a reference to an off-chip storage system, such as a network attached storage (NAS). The gateway enables data from a host or NAS to be provided to one or more accelerators, which are designed as a single-chip processor 2 or multiple single-chip processors 2, which may be arranged on multiple interconnected cards. The gateway implements relaying and separation between the accelerator and the host, as described later.

[0052] The interconnect 34 is configured to enable different processor blocks 4 in the array 6 to communicate with each other on the chip 2. In the IPU described in the earlier patent application, communication between blocks 4 on the accelerator 2 occurs in a time-determined manner. However, other forms of inter-block exchanges are also possible. Dependencies may exist between parts of a program running on different blocks 4 in the array 6. That is, the processing of data on one block may depend on results from another block, for example, the block may provide results that another block depends on. Therefore, a technique is needed to prevent a piece of code on one block 4 from running before the data it depends on and available through another piece of code on another block 4.

[0053] Parallel programming models for AI and data science typically follow a three-phase iterative execution model: compute, barrier, and exchange. This means that data transfers to and from accelerators typically rely on barriers to provide data consistency between accelerators and between each accelerator and the host. Commonly used data consistency models are Bulk Synchronous Parallel (BSP), Stale Synchronous Parallel (SSP), and Asynchronous.

[0054] In SSP, faster worker threads among multiple worker threads are allowed to run ahead of the slowest worker thread by several clock cycles. Worker threads can see updates to shared parameters with a range of timestamps. For example, a worker at clock t can see all updates from workers up to the update timestamped at t-Δ. BSP is a special case where Δ=0, and thus workers may not run ahead of each other.

[0055] In the asynchronous data consistency model, shared parameters can be read and / or written at any time.

[0056] The embodiments of the invention described herein use the BSP model, but it will be apparent that other synchronization models may be utilized instead.

[0057] Reference Figure 2 and Figure 3 , which shows an implementation of a BSP exchange scheme, wherein each block 4 performs a computation phase 33 and an exchange phase 32 in alternating cycles, separated from each other by a barrier synchronization 30 between the blocks. Figure 2 and Figure 3In the case shown, a barrier synchronization is placed between each calculation phase 33 and the subsequent exchange phase 32. During the calculation phase 33, each block 4 performs one or more computational tasks locally on the block, but does not communicate any results of these operations with any other block 4. During the exchange phase 32, each block 4 is allowed to exchange one or more computational results from the previous calculation phase to one or more other blocks and / or exchange one or more computational results from the previous calculation phase from one or more other blocks, but does not perform any new operations until it has received any data on which its (one or more) tasks depend from other blocks 4. It also does not send any data other than the data calculated in the previous calculation phase to any other block. It is not excluded that other operations such as internal control-related operations may be performed during the exchange phase. It should also be noted that during the calculation phase 33, the block 4 performing the operation may be allowed to communicate with a gateway outside the array of the block 4 being synchronized - as long as this does not involve communication with other blocks 4 within the group being synchronized. Communication outside the blockset may optionally utilize the BSP mechanism, but may alternatively not utilize the BSP and may use some other synchronization mechanism of its own.

[0058] According to the BSP principle, a barrier synchronization 30 is placed at the handover from the calculation stage 33 to the exchange stage 32, or at the handover from the exchange stage 32 to the calculation stage 33, or at the handover of the two. That is, (a) before any block 4 in the group is allowed to enter the next exchange stage 32, all blocks 4 are required to complete their respective calculation stages 33, or (b) before any block 4 in the group is allowed to enter the next calculation stage 33, all blocks 4 in the group are required to complete their respective exchange stages 32, or (c) both conditions are implemented. In all three variants, it is the individual processors that alternate between the different stages, and it is the entire component that is synchronized. Then, the sequence of exchange stages and calculation stages can be repeated through multiple repetitions. In BSP terminology, each repetition of the exchange stage and the calculation stage is sometimes called a "superstep" (although it should be noted that the term is not always used consistently in the article: sometimes each individual exchange stage and calculation stage is individually referred to as a superstep, while in other places, as in the terminology adopted in this article, the exchange stage and the calculation stage are collectively referred to as supersteps).

[0059] Note also that it is not excluded that groups of multiple different independent tiles 4 on the same chip 2 or on different chips may each form separate respective BSP groups that operate asynchronously with respect to one another, wherein the BSP cycles of computation, synchronization, and exchange are applied only within each given group, but each group operates independently of the other groups. That is, the multi-tile array 6 may include multiple internal synchronization groups, each of which operates independently and asynchronously with respect to other such groups (discussed in more detail later). In some embodiments, there is a hierarchical grouping of synchronization and exchange, which will be discussed in more detail later.

[0060] Figure 2 The BSP principle is shown implemented in a group 4i, 4ii, 4iii of some or all of the tiles in the array 6, in which case: (a) a barrier synchronization (see above) from the calculation stage 33 to the exchange stage 32 is applied. Note that in this arrangement, some tiles 4 are allowed to start calculation 33 while some other tiles 4 are still being exchanged.

[0061] According to embodiments disclosed herein, this type of BSP may be facilitated by incorporating additional and dedicated specialized functionality into machine code instructions for performing barrier synchronization (i.e., synchronization instructions). The synchronization instructions may be executed on a processor of a tile to initiate a swap phase in which data is swapped to synchronize data stored in the memory of the tile.

[0062] The synchronization instructions have operands that define a synchronization mode. One such mode is the inter-block mode synchronization chip, which causes all blocks on the chip to implement synchronization barriers for data exchange. This is managed by the compiler when compiling the instructions for each block, as each block is executed according to a predetermined time-based compilation protocol.

[0063] As described above, several accelerators (eg, IPUs) may be combined to produce an accelerator machine 161 having improved processing capabilities compared to a single accelerator. Fig.12 As shown. The accelerator machine 161 includes a plurality of (four in this example) accelerators 162 connected in an array, each accelerator 162 connected to its neighbor via a link 8. The machine 161 also includes two gateways 163, which are configured to connect the machine 161 to one or more hosts (not shown). Each gateway 163 is connected to two of the four accelerators 162 via a gateway link 9.

[0064] As will be explained in further detail, after the data exchange synchronization point, the gateway 163 is able to exchange data with the accelerator 162 connected thereto in the exchange phase. The data exchange synchronization point is triggered as a result of executing a synchronization instruction, which is part of the precompiled code running on the accelerator. At the beginning of the data exchange synchronization point, the synchronization instruction can be executed on the processor of the block. The execution of one or more synchronization instructions by one or more blocks of the accelerator 162 results in one or more synchronization requests issued by the one or more blocks. The accelerator 162 aggregates these synchronization requests and then issues the aggregated synchronization request to its associated gateway 163. The gateways can be connected to send synchronization signals between them so that a synchronization area can be formed by multiple gateways and accelerators. One function of the synchronization signal is to promote the data exchange between the gateway 163 and the associated accelerator 162 during the exchange phase of the BSP model, but they have other data-independent applications. Each gateway 163 has a local memory and is configured to obtain (from a host, from a remote storage, or from another gateway) data to be sent to the accelerator at the data exchange synchronization point and store it. The data is stored in local memory prior to a synchronization request from the accelerator 162 in preparation for its transfer to the accelerator. One function of the gateway is to provide the requested data to the accelerator when the accelerator needs it. The gateway can obtain data from the host or remote storage through different mechanisms, as discussed later.

[0065] Each gateway 163 is also configured to exchange data with other gateways. Gateway 163 can distribute copies of data to be sent to accelerator 162 to other gateways. These other gateways 162 can then distribute the data to the accelerators 162 to which they are connected. Therefore, other gateways 162 that receive copies of the data do not need to independently obtain the data from storage (e.g., a host or remote storage), thereby preventing multiple gateways from retrieving redundant data from storage. This will be described in detail later. In addition, as will be described in more detail later, gateway 163 is configured to be able to perform multiple different types of data transfer. Gateway 163 is configured to exchange data with other gateways. Gateway 163 is configured to exchange data with one or more accelerators 162 to which it is coupled. Gateway 163 is configured to exchange data with one or more hosts (not shown).

[0066] Reference Figure 4 , which shows an example of how the synchronization request / confirmation mechanism works in the case where one or more blocks 53 of the accelerator 51 issue a request for synchronization to the gateway 52.

[0067] The gateway 52 includes a register 59, which includes an indication of a synchronization region for an upcoming synchronization to be performed. The register 59 can be implemented in a shared register block (SRB) of the gateway 52. ​​Before the barrier synchronization, the block 53 of the accelerator 51 is configured to send an indication 32 of the synchronization region to which it belongs for the upcoming synchronization. Since many blocks 53 of the accelerator 51 may belong to the same synchronization region, the accelerator 51 specifies a block belonging to a specific synchronization region to write the indication 32. The synchronization region indicates which blocks will participate in the synchronization together. In some cases, the synchronization region may only include blocks 53 on the same chip, in which case it should be understood that the gateway is not involved. In other cases, the synchronization region may be an external synchronization including blocks 53 on different chips. In some cases, the synchronization region includes blocks on different accelerators. In some cases, the synchronization region includes a gateway, a host and / or a remote storage.

[0068] Although the indication of the synchronization area is presented here as being sent from the accelerator 51 to the gateway 52, in some other embodiments, the indication may be determined by the gateway 52 and stored in the register 59. The gateway 52 may make this determination autonomously based on its precompiled code. In some other embodiments, the indication may be provided as part of the synchronization request 56 received from the accelerator 51, or as part of the synchronization information provided out of band (e.g., PCIe write) before asserting the synchronization request.

[0069] The data exchange synchronization point is triggered as a result of precompiled synchronization instructions in the code running on the blocks 53 of the accelerator 51. At the beginning of the data exchange synchronization point, one or more synchronization instructions can be executed on the processors of one or more blocks 53. Each block that has executed the synchronization instructions sends a synchronization request, which is received at the synchronization logic 54 of the accelerator 51. The synchronization logic 54 aggregates these synchronization requests 55 and sends the aggregated synchronization request 56 to the gateway 52.

[0070] The gateway 52 receives a synchronization request 56 from the accelerator 51 and may allow the synchronization barrier to be passed. This involves sending a synchronization confirmation 57 to the accelerator 51 in response to the synchronization request 56. Allowing the synchronization barrier to be passed enables the blocks 53 of the accelerator 51 to exchange data with each other, and in some cases, allows data to be exchanged with the gateway 52 itself. Data exchange with the gateway 52 may involve transmitting data received from a host (not shown) at the gateway 52 to one or more blocks 53 of the accelerator 51. Data exchange with the gateway 52 may involve transmitting data received from another gateway (not shown) at the gateway 52 to one or more blocks 53 of the accelerator 51. The data received from another gateway may originate from another accelerator. This is a mechanism for implementing data exchange between accelerators via a gateway. The data received from another gateway may originate from another host. Another mechanism is to enable an accelerator connected to a gateway to directly write to another accelerator connected to another gateway via a structural port between gateways through the facilities of the gateway. In order to implement the above mechanism, all storage locations in each grouped accelerator / gateway (i.e., chassis / group / rack, etc.) form part of a single global address space.

[0071] The gateway 52 has three data exchange boundaries: (i) gateway-accelerator; (ii) gateway-external; and (iii) gateway-gateway. These boundaries have different requirements and are therefore governed by different protocols. However, they must be coordinated so that the accelerator 51 data is available in the gateway memory when requested by the accelerator 51 (i.e., when synchronized), but the gateway memory storing data for the gateway 52 does not overflow.

[0072] As described above, before synchronization, an indication of the synchronization area of ​​a group of blocks 53 of the accelerator is stored in a register 59. In some embodiments, a write 50 is made to the register 59 before a synchronization request 56 is issued to the gateway 52. ​​The block may send an indication at the end of the previous exchange phase or at the beginning of a calculation step before the exchange phase in which the corresponding synchronization is to occur. A separate write 50 to the register 59 is performed for each synchronization barrier. Upon receiving a synchronization request 56, the gateway 52 is configured to consume the indication corresponding to the synchronization request from the register 59. The gateway 52 is configured to send a confirmation 57 of the synchronization request to the accelerator 51 only if the indication corresponding to the synchronization request 56 has been written to the register 59. In other words, the gateway 52 will send a confirmation 57 of the synchronization request to the accelerator 51 only if the value has been refreshed since the last barrier.

[0073] If there is a delay in writing the indication of the synchronization area to the register 59 (for example, because one or more blocks 53 of the accelerator cannot determine their synchronization area until the calculation phase ends), the synchronization request may be received before the register is updated with the corresponding indication of the synchronization area. In this case, the gateway 52 waits until the register 59 receives the corresponding indication of the synchronization area before sending the confirmation 57. Therefore, the system may experience a small delay while waiting for the register 59 to be refreshed.

[0074] The gateway 52 generates a synchronization confirmation 57 and sends it to the correct block, chip and / or accelerator using the indication of the synchronization area stored in the register 59. For example, if the indication of the synchronization area is that the synchronization area includes the accelerator 51 and additionally includes another accelerator (not shown), the gateway 52 sends a synchronization confirmation to the accelerator 51 and the other accelerator in response to receiving the synchronization request. The gateway 52 can read the indication of the synchronization area from the register 59 and propagate the synchronization confirmation or request 57 accordingly according to the indication.

[0075] The indication of the synchronization area stored in the register 59 includes an indication of whether data transfer from the gateway 52 itself is required as part of the synchronization. The indication may be implicit in the indication of the synchronization area stored in the register 59. If the gateway 52 determines that data transfer is required, the gateway 52 applies a credit control mechanism to determine whether to allow the synchronization barrier to be passed. If the gateway 52 determines that data transfer is not required, the gateway 52 sends a synchronization confirmation 57 to the accelerator 51 without applying the credit control mechanism. For the credit control mechanism, if there are one or more first credit sets (referred to as exchange synchronisation point (ESP) credits) available in the storage of the gateway 52 (Local Sync Barrier Module (LSBM) described later), the gateway 52 is configured to synchronize the data from the gateway storage (not in the LSBM)) by sending the synchronization confirmation 57 to the accelerator 51. Figure 4 ) is transmitted to the accelerator 51 in response to the receipt of the synchronization request 56 to allow the synchronization barrier to be passed. If the available ESP credit is zero, the gateway 52 will not acknowledge 57 the synchronization request 56, and will not transfer the data from the gateway memory (not in Figure 4 ) is transmitted to the accelerator 51, thus causing synchronization to stop. This credit control mechanism allows the gateway 52 and the accelerator 51 to remain synchronized with each other in the BSP protocol, which will be described in more detail below.

[0076] In some embodiments, the gateway 52 and the accelerator 51 each include pre-compiled code, allowing the gateway 52 to provide the required data to the accelerator 51 at the correct time.

[0077] After the synchronization logic 54 of the accelerator 51 has sent the synchronization request 56, the synchronization logic 54 will wait for the synchronization confirmation (sync_ack) 57 from the gateway 52. ​​When the synchronization logic 54 of the accelerator 51 receives the synchronization confirmation 57 from the gateway 52, it will return the synchronization confirmation signal 57 (sync_ack) to the block 53 that issued the synchronization request 55. All the blocks 53 of the synchronization request will automatically pause until the synchronization confirmation 58 (sync_ack) from the external synchronization logic 54 is returned. In response to the synchronization confirmation 58, the blocks 53 resume the issuance of instructions for the supervisor, that is, they re-enter the calculation phase.

[0078] The actual data (content) can be sent between the accelerator block 53 and the gateway 52 through different channels of synchronization requests 55 / 56 and synchronization confirmations 57 / 58. In addition, it will be understood that given the description of the synchronization and aggregation functions disclosed herein, the technician will be able to build different types of circuits to implement the disclosed functions. For example, the synchronization logic 54 can use dedicated lines to send synchronization requests 56 and synchronization confirmations 57 / 58. As an alternative to dedicated lines, the synchronization logic 54 can additionally use data packets carried by the interconnect. For example, each synchronization request 55 / 56 and / or synchronization confirmation 57 / 58 can be sent in the form of one or more data packets. For example, each synchronization request 55 / 56 and / or synchronization confirmation 57 / 58 can be sent in the form of one or more data packets.

[0079] Reference Figure 5 , which explains the concept of synchronization regions in more detail. Figure 5 As shown, in an embodiment, the indication of the synchronization area written to the register 59 of the gateway 52 can be used to specify one of a plurality of different possible external synchronization areas, such as area_1 or area_2. In an embodiment, these areas correspond to different levels. That is, each higher level 92 (e.g., area 2) surrounds two or more areas 91A, 91B of at least one lower level. Fig. 9 For example, the two leftmost gateways and the accelerator may have a sync region 0, where one of the two gateways is the master. Likewise, the two rightmost gateways and the accelerator may have a sync region 0, where one of the two gateways is the master. Then there may also be a sync region 1, which is the entire schematic (and any arbitrary gateway may then be designated as a sync master).

[0080] Programs may then use several levels of synchronization:

[0081] 1. Only intra-accelerator synchronization - blocks on the same accelerator can be synchronized

[0082] 2. IPU+Gateway (data) sync only - a single accelerator requests synchronization from its gateway (e.g., to coordinate the exchange of data)

[0083] 3. Leftmost sync region 0 (with or without credit at each gateway)

[0084] 4. Rightmost sync region 0 (with or without credit at each gateway)

[0085] 5. Synchronize region 1 (with or without credit at each gateway)

[0086] The indication may indicate the participation of the gateway in synchronization (i.e., data will be transferred between the gateway 52 and the accelerator 51). The indication may indicate the participation of another gateway other than the gateway 52, with which the accelerator 51 may communicate via the gateway 52. ​​Thus, when the corresponding synchronization instruction is executed, the block 53 executing the synchronization instruction will be synchronized with the host 63 via data transfer with the gateway 52. ​​Where the participation of another gateway is indicated, a synchronization request from the accelerator 51 (after the synchronization request is aggregated with other synchronization requests received at the gateway 52) may be passed upstream to reach the other gateway. The gateway 52 waits for a synchronization confirmation from the other gateway before providing a synchronization confirmation to the accelerator. This will be referred to later. Figure 8 Describe this scenario in more detail.

[0087] In response to the indication in register 59 indicating an external synchronization region, the gateway 52 sends a synchronization confirmation 57 to the accelerator in the external synchronization region. The dedicated hardware synchronization logic 54 in the accelerator receives the synchronization confirmation (sync_ack) 57 from the gateway and sends a synchronization confirmation 58 to the indicated group of blocks 4. The synchronization logic 54 returns a synchronization confirmation signal 58 (sync_ack) to the block in the signaled synchronization region only when the synchronization request (sync_req) 55 has been received from all blocks 4 in the region (but if the synchronization logic 54 is not globally synchronized, it will not wait for any other blocks outside the region).

[0088] Note that in other embodiments, the synchronization region that can be specified by the indication in register 59 is not limited to being hierarchical in nature. In general, the indication in register 59 can be provided with a pattern corresponding to any type of grouping. For example, the pattern can be implemented to select from a mixture of only non-hierarchical groups, or hierarchical groups and one or more non-hierarchical groups (wherein at least one group is not completely nested in another group). This advantageously enables a programmer or compiler to flexibly select between different layouts of internally synchronized groups with minimal code density, wherein the internally synchronized groups can run asynchronously with each other until more extensive synchronization is required.

[0089] As mentioned previously, some synchronization barriers involve synchronizing the accelerator's blocks with data from the host, while some do not. Figure 6 An example of a global synchronization region 92 is schematically shown. The system is allowed to execute N supersteps, thereby passing N synchronization barriers 80, before applying a barrier 90 that also requires synchronization with the host 63. At the synchronization barrier 90 with the host 63, the data (which has been transferred from the host 63 to the gateway 52) is transferred from the gateway 52 to the accelerator 51. The N synchronization barriers require synchronization requests from all (non-abstaining) blocks 4 in the relevant synchronization group 92, but do not require synchronization requests from the host 63. Subsequent synchronization barriers 80 require synchronization requests from all (non-abstaining) blocks 4 in the synchronization group 92. In addition, in order to pass the synchronization barrier 80, the gateway is required to store a sufficient number of ESP credits to pass the specific barrier. After the barrier 90, an exchange 50" can be performed between the gateway and one or more blocks 4, for example, so that one or more blocks 4 report the results of the operation to the host 63.

[0090] Now refer to Figure 7 , Figure 7 How the host 63 interacts with the accelerator 51 and exchanges data is shown in more detail. The host 63 is configured to provide data for processing by the accelerator 51. The accelerator 51 is configured to process the data and deliver the results of the processing to the host 63. The gateway 52 is responsible for streaming data between the host 63 and the accelerator 51 in a managed manner in order to exchange data. In this example, the accelerator 51 can be an IPU as described above with reference to the previous figures. However, the gateway 52 can be used to connect the host 63 with other types of accelerators 51.

[0091] Data synchronization between the host 63, the gateway 52, and the accelerator 51 by exchanging synchronization points ensures the consistency of the gateway data and the readiness of the I / O operations. The availability of data between the gateway 52 and the accelerator 51 is handled via a credit mechanism of ESP credits. One credit is allowed to pass through one ESP. Before the ESP, the preparation of the gateway memory 114 is handled by executing "pre-work" instructions by the gateway 52. ​​Data processing after the ESP is performed by executing "post-work" instructions. The PPE execution engine 123 described later executes the pre-work and post-work instructions.

[0092] like Figure 7 As shown (and also refer to Figure 5), the gateway 52 includes at least one "local synchronization propagation module" (LSPM) 117 and at least one "local synchronization barrier module" (LSBM) 118. The LSBM 118 can be considered as a proxy for the PPE, and it enables programs running on the accelerator to process batches of data to be decoupled from the host. The accelerator 51 / gateway 52 synchronization can run asynchronously with the activity of the host 63 providing data to the gateway 52. ​​The LSBM 118 is configured to store the ESP credits discussed above. The LSBM is accessible to the LSPM 117.

[0093] The LSBM 118 comprises hardware circuitry configured to enable the host 63 to participate in a corresponding synchronization group 92, wherein the LSBM 118 is arranged to act as a proxy for the PPE. If the synchronization request 56 issued by the block 4 is a synchronization involving a gateway, it will use both the LSPM 117 and LSBM 118 of the gateway 52, while a synchronization request 56 for synchronization that does not involve data transfer between the gateway 52 and the accelerator 51 will be received by the LSPM 117 and returned to the requesting block without involving the LSBM 118. Therefore, the blocks 4 determine when (if ever) the accelerator 51 needs to interact with the gateway via the LSBM 118 based on the programs they execute.

[0094] If the accelerator 51 needs to interact with the gateway, the LSBM 118 is then configured to allow the sync barrier to be passed upon receiving the sync request 56, provided that the number of ESP credits is greater than zero. Allowing the sync barrier to be passed involves generating a sync confirmation (not shown) and sending the sync confirmation to the accelerator 51.

[0095] As described above, the gateway 52 stores a set of credits associated with the interface between itself and the accelerator 51. These credits are referred to in the description as exchange synchronization point (ESP) credits. However, those skilled in the art will appreciate that this name is only used to conveniently identify the credits and is not meant to limit the nature of the credits. ESP credits may also be referred to as barrier credits because they control whether data exchange operations can be performed on a barrier.

[0096] If the number of ESP credits in LSBM 118 is zero, then upon receipt of a synchronization request 56 and a corresponding indication in register 59 that a data transfer with the gateway is required, LSPM 117 does not allow the synchronization barrier to be passed and therefore does not allow blocks 4 in group 92 to continue running again until the number of ESP credits is greater than zero. The generation of ESP credits may be achieved when data for transfer to accelerator 51 at the exchange synchronization point becomes available in gateway 52. ​​In some cases, the data may become available due to being sent from host 63 or network attached or other external storage. In other cases, the data may become available due to being transferred from another gateway. The data received from another gateway may be data from another accelerator or another host or remote storage.

[0097] In some embodiments, the gateway 52 may maintain multiple ESP credit sets. There may be different credit sets for different synchronization groups. In this case, a synchronization request 56 corresponding to one synchronization group may cause the gateway 52 to confirm the request (if the number of ESP credits for the group is not zero), while a synchronization request 56 corresponding to another synchronization group may not cause the gateway 52 to confirm the request (if the number of ESP credits for the group is zero). There may also be different credit sets for different accelerators configured to communicate with the gateway 52. Fig.12 As shown, each gateway 163 is configured to communicate with two accelerators 162, and therefore, the gateway 52 can store two ESP credit sets for each accelerator 162. If each accelerator 162 has two possible synchronization groups that require gateway data transfer, this results in each gateway 163 storing a total of four credit sets.

[0098] Blocks 4 of the sync group may be allowed to proceed through the N sync barriers (where the sync request is forwarded to and acknowledged by LSPM 117) without any delay in reaching the gateway, after which they must synchronize with the gateway via LSBM 118 (and data may then be exchanged to and / or from the gateway). See, for example, Figure 6 .

[0099] As described above, the software running on block 4 is programmed to request synchronization with the gateway by sending an indication of whether the gateway participation is required for synchronization (which indication can be included in the synchronization request or sent separately). This indication is stored in register 59 of the gateway 52. ​​In such an embodiment, LSBM 118 applies the above-mentioned credit control mechanism only to barriers corresponding to synchronizations marked as requiring gateway participation (the gateway's "participation" for any given barrier is a proxy authorization (LSBM) by LSPM 118 to perform a synchronization confirmation (sync ack) on behalf of the host, or occasionally explicitly granting more ESP credits to LSBM 118). In an embodiment, gateway participation is selected by different variants of the synchronization zone indication stored in register 59. That is, for each synchronization group 91, 92, the synchronization zone indication can actually take two variants: zone_1_host, zone_1_no_host; and zone_2_host, zone_2_no_host. The execution units of the block are configured to cause the synchronization logic 54 to signal the gateway participation flag accordingly. However, in other embodiments, it is not excluded that other mechanisms for requesting gateway participation may be implemented, or that gateway participation is even hardwired and therefore always imposed.

[0100] In an embodiment, preparation for the barrier performed by the gateway may include preparing data to be obtained by the accelerator 51, such as an experience data set required by the accelerator 51 in the next stage of learning a model. In this context, preparation may include obtaining data from a storage disk or other medium, formatting data in a format required by a training algorithm running on the accelerator 51, or decompressing image data. In addition, preparation for the barrier may include consuming output data generated by the accelerator 51. As discussed later, some or all of the above preparations may be performed at the gateway 52. ​​The gateway 52 is at least located in the path between the storage disk or other medium and the accelerator 51.

[0101] Sync requests 56 to LSBM 118 may be delivered from a processing element as network or (PCIe) packets, and / or sync acknowledgements 57 may be returned as network (or PCIe) packets. In general, a gateway may participate in any one or more levels of synchronization.

[0102] In general, the concept of ESP credits can be applied to any multi-block architecture, not just the example architecture disclosed herein. It is also not necessarily limited to the BSP application context. The disclosed technology has particular synergy with systems that employ a single rendezvous point (such as, BSP), or when the number of different rendezvous points between the host or other outside world system and the machine in question is limited to only one rendezvous point or a very small number (relative to, say, CSP). Nevertheless, the applicability of the present disclosure is not absolutely limited to this aspect. In any system or application, latency reduction can be achieved by enabling blocks to pass through a specified number of synchronization barriers without involving a gateway, thereby reducing the number of times the multi-block subsystem must interact with the gateway and thus reducing the number of latency penalties incurred by interacting with the gateway.

[0103] Furthermore, although the embodiments have been illustrated in terms of a PCIe interface between cards or with the host 63, this is not limiting and other types of interfaces may be used, such as Ethernet.

[0104] Furthermore, embodiments are not limited to synchronous communication between the host system 63 and the accelerator 51, which may otherwise operate asynchronously. In embodiments, the gateway 52 may be used for synchronization between two independent BSPs or other parallel processing subsystems that operate synchronously internally but asynchronously with respect to each other. The gateway 52 allows the size of the synchronization group to be increased to larger sizes and implements a more efficient tree structure for those larger groups.

[0105] The batches of data received at the gateway 52 are stored in the memory 114. The memory 114 is a local memory (e.g., DRAM) that is reserved for use by the gateway 52. ​​In response to the synchronization request 56, the gateway 52 can retrieve the data from the memory 114 and transmit it to the accelerator 51. Path 116 shows the flow of each batch of data. Note that each batch of data is kept in the memory 114 for a period of time, which can vary from batch to batch. This depends on the time when the batch enters the gateway 52 and the time when it is pushed to the accelerator 51, and these times are not necessarily related.

[0106] LSPM 117 may be configured to indicate to gateway 52 the timing of transferring data from memory 114 to accelerator 51 or vice versa. This allows LSPM 117 to specify the proper timing of deploying data from accelerator 61 to memory 114 in order to prevent overflow of gateway memory 114.

[0107] Additionally, the flow of data from the host / remote storage into the gateway memory 114 is managed so as to avoid overflowing the gateway memory 114 .

[0108] exist Figure 7 In the example, data for processing by the accelerator 51 is transferred from the host 63 to the gateway 52, which stores the data in the local memory 114. The data can be pulled by the gateway 52 via an RDMA read, or can be written by an RDMA write performed by the host 63 to the gateway 52.

[0109] Reference Fig.11 , which shows an alternative, in which the data 116 is retrieved from the network attached storage 151 by the gateway 52. ​​The network attached storage 151 is also referred to as remote storage in this article. Fig.11 In the present invention, similar reference numerals are used to represent Fig.11 components similar to the components of the

[0110] exist Fig.11 In the example, the host 63 sends a descriptor 119 to the gateway 52. ​​The descriptor 119 identifies the location of the network attached storage 151 accessible to the gateway 52. ​​When the data acquisition instruction is executed with reference to the descriptor 119, the gateway 52 retrieves the data 116 from the network attached storage 151. The gateway 52 then stores the data 116 in the memory 114 before transmitting the data 116 to the accelerator 51.

[0111] In some embodiments, rather than transmitting the descriptor 119 from the host 63 to the gateway 52, the descriptor is included in precompiled code stored by the gateway 52. ​​In this case, the gateway 52 autonomously retrieves data from the remote storage 151 without intervention from the host. In some examples of this application, the gateway 52 includes a system on chip (SoC) used as a stand-alone device, such that an external host 63 is not required. The entire application stack runs directly on the SoC, or on an SoC in a broader system. The gateway 52 can be configured to operate in a first mode and a second mode, in which the gateway 52 interacts with an external host 63 processor in the first mode and in which such an external host 63 is not required in the second mode. The remainder of the gateway 52 (e.g., see Figure 8 The streaming engine described herein performs the same functions regardless of which of these modes the gateway 52 is configured to operate in.

[0112] Reference Figure 8 , which illustrates gateway 52 in more detail. Figure 8 The various paths that data takes through gateway 52 are shown.

[0113] Figure 8It is shown how data 120 for processing by the accelerator 51 is transferred from the host 63 or remote storage 151 to the memory 114. As already mentioned, in some examples, the data 120 is transferred from the host 63 to the gateway 52. ​​In other examples, the data 120 is received from the local or remote storage 151 in response to a read request made by the gateway 52 to the remote storage 151 (e.g., a network attached storage). The gateway 52 retrieves the data 120 from the remote storage 151 via RDMA. The data 120 is received via a data center port. In addition to retrieving data, the gateway 52 also writes data (not shown) to the host 63 / remote storage 151. Data writing is performed via a data center port. In the exchange phase, data can be transferred from the gateway memory 114 to the accelerator 51.

[0114] During the exchange phase, instead of or in addition to transferring data from the gateway memory 114 to the accelerator 51, data may be transferred from the accelerator 51 to the gateway 52. ​​The accelerator 51 is configured to send data to the gateway 52 in the form of data packets, each of which includes a header indicating an address. The gateway 52 uses the addresses of the data packets to determine where to send them. For example, the data packets may be stored in the local memory 114. The data packets may be sent to another gateway 128. The data packets may be dispatched to an accelerator connected to another gateway 128. The data packets may be sent to the host 63 / remote storage 151.

[0115] The data 120 passes through the gateway 52 to the memory 114 under the control of the streaming engine 124 (the streaming engine 124 is also responsible for retrieving the data 121 from the memory 114 to deliver it to the accelerator 51). The streaming engine 124 performs streaming operations on the data. These operations for batch data can be specified by a work descriptor (WD). The streaming engine 124 includes two execution engines and a code memory (not shown). One execution engine is a data mover engine (Data Mover Engine, DME) 122, and the other execution engine is a pre / post work engine (Pre / PostWork engine, PPE) 123. They execute instructions loaded into the code memory as an executable image, which is generated by the compiler. The streaming engine 124 has a set of work instructions for execution by the DME 122 and a set of work instructions for execution by the PPE 123. The instruction sets for the DME and PPE are coordinated by the WD and set at compile time. These instructions for a single data exchange synchronization point may be grouped together into a single WD. The DME 124 is operated by specific DME instructions found in the DME portion of the executable image. The DME 124 uses the WD to navigate to a set of data mover (DMOV) instructions associated with a given ESP. The PPE 123 is operated by specific PPE instructions found in the PPE portion of the executable image. The PPE 123 uses the WD to navigate to a set of pre / post work instructions associated with a given ESP.

[0116] Before data exchange with the accelerator 51, the pre-work of the PPE must be ready. Only after the exchange is completed can the post-work of the PPE in the WD begin. The data exchange is immediately performed after the synchronization request 56 is confirmed and signaled to both the accelerator 51 and the streaming engine 124. This request / confirmation signals the "Exchange Synchronization Point" (ESP).

[0117] The streaming engine 124 supports different data streaming models.

[0118] All models support a configuration that allows the host to strictly control the consumption of ESP credits. This supports coordination of I / O operations between the host 63, the gateway 52, and the accelerator 51, as well as a mechanism to stop the accelerator 51 when it is not needed by other accelerator-level I / O mechanisms that utilize the gateway memory 114. It can also be a mechanism for setting breakpoints or single-stepping through the complete structure of the accelerator. When running any model under strict flow control from the host 63, the ESP credits authorized by the host 63 are transferred to the "ESP Credit Register" (part of the LSBM 118) by the PPE scheduler. The ESP Credit Register can be read / written by the hardware and firmware of the gateway 52.

[0119] The first streaming model supported by the streaming engine 124 is called "Advanced Gateway (GW) Push". In Advanced GW Push, the PPE 123 streams data from / to external storage and the gateway (GW) memory 114, while the DME 122 pushes data to the accelerator 51. Execution is based on instructions from a compiled executable image saved by the gateway. The generation of executable images for the streaming engine 124 is integrated with the accelerator compiler. The compiler generates two related compiled code sequences or executable images. The first of the two compiled code sequences or executable images is executed on the accelerator 51, while the second is executed on the gateway. In some embodiments, the host 63 can provide the compiled code sequences to the accelerator 51 and the gateway 52.

[0120] The "gateway push model" is a usage model where the gateway 52 pushes data. This model differs from the "gateway pull model" (discussed later) in that it pushes data to the accelerator 51 at an agreed point in time (at an agreed ESP). This general push model can support different types of memory consistency protocols or bridge models for parallel programming. Examples include bulk synchronous parallelism (BSP), delayed synchronous parallelism (SSP), and asynchronous parallelism.

[0121] The advanced gateway (GW) push model uses a credit mechanism to control the availability of data input (relative to the accelerator) to be pushed, and the availability of the data buffer of the gateway 52 to which the accelerator 51 outputs data. The gateway 52 executes both data mover instructions (DME 122 is pushing data to the accelerator 51) and pre / post work engine instructions for transferring data with external nodes (host, NAS or other gateways). When the accelerator input data is not available due to external I / O bottlenecks, the PPE 123 is responsible for effectively stopping the DME 122 by losing ESP credits. A similar stop is also required when the output data of the accelerator (to the remote host / storage) accumulates in the memory 114 of the gateway 52 due to external I / O bottlenecks.

[0122] This model allows the gateway 52 to deliver data with lower latency because it allows data to be pre-fetched from the GW memory 114 into a high-speed gateway transfer memory 127 (such as SRAM) before the push to the accelerator 51 occurs. In addition, push is inherently a lower latency operation compared to pull because the pull operation requires a round trip. That is, the pull operation requires a read request and then the data is returned in response to the read request. On the other hand, push only involves the transmission of data.

[0123] Another advantage of the advanced gateway push model is that the accelerator 51 benefits from not spending valuable computing resources on pulling data, but instead offloading the data movement to the gateway's DME 122.

[0124] The memory consistency model (BSP, SSP, asynchronous, etc.) described above can be used in conjunction with the push model. The accelerator 51 runtime must then ensure that external barriers will trigger DME 122 and PPE 123 data movement. In the case of a push operation, the ESP credit will be decremented by one by the gateway 52.

[0125] The second streaming model is called advanced accelerator pull. In this streaming model, the PPE 123 streams data from / to external storage into the gateway memory 114. The accelerator 51 then pulls the data from the gateway 52 via (one or more) PCIe read operations. The execution of the PPE 123 is based on instructions from the executable image in the code memory.

[0126] In this model, DME 122 is disabled and does not perform the above operations. On the other hand, PPE 123 is active and obtains data from external storage by issuing a "pull request" (i.e., a read request) and stores it in memory 114. The accelerator 51 will then pull data from memory 114 at a predefined ESP. The advanced accelerator pull model utilizes an executable image containing pre- / post-work instructions without DMOV instructions. The host 63 synchronizes the accelerator 51 via the ESP credit mechanism so that it pulls valid data prepared in the gateway memory 114 at the intended ESP.

[0127] Likewise, for this model, when accelerator 51 input data is unavailable due to an external IO bottleneck, PPE 123 is responsible for stopping LSPM (via a credit mechanism). Similar stopping needs to be performed when accelerator 51 output data (to remote host / storage) accumulates in the gateway's memory 114 due to an external I / O bottleneck.

[0128] The third streaming model is referred to as simple accelerator pull. In this streaming model, the host 63 streams data into / out of the gateway memory 114. The accelerator 51 pulls data from the gateway 52 via (one or more) PCIe read operations. In this case, the gateway 52 does not execute PPE instructions, but instead acts as a slave of a predefined I / O scheme between the host 63 or NAS and the gateway 52.

[0129] In this model, the gateway memory 114 is used as a storage area, where the host 63 controls its contents. No instructions are executed in the gateway 52 to load the data in the memory 114. In addition, the DME 122 is disabled and does not execute instructions. The PPE 123 does not execute instructions, but still acts as a proxy to update the ESP credits provided by the host 63 to the accelerator 51 for discovering when data is available.

[0130] The gateway memory 114 allocated for streaming of data is maintained by the host 63 as if it were PCIe attached memory, the only difference being that RDMA is used instead of PCIe.

[0131] In the streaming push model described above, the gateway 52 hides the accelerator's memory access latency by using the gateway memory 114 as an on-chip streaming buffer. The overall benefit of the streaming engine 124 is that data movement can be overlapped with accelerator operations and preloaded into the gateway 52's memory 114 before the scheduled push operation. The advanced GW push model has an additional benefit in that it frees up accelerator resources that would otherwise be used for DMA operations.

[0132] The execution of the data streaming operation in the gateway 52 is performed by the streaming engine 124, which will run all or a subset of the instruction set of the gateway 52 according to the operation model. The instructions are loaded into the gateway memory 114 as an executable image. The generation of the executable image for the streaming engine 124 will be integrated with a specific accelerator / gateway compiler environment, in which the compiler generates relevant code for running on the accelerator 51 and the gateway 52.

[0133] As can be seen, the streaming engine 124 includes a collection of hardware and software components that work together to ensure that data I / O is supplied to the accelerator in a performance-optimal manner. Depending on the operating model of the gateway 52 or the streaming engine 124, the streaming engine 124 can push data in a "just in time" manner (i.e., at a planned data exchange phase that represents a conditional entry point to the next accelerator calculation step), or can make the data available in the gateway memory 114 so that the accelerator 51 can pull it in the same "just in time" manner. The preparation of relevant data in the gateway memory 114 before the data exchange phase is completed via pre-scheduled data streaming instructions executed by the gateway streaming engine 124. The push model can additionally pre-fetch data from the gateway memory 114 into the gateway transfer memory 127 (e.g., SRAM) to reduce delays during the data exchange phase. The concept of bringing data "just in time" into the gateway memory 114 is useful for situations where the gateway memory 114 is not large enough to store all the data required by the accelerator operation algorithm.

[0134] The PPE engine uses the WD to navigate to a set of pre-work (PRE) and post-work (POW) instructions associated with a given ESP. The terms "pre-work" and "post-work" indicate whether the operation occurs before or after the data exchange phase between the WD and the accelerator or other target. The primary responsibility of the PRW instruction is to bring data from the host 63 into the gateway memory 114 (e.g., from the host 63, remote storage 151, or from another gateway 128), or as preparation for one or more DMOV push instructions. The primary responsibility of the "post-work" is to move data out of the GW memory 114 (e.g., to reach the host 63 or remote storage 151). PPE instructions are located in the PPE-specific image portion.

[0135] DME 122 is active in the "gateway push" operating model as described above. In the push model, DME 122 uses WD to navigate to a set of data mover (DMOV) instructions associated with a given ESP. DMOV instructions push data to the accelerator. WD and DME-related instructions are located in the DME-specific image section. DME instructions originating from the image in the physical DDR memory of the gateway 52 are converted into a list of DMA descriptors that are executed by the DME's DMA machine as part of the DMOV instructions. DME 122 will prepare DMA descriptors for several planned data exchanges controlled by stop criteria that allow full control over the size of each batch of data exchanged with the accelerator 51.

[0136] DME 122 uses an advanced programmable multi-channel DMA machine designed to stream data into and out of accelerator memory. DME 122 supports streaming data over one or two high-speed data buses to a single accelerator 51 using load distribution. If the accelerator 51 is agnostic to the data load sequence, load distribution is implemented by local DME decisions and is not controlled by information found in the executable image.

[0137] When all pre-work related instructions for a WD and all post-work instructions with end criteria for that WD are completed, the WD is considered "ready to execute" (or fully ready). Only then will the ESP credits for the WD be added to the ESP credit collection in the LSBM 118.

[0138] When the "end of swap" criteria are met, the WD is considered "completed". This is when all deploy operations (DMOV) are completed and all output data received from the accelerator 51 is determined to be equal to the expected output size. The expected output size is indicated in the WD.

[0139] The gateway 52 needs a method for the PPE 123 to signal the DME 122 when the WD is fully ready, and this is accomplished by adding ESP credits to the DME 122 (also referred to as WD credits or exchange credits). A PPE 123 engine running several WDs ahead of the DME 122 is allowed to add several ESP credits. This avoids the accelerator having to wait for the PPE work to complete at each ESP. Ideally, at each ESP transition, ESP credits should already be available so that the barrier can be passed without stopping the accelerator.

[0140] One credit represents the ability of the DME 122 to transfer all data for the first data exchange with the accelerator 52. The PPE 123 increments the ESP credit by adding a new credit each time the PPE completes data prefetching (i.e., completes the previous work) for the next sequential ESP. If the PPE 123 does not complete preloading data from an external node for the ESP in time, the DME 122 will find that its ESP credit is zero and will stop executing until the PPE 123 increments the credit count. Stopping one accelerator 51 due to missing data will effectively stop the entire set of cooperating accelerators that are running synchronously (i.e., sharing the same barrier synchronization network).

[0141] Each DMOV instruction is executed in hardware by the DME 122 as a DMA operation. These DMOV instructions are executed when the gateway push model is applied. The DMOV instruction moves the data residing in the referenced data buffer (in the gateway memory 114) to its destination. The destination is typically a memory location of the accelerator 51, but other destinations are also supported.

[0142] Since the streaming of data is batched per ESP, the DME 122 will stop transmitting data when the required number of buffers from the gateway memory 114 has been transmitted. The number of bytes exchanged per ESP batch is indicated by the parameter field in the WD for both 1) streaming engine 124 push operations and 2) writing to the gateway memory 114. The number of bytes expected to be pushed is equal to the number of bytes in all buffers scheduled for the same WD. If there is a mismatch, this will result in an abnormal situation.

[0143] DME 122 is configured to retrieve data from memory 114 using physical memory addresses without the support of a memory management unit (MMU).

[0144] For an accelerator 51 with a dual bus attachment to a gateway 52, there is no information in the DMOV indicating which bus the data should be directed to. The DME 122 controls the selection of the bus in order to balance the traffic sent on the two buses.

[0145] The DMOV may be linked to a pre-initialized data buffer in the gateway memory 114 and therefore, in this case, no associated upfront work instructions are required to fill the buffer.

[0146] Alternatively, a single DMOV (with a single memory data buffer in memory 114) can be linked to a set of pre-work instructions for data collection operations. Each such referenced pre-work instruction will bring data from a specific source and location into the same data buffer at a different offset, thereby forming a collection operation. The pre-work instructions are scheduled in the same WD as the DMOV that prepared the data. A single pre-work operation can provide data to be pushed by several DMOV operations.

[0147] The pre / post work engine instruction set is executed by a pre / post work engine implemented in software. "Pre work" needs to be executed relative to a given ESP, and "post work" needs to be executed relative to a given ESP.

[0148] Autonomous execution of instructions by the PPE can be implemented in both "gateway push" and "advanced accelerator pull" models of operation. The PPE 123 uses RDMA, NFS, NVMoF, iSCSI, or any other supported file access protocol to move data to or from the gateway external memory / storage 114. The execution of streaming operations is directly controlled by PPE instructions found in the "late / early work portion" of the executable image. The PPE 123 can be considered a software-based streaming processor that takes instructions from the image file and transforms these instructions into local / remote storage operations. These transfers will be made between the gateway memory 114 and the external memory / storage.

[0149] PPE 123 executes in parallel with DME 122, and because DME 122 depends on the results of PPE 123, PPE 123 must complete its work before the data mover operation is scheduled for execution by DME 122. This is addressed in the executable image by grouping together DME 122 and PPE 123 instructions that belong to the same data exchange synchronization point using work descriptors.

[0150] Each PRW instruction retrieves data from external storage and stores the data in a precompiled data buffer (in the gateway memory 114) to which the PRW instruction is directed. Depending on the source of the data, the PRW instruction has different variants. These variants require different sets of parameters that detail the external IO operation. These details are looked up in the referenced IO template established in the control plane via the gateway control channel before execution begins.

[0151] The compiler pre-assigns areas in memory 114 for buffers referenced by PRW instructions. These buffers are used to store data retrieved from external storage when the PRW instructions are executed.

[0152] The ESP credit set is incremented by the PPE 123 for each WD when all preceding work-related instructions scheduled for that WD are completed, and only if all preceding work-related instructions scheduled for all previous WDs are also completed, and only if all following work-related instructions having end criteria with respect to that WD are also completed.

[0153] There are different variants of the PRW instruction depending on the source / destination of the data.

[0154] The execution order of PRW instructions is the order in which they are represented in the executable image. However, smaller batches of PRW instructions will run in parallel to optimize I / O performance at remote locations. When data is needed, one or more PRW instructions from one or more WDs are executed before the WDs. This is necessary to fill the data "pipeline" to be consumed by the WDs. The gateway 52 has a parallel execution engine for the preliminary work, allowing it to complete the preliminary work of filling the data "pipeline".

[0155] The order in which the PRW instructions are completed may be different from the order of the instructions in the executable image. However, this out-of-order completion is not a problem because the data eventually arrives in the gateway memory 114 without any order requirements. When it comes to the deployment order of this data to the accelerator 51, the DME 122 ensures that the instruction order is the order represented by the executable image.

[0156] The PRW instruction always has an end criterion. The GW 52 schedules the PRW instruction to be completed at an appropriate time before a given WD that the accelerator 51 needs to supply data. The end criterion is represented by the WD in which the PRW instruction is included. In the case where the WD cannot be supplied with data in a timely manner, the data exchange phase will be delayed until the data is available. This effectively stops the calculation phase of the accelerator 51 until the data is available. The occurrence of such stops is counted, and feedback from such monitoring will help optimize the gateway and / or compiler.

[0157] The POW instruction performs "post work" associated with a given ESP. Its main function is to move data from the gateway memory 114 to external storage (e.g., the host 63 or remote storage 151). The data stored in the gateway memory 114 is data received from the accelerator 51. Depending on the destination of the data, the POW instruction has different variants. These variants require different sets of parameters that detail the external IO operation.

[0158] It is up to the compiler to link the POW instruction to the data buffer in memory 114 on which to operate.

[0159] For post-processing, since the results are not communicated to the accelerator 51, but instead stored in the host 63, remote storage 151 storage, or gateway memory 114, instructions can be executed out of order with no implicit semantics associated with the write order of pure data.

[0160] The POW instruction always has a mandatory start criterion, which represents the earliest time point at which the instruction can be executed. The instruction can be executed after the mandatory start point but not earlier than it. Therefore, the POW instruction is triggered to start at a given WD. The triggering WD is represented as the WD containing the POW instruction. When the previous WD is completed, the accelerator 51 must have completed writing to the buffer of the POW instruction.

[0161] There are different types of POW instructions. A first type of POW instruction involves moving data from local GW memory 114 to remote storage 151. This can be configured by the host 63 through an indication (e.g., descriptor 119) sent via a control channel. A second type of POW instruction involves moving data from local gateway memory 114 to the host 63. This can also be configured by the host 63 through an indication sent via a control channel. A third type of POW instruction involves manipulation of data stored in the gateway memory 114.

[0162] The POW instruction may also have an optional end criterion represented by a parameter of the POW instruction. This may have the following effects. First, this optional end criterion may enable the POW instruction to prepare data for a specific WD, much as a previous work instruction has an end criterion implicitly given by the WD to which it belongs. Second, in the case where the gateway compiler is reusing the "output" buffer used by the POW instruction for export to an external node, it is extremely important to protect the buffer that still holds unsaved data from being overwritten by the accelerator 51. In this case, the program can protect the buffer by placing a so-called Named Execution Barrier (NEB) instruction as a termination point in the DME instruction stream until all POWs have completed flushing the buffer, thereby freeing the buffer for reuse and more accelerator 51 output operations. These NEB instructions will be described later.

[0163] If the POW instruction cannot meet its termination criteria, the PPE 123 will halt the local DME 122, and the result is that all accelerators are synchronized at the same synchronization level. The PPE 123 parses the POW instruction and finds the termination criteria. There may be several POW instructions with the same termination criteria or with different termination criteria or no termination criteria.

[0164] As described above, the compiler may place termination / pass "execution barriers" at a given execution point in time. (NEB) instructions are referred to as specifying an "execution barrier" completion (NEBC) object, which collects the number of completion reports from objects that are instructed to signal a NEBC upon completion (e.g., POW instructions).

[0165] A NEB instruction always belongs to a WD, i.e., it is encapsulated by a WD. It can be inserted into all three instruction streams (DME, PPE_PRE, and PPE_POST).

[0166] The "Terminated" state represents a termination signal to the DME / PPE, instructing the DME / PPE not to continue executing instructions in the WD. Another possible state is "Passed", which allows the DME / PPE to continue executing instructions in the WD, thereby passing the NEB instructions. The state changes from "Terminated" to "Passed" when all instructions linked to this end criterion have reported completion by incrementing the "Completions_seen" counter in the NEBC object.

[0167] The concept of "execution barrier" should not be confused with the ESP synchronization primitive that can be used to control barriers in the bulk synchronous parallel (BSP) memory consistency model. In some examples, the NEB instruction insertion point is associated with a specific ESP for the accelerator program, but there is no direct requirement to do so. The NEB can be used as a general termination point for various synchronizations.

[0168] A first example of using the NEB instruction can be given, in which (one or more) NEB instructions are inserted into the WD at the beginning of the DME instruction stream. NEB represents a precondition for executing the DME instruction. The precondition is used to control the flushing of the accelerator output buffer (or ring buffer fill threshold) to an external node (e.g., host 63 or remote storage 151) via the POW instruction. The ESP credit set will not be incremented until the following two occur: the NEB precondition is met and the PRW instruction is completed. This means that the WD can be cached by the DME, but will not be further executed if there are no available ESP credits. When the DME 122 has completed the execution of the PRW instruction, it will first check whether all NEB instructions in the WD are in the "pass" state. If they are in the "pass" state and all other preconditions for giving credits are met, the credits will be incremented. If the DME execution engine sees that the NEB instruction is in the terminated state, it will raise an exception. This exception indicates that the PPE has incorrectly added credits despite being in the "terminated" state, or there are some triggering conditions in the implementation of the DME / PPE.

[0169] A second example of using NEB instructions can be given in which NEB instructions are inserted into the post-work instruction stream to flow-control the export of data from the gateway 52 to the host 63. In this case, the host 63 controls the state of the NEBC. In this model, the host controls whether the PPE 123 is allowed to execute a POW instruction to transfer data to the host 63, thereby indicating through the NEB. This is controlled by the host providing an update to the state of the "linked" NEBC object to set the state to the "pass" state. The host only allows the "pass" state to be set when all linked POW instructions have been completed.

[0170] The end criteria are always placed on the "next occurrence" of NEB in the instruction stream. "Next occurrence" is understood to be related to the execution of POW.

[0171] A third example of using the NEB instruction can be given in which the NEB instruction is inserted into the pre-work instruction stream to flow control the data import feed from the host 63. In this case, the host 63 is controlling the state of the NEBC. In this model, the host controls whether the PPE 123 is allowed to execute a PRW instruction to transfer data from the host 63 or remote storage 151 to the memory 114, thereby indicating through the NEB. This is controlled by the host 63 providing an update to the state of the "linked" NEBC object to set the state to the "through" state.

[0172] The NEBC object is always initialized to the terminated state at the beginning of program execution. The same reinitialization is performed when starting from the next instruction after NEB. When setting the state to "terminated", "done_seen" is also set to zero.

[0173] In the case of a DME, the DME 122 itself may not have reached the point in its execution where it sees the NEB, and if all linked instructions have completed when the NEB instruction is seen, then "completion_seen" is the same as "expected_completions" and the status will be considered "passed", so execution can continue without waiting. Otherwise, the DME 122 waits until all linked instructions have completed.

[0174] In the gateway 52, each accelerator 51 has a streaming engine 124, wherein each streaming engine 124 can operate in the various modes that have been described.

[0175] There are several streaming engine instances available in the architecture. There is one streaming engine 124 per accelerator 51, where each streaming engine 124 is executing an image. Each streaming engine 124 feeds data to the accelerator 51 via one or more high-speed buses (e.g., PCIe Gen4).

[0176] There are a number of different possible streaming flows that can be implemented using the streaming engine 124. For example, in a first possible streaming flow, the gateway 52 can implement streaming of data destined for the accelerator 51. The streaming of such data can be initiated by another accelerator configured to provide the data. Alternatively, the streaming of data can be initiated by the DME 122 of the gateway 52, which executes instructions to transfer data from the memory 114 to the accelerator 51. Such data may have been received at the gateway 52 from the host 63 or remote storage 151.

[0177] In a second possible streaming flow, the gateway 52 can implement streaming of data to a remote accelerator. The accelerator 51 can provide a data packet with an address identifying the remote accelerator in the global address space to the gateway 52. ​​The gateway 52 is configured to forward the data packet to another gateway 128 using the address for delivery to the remote accelerator.

[0178] In a third possible streaming flow, the gateway 52 may enable streaming of data flowing into the local gateway memory 114. This may be the result of a local gateway offload. The data transfer to the memory 114 may come from the accelerator 51 at the ESP. The data transfer to the memory 114 may be the result of local RDMA or host RDMA. The data may be transferred to the memory 114 from an external storage such as the host 63, NAS 151, or from another gateway 128. Transferring data from such external storage to the memory 114 is part of the preliminary work performed by the PPE 123.

[0179] In a fourth possible streaming flow, the gateway 52 may implement streaming of data flowing into the memory of another gateway 128. The data transfer may be initiated by the gateway 52 itself. The data transfer may be initiated by the accelerator 51, which provides the gateway 52 with a data packet having an address identifying the other gateway 128 in the global address space. The data transfer to the other gateway 128 may be the result of a previous work instruction executed by the other gateway 128 to pull data from the gateway memory 114.

[0180] In a fifth possible streaming flow, the gateway 52 can implement streaming of data to the remote storage 151. Data is transferred from the gateway storage 114 to the remote storage 151 via one or more of the following protocols: RDMA protocol, Network File System (NFS) protocol, Non-Volatile Memory over Fabrics (NVMoF) protocol, and Internet Small Computer System Interface (iSCSI) protocol. The data transfer is initiated by the gateway. This transfer to the remote storage 151 may be caused by the execution of the post-work instruction by the PPE 123.

[0181] In a sixth possible streaming flow, the gateway 52 may implement streaming of data to the host 63. The data is transferred from the gateway memory 114 to a pinned host memory or an RDMA accessible host memory. This transfer to the host 63 may be caused by the execution of a post-work instruction by the PPE 123.

[0182] In the seventh possible streaming flow, the gateway 52 can implement streaming of data from one or more remote NFS servers. The data transmission from these servers can occur in response to a request sent by the gateway 52.

[0183] As mentioned earlier, parallel programming models for AI and HPC typically follow a three-phase iterative execution model: compute, barrier, and exchange (data transfer, collection, and broadcast). This means that accelerators typically require data transfers to / from the accelerator at pre-compiled data exchange synchronization points and / or collections performed at accelerator requests. The request represents a synchronization point at which the accelerator 51 has completed processing the available data and now needs to export some data and import some data. The gateway 52 will schedule its data movement immediately after the confirmed accelerator exchange request.

[0184] The gateway's streaming engine 124 optimizes data movement, so data buffer "objects" are essential to hold data. By passing pointers to buffers (in gateway memory 114) during execution, the system implements zero copy semantics during operation. The data buffers are pre-initialized in the loaded image, or filled by the PPE 123. In both cases, the DME 122 can use a reference to the buffer in memory 114 to transfer data to the accelerator 51 at the ESP.

[0185] There may be situations where there is no need for the upfront work of preparing the accelerator data, such as when the data is already ready and embedded in the loaded executable image. In this case, the PPE 123 will also be responsible for issuing ESP credits to the DME 122.

[0186] There may also be an ESP with no data movement to the accelerator 51 (e.g., just accelerator output data), and in this case, the PPE 123 will also be responsible for issuing ESP credits to the DME 122. In this case, the PPE 123 will increment the ESP credits for the upcoming ESP in response to determining that there is no data movement to the accelerator 51 during the upcoming ESP.

[0187] ESP credit is always added from PPE 123.

[0188] For previous work instructions only: if the previous work of a WD is completed ahead of the previous work in the previously issued WD, the design will need to queue the previous work completion information and increment the number of ESP credits after processing all previous WDs that have been completed.

[0189] For accelerator data import (i.e., data transfer from gateway 52 to accelerator 51), WD describes the number of bytes to be transferred in both directions (i.e., between accelerator 51 and gateway 52) during the exchange. The accelerator 51 in the push model is already compiled in the same information, so it knows when all the expected data for the exchange has been received, and starts the calculation phase immediately after all the data has been received. In the pull model, the accelerator 51 controls when the exchange ends by stopping reading data from the gateway 52.

[0190] For accelerator data export: the accelerator 51 knows from its compiled code how much data to send to the gateway 52 for a given ESP, and the gateway 52 knows how much data to expect by reading information from the WD.

[0191] When the gateway 52 has received the expected exact number of bytes from the accelerator 51, it will proceed to execute the next WD. While executing the next WD, the gateway 52 may perform post-processing, including local operations on the data in the gateway memory 114. Additionally or alternatively, the gateway 52 may perform post-processing to transfer the data to its final destination. Alternatively, the gateway 52 may not perform post-processing. For example, it may allow the data to stay in the gateway memory 114, allowing the memory 114 to be used as a non-accelerator data cache for later readback. While executing the next WD, the gateway 52 may perform pre-processing that needs to be completed before the next ESP. Additionally or alternatively, the gateway 52 may execute a DMOV instruction to be executed after the next ESP. If there are available ESP credits, the DMOV instruction is used to preload data into the gateway transfer memory 127 before the ESP. If there are no ESP credits, the DME 122 waits for ESP credits and performs preloading when ESP credits are available.

[0192] If the PPE instructions (i.e., both post work (POW) and pre work (PRW) instructions) are directed to remote storage 114 of static data known to be available on a storage node, then data synchronization with the node is not required as long as the gateway supports the storage protocol for directly accessing the data.

[0193] The host 63's memory is small relative to the amount of data it transfers to the gateway 52 and accelerator 51, so the host 63 needs to bring the data into its memory "piece by piece." Due to this "piece by piece" nature, there needs to be a synchronization mechanism between the gateway 52 and the host 63 to control when the data is available for RDMA reads initiated by the gateway 52 (gateway data import). Similarly, similar synchronization is required for RDMA writes initiated by the gateway 52 (i.e., gateway data export). The challenge facing the entire AI equipment is to keep data streaming in and out of the gateway / accelerator continuously, so this synchronization mechanism is critical to AI performance. The system needs a well-designed solution with minimal overhead in order to scale to large AI structures.

[0194] The streaming engine 123 has several modes of operation for moving data between the gateway and the host.

[0195] In a first mode of operation, the streaming engine 124 operates as a slave device of the host 63 under commands from the host 63. In a second mode of operation, the streaming engine 124 executes based on pre-compiled instructions stored in its code memory.

[0196] In the first operation mode, under the control of the host 63 , the streaming engine 124 acts as a slave device of the host 63 and performs operations of storing data in the memory 114 and retrieving the data from the memory 114 to deliver to the accelerator 51 .

[0197] In the second mode of operation, the streaming engine 124 pre-fetches data from the host 63 or remote storage 151 according to a pre-compiled executable file exported from a compiler, which is used to generate code for a complete system consisting of an accelerator and a gateway. Since the compiler is used to generate code for the gateway 52 (which obtains data to be delivered to the accelerator 51) and the accelerator 51 (which processes data), the host 63, the gateway 52 and the accelerator 51 can act synchronously with each other. The gateway 52 file anticipates the data required by the accelerator 51, and prepares the data for deployment before the relevant computing stage by storing the data in the memory 114. The gateway 52 prepares the data for transmission to the accelerator 51 at the right time according to the code generated by the compiler. In response to a synchronization request 56 from the accelerator 51, the DME 122 transmits it to the accelerator 51 at the right time for the accelerator 51 in a delay-optimized manner. The DME 122 is located near the accelerator 51 for delay-optimized delivery.

[0198] In the third operation mode, the accelerator 51 notifies the gateway 52 before the next N barriers which data is to be transferred from the memory 114 to the accelerator 51 for the corresponding N barriers. In this operation mode, the accelerator compiler can foresee future I / O operations and therefore schedule such commands to the gateway 52 so that the gateway 52 has enough time to deliver the data.

[0199] The compiler generates a set of computer code instructions executed by the accelerator 51. These sets of computer code instructions can be referred to as executable images. In some embodiments (e.g., in the second operating mode described above), the compiler can also generate a set of related streaming engine data movement / processing commands implemented by the gateway 52.

[0200] The compiler generates an executable image for each streaming engine. From the accelerator's perspective, the executable image references a flat, contiguous XPU virtual address (XVA) space. This XVA space covers the internal accelerator memory and the "streaming engine sandbox" memory that is mapped to the same XVA space via the memory management unit (MMU). The execution image also references a "host sandbox" virtual address (HSVA) space that covers the required host memory accessible to the streaming engine 122. This HSVA space is related to the GW operation model ("GW push model" and "advanced XPU pull model").

[0201] In both virtual address spaces (XVA and HSVA), the compiler is responsible for defining the existence of buffer resources and addressable elements required by the streaming engine 122, accelerator 51, and host 63.

[0202] The compiler is also responsible for reusing gateway buffers in memory 114 between iterations and sequences of defined WDs when it sees fit and when required due to limited gateway memory 114. As long as there is enough memory assigned to the gateway 52, no buffer reuse optimization is needed.

[0203] For a gateway 52 configured to communicate with two or more accelerators, it is currently not possible for one accelerator to access the streaming engine sandbox assigned to other accelerators. This is enforced by the MMU settings within each accelerator or accelerator support chip. The XVA spaces of different accelerators do not overlap in the physical gateway memory. The streaming engine runs in its own "XPU sandbox" and forces all accesses to remain in its own sandbox at runtime. Due to the accelerator's onboard MMU, it is possible to build a common memory area shared between these streaming engines.

[0204] Refer again Figure 7 In the data transfer to the accelerator shown, in some examples, the gateway 52 receives data from the host 63 or remote storage 151 and stores it in the memory 114, and then makes it available in the fast gateway transfer memory 127 for transfer to the accelerator 51. The DME 122 preloads the fast gateway transfer memory 127 from the memory 114 according to the DME instructions. In response to the completion of the handshake request, the contents of the gateway transfer memory 127 are transferred to the accelerator 51. This preloading of the gateway transfer memory 127 is used in the push model described above. In some examples, the preloading of the gateway transfer memory 127 is performed only when the number of ESP credits is greater than zero.

[0205] Reference Fig.14 , which shows how the preparation of data, the exchange between the gateway 52 and the accelerator 51, and the processing of the data are related. The preparation and deployment phases are performed by the gateway 52, while the calculation phase is performed by the accelerator 51. The data is prepared by the gateway 52 before the associated calculation phase. The data is stored as close to the accelerator 51 as possible. When the accelerator 51 is able to accept the data and indicates so by sending a synchronization request 56 to the gateway 52, the gateway 52 deploys the data using the full capacity of the port linked to the accelerator 51 without external dependencies. While the accelerator 51 is processing the deployed data, the gateway 52 prepares the data for the next phase to be deployed. The engine can scale its operations across all available gateway data center ports.

[0206] The gateway 52 can receive data from the host 63 or remote storage 151 and perform storage and expansion of data required by additional gateways. The data can be transmitted to the additional gateways. The data transmitted to the additional gateways can then be provided to the accelerators associated with those additional gateways. This may help avoid bottlenecks. For example, instead of each gateway independently retrieving data from the remote storage 151 and thus causing a bottleneck when accessing the remote storage 151, one gateway 52 can retrieve data from the remote storage 151 and provide the above data to multiple gateways. This can solve the bottleneck problem when accessing the remote storage 151.

[0207] When the gateway 52 receives data from the host 63 or remote storage 151, the gateway 52 processes the data before providing it to the accelerator 51. This processing may be performed by the streaming engine 124. The processing may include one or more of: data augmentation (noise injection), decompression, decoding (e.g., decoding of image and video data such as images in JPEG format and videos in H264 format). This processing is not performed in the simple accelerator pull model discussed above.

[0208] In order to keep memory usage to a minimum, data is compressed when it is loaded into the gateway 52, and decompressed at the latest possible time before delivery to the accelerator 51. The gateway 52 may provide a latency-optimized hardware decompression engine (not shown) for certain types of compression. In addition, decompression may be implemented in the gateway software, providing extended support for arbitrary compression algorithms.

[0209] By performing data augmentation (e.g., noise injection) in the gateway 52, the original data can be stored once in its original format and retrieved once. The data can then be replicated to multiple accelerators, with the gateway 52 applying different augmentation settings to each replicated copy. The gateway 52 provides a set of augmentation methods in hardware and provides the gateway software with the ability to implement different algorithms for the augmentation.

[0210] In one embodiment, the streaming engine 124 provides two data acceleration features. The streaming function provides a copy feature and a copy and transpose feature. This allows training data to be copied from one gateway to many other gateways, thereby reducing IO connection requirements.

[0211] Data is received from the host 63 or remote storage 151 at the gateway 52 and stored (after passing through the path 120) in the memory 114 by the PPE 123. The DME 122 retrieves the data to be sent along the path 121 from the memory 114 and causes the data to be sent to the accelerator 51. The data is sent from the memory 114 to the accelerator 51 via the indicated accelerator port. As already described, the data transfer along the path 121 is triggered by a synchronization signal.

[0212] Gateway 52 allows data provision to accelerator 51 (which involves transferring data over path 121) to be decoupled from data retrieval from host 63 or remote storage 151. In other words, gateway 52 enables data transfer from host 63 or remote storage 151 to occur before the operations performed by accelerator 51.

[0213] Figure 8 Two other data paths are shown that allow data to be exchanged between the gateway 52 and other gateways. The gateway 52 includes a path 125 from which data can be transmitted between the accelerator 51 (coupled to the gateway 52 through the accelerator port shown) and another accelerator (not shown) via another gateway 128 (coupled to the gateway 52 through the fabric port shown). The gateway 52 and the other gateway 128 act as switches on the path 125 and implement an extended data exchange structure between accelerators. The other gateway 128 can be configured to transmit data to / from another host to which it is connected. Data transmission along the path 125 can be unicast (i.e., data directed to a single accelerator), broadcast (data sent without being directed to a specified accelerator), and multicast (data directed to multiple specified accelerators). In broadcast mode, the data packets sent on the fabric port contain a multicast group ID. Each gateway has a table containing a list of destinations for each multicast group ID. When a gateway receives such a data packet, it looks up the destination list corresponding to the multicast group ID included in the data packet in the table and sends the data packet to those destinations.

[0214] In one embodiment, the XPU port is an implementation of a custom root complex that provides dedicated data movement capabilities. In addition to transmitting packets to / from the gateway memory 114, the XPU port also provides peer capabilities to / from the structure port. Packets targeted at the memory space mapped to the remote accelerator are detected at the XPU port and directed to the appropriate structure port. The receiving structure port will direct the packet to the correct destination accelerator port. In addition, the gateway can forward packets from one structure port to another. This allows passing through arbitrarily large structures. In this way, a complete accelerator-to-accelerator exchange can be achieved through the gateway structure.

[0215] Figure 8 Also shown is a data path 126 for exchanging data between the gateway 52 and another gateway. The data path 126 is used for the exchange of synchronization and management messages between the gateway 52 and another gateway 128. In addition, the data path 126 is used to exchange data between the memory 114 associated with the gateway 52 and the memory associated with another gateway 128. When the preliminary work is performed by the PPE 123, the data exchanged via the data path 126 is exchanged as part of the preliminary work.

[0216] In response to the execution of the previous work instruction by the PPE 123, data may be transferred from the memory of another gateway 128 to the memory 114. The data is then available in the memory 114 for transfer at the upcoming ESP (e.g., by a PCIe read operation from the accelerator or by the execution of a DME instruction by the DME 122) to the accelerator 52. When the PPE 123 completes execution of the previous work instruction for transferring data to its memory 114, it increments its ESP credit set.

[0217] As previously mentioned, a synchronization region / group may include multiple gateways. In this case, instead of or in addition to receiving a synchronization request from an associated accelerator 51, a synchronization request may be received at a gateway 52 from another gateway 128. In this case, the other gateway 128 may be referred to as a "downstream gateway".

[0218] Now refer to Fig.15 , which shows a gateway 52 in communication with another gateway 128 and an additional third gateway 152. When a synchronization request 129 is received from another gateway 128, in the case where the gateway 52 is not a synchronization master (i.e., the gateway 52 is a synchronization slave), the gateway 52 can allow the synchronization barrier to be passed by sending a synchronization request 153 upstream to the third gateway. The synchronization request 129 can first be aggregated with one or more synchronization requests (e.g., synchronization request 56) received from a local accelerator (e.g., accelerator 51). In this case, the aggregated synchronization request 153 is sent upstream to the third gateway.

[0219] Alternatively and, for example, where gateway 152 is not connected to the synchronization zone of gateway 52, when a synchronization request 129 is received from another gateway 128, gateway 52, where gateway 52 is the master, may allow the synchronization barrier to be passed by sending a synchronization confirmation 154 to the other gateway 128. Where gateway 128 is the master, it also confirms (e.g., by sending a confirmation 155) any synchronization request received from a local accelerator (e.g., accelerator 51), as long as the synchronization request is received from all configured downstream gateways.

[0220] The ESP credits in the LSBM 118 maintained by the gateway 52 may be used to control the forwarding of synchronization requests between the gateway 52 and another gateway 128. As with the barrier between the accelerator 51 and the gateway 52, the ESP credits are used to control the forwarding of synchronization requests between the gateway 52 and another gateway 128 only if the local accelerator (e.g., accelerator 51) that sends the synchronization request 155 to the gateway 52 indicates that the gateway is participating. As previously described, the indication may be stored in the register 59. If the gateway is not indicated to participate, a synchronization request 153 is sent upstream when the synchronization request 129 is received, and when the synchronization confirmation 154 is returned, the synchronization barrier is passed.

[0221] Assuming that the accelerator 51 instructs the gateway to participate, if the number of ESP credits associated with the accelerator 51 is non-zero, and the gateway 52 has received a sync request 129 from a downstream gateway 128, if the gateway 52 is not a sync master gateway (i.e., it is a sync slave gateway), the barrier is passed upstream. The sync request 129 is aggregated with the sync request 56 from the accelerator 51 to form a sync request 153, which is sent to the upstream gateway 152. Upon receiving a sync acknowledgement 156 corresponding to the sync request 153 for synchronization requiring gateway participation, the ESP credits in each LSBM 118 in the synchronization chain are decremented.

[0222] Assuming the accelerator 51 instructs the gateway to participate, if the number of ESP credits associated with the accelerator 51 is non-zero, and the gateway 52 has received a sync request 129 from a downstream gateway, if the gateway 52 is a sync master gateway, it will send a sync confirmation 154 to the downstream gateway 128 and its own streaming engine(s) 124. Upon receiving the sync confirmation, the streaming engine 124 decrements the number of ESP credits held by the LSBM 118.

[0223] Therefore, the LSPM 117 of the gateway 52 can prevent the sync request from being propagated to other gateways (ie, LSBMs) in the absence of ESP credits in the LSBM 118. This ensures that when the sync master eventually generates an acknowledgement, all accelerators will begin executing their supersteps simultaneously.

[0224] The gateway 52 includes multiple interfaces, for example, an interface to the accelerator 51, an interface to another gateway 128, an interface to a third gateway 152. The gateway 52 includes a register indicating the directionality of each of these interfaces for synchronization purposes, that is, whether an entity such as another gateway 128 is upstream or downstream of the gateway 52. ​​Therefore, the register indicates: in response to the gateway 52 receiving a synchronization request from a downstream entity, which interfaces the synchronization request is to be sent to by the gateway 52. ​​In the case where the register indicates that no interface is used for the sending of a synchronization request, this indicates that the gateway 52 is a synchronization master. In this case, the gateway 52 sends a synchronization confirmation through all interfaces on which it has received a synchronization request.

[0225] In the case where the gateway 52 acts as a slave device gateway, it can receive one or more synchronization requests from an accelerator associated with it (e.g., accelerator 51). These synchronization requests are aggregated by the gateway 52, which then passes them upstream to another gateway 128 (assuming that there are available ESP credits for each local accelerator from which the indicating gateway participates in which the synchronization request is received). Assuming that the other gateway 128 is also a slave device, the other gateway collects the above requests and all synchronization requests from its own local accelerators, and then forwards the new aggregated synchronization request to the next gateway (assuming that there are available ESP credits for each local accelerator from which the indicating gateway participates in which the synchronization request is received). This occurs in parallel on the synchronization network. Eventually, the master device gateway receives synchronization requests from all downstream gateways and their own associated accelerators. Then, and only then, is the synchronization completed, and a synchronization confirmation is generated by the master device gateway (assuming that there are available ESP credits for each local accelerator from which the indicating gateway participates in which the synchronization request is received) and the synchronization confirmation is sent downstream to the entity from which the synchronization request was received (i.e., the local accelerator or downstream gateway). Each downstream gateway that receives a synchronization confirmation will send a synchronization confirmation to the entity from which the synchronization request was received.

[0226] As described above, synchronization requests may be received at the gateway 52 from multiple local accelerators (not just the exemplary accelerator 51). Each accelerator is associated with a different set of ESP credits. The gateway 52 will pass the aggregated synchronization request upstream (in the case of a slave device), or acknowledge the synchronization request (in the case of a master device), only when all ESP credits for each accelerator from which a synchronization request has been received (and the gateway is indicated to participate) are non-zero.

[0227] As described above, after sending the synchronization confirmation to the accelerator 51 , the gateway 52 is configured to exchange data with the accelerator 51 .

[0228] refer to Fig.10, which shows the gateway functionality implemented by the streaming engine 124. PPE 123 executes in parallel with DME 122, but because DME 122 relies on the results of PPE 123, PPE 123 needs to provide its results before the DME operation is scheduled. This is handled in a pre-compiled executable image or by a user program sequence of commands delivered from the accelerator 51 to the gateway 52.

[0229] like Fig.10 As shown, there is a module 142 (shown as a GDxSM module) located between the PPE 123 and the network stack 141. The GDxSM module 142 includes two modules, namely, the Gateway Data Import Synchronization Module (GDISM) and the Gateway Data Export Synchronization Module (GDESM). Both modules handle the synchronization of I / O buffer elements between the gateway and the host.

[0230] Synchronization is flow controlled and ensures gateway data consistency and readiness for IO operations at the Exchange Synchronization Point (ESP).

[0231] The first set of credits (which have been discussed in detail) are ESP credits. ESP credits manage the passage of synchronization barriers between an accelerator 51 and a gateway 52 or between a gateway 52 and another gateway 128. Using ESP credits, a barrier credit mechanism is used to control data transfers between a gateway 52 and an accelerator 51. The availability of one ESP credit means that a data exchange operation can be performed against one barrier.

[0232] The second set of credits manages the transfer of data to the gateway 52 (from the host 63, remote storage 151, or another gateway 128). These credits are stored by the GDxSM 142. More specifically, these credits are stored in the GDISM of the GBxSM 142. The second set of credits may be referred to as GDISM credits. Those skilled in the art will appreciate that the term "GDISM credits" is just a name, and the nature of the credits is not limited by the name.

[0233] In response to determining that there is a non-zero number of GDISM credits available, the gateway 52 executes preliminary work instructions to retrieve data from the host 63, remote storage 151, or another gateway 128. If the gateway 52 determines that there is a zero number of GDISM credits available, it does not retrieve data. The host 63 sends an indication for updating / incrementing the GDISM credits, which it sends using RDMA. When the streaming engine 124 is notified of an update to the GDISM credit register via an RDMA write from the host 63, the streaming engine 124 will update the credit register accordingly. The gateway 52 decrements the number of stored GDISM credits in response to the PPE 123 completing preliminary work, which is to transfer data from external storage to the gateway 52.

[0234] The GDISM credit control mechanism can prevent the execution of the PRW instruction too early. GDISM controls how many WDs must precede the currently executing ESP before the PRW engine is allowed to work.

[0235] The host 63 can be configured to perform the same credit update on the GDISM credits of a group of gateways. Credit updates are performed using RDMA and protocols on top of RDMA for reliable broadcast. This may be necessary in the case where a synchronization group includes multiple gateways. In this case, the group of gateways may need to have the same number of GDISM credits available, otherwise one of the accelerators may stop and therefore terminate all other accelerators.

[0236] In some examples, GDISM credits are also used to control data transfer from the gateway to the host. The same set of GDISM credits (i.e., the second set described above) used for data transfer from external storage to the gateway 52 can be used to control data transfer from the gateway 52 to the external storage (e.g., the host 63, remote storage 151). In response to the gateway 52 sending data to the external storage, when the PPE 123 completes its command in the WD, these GDISM credits representing both import and export credits are decremented. If the number of GDISM credits is not zero, the gateway 128 will only send data to the external storage.

[0237] In this way, GDISM credits can be used to throttle POW instructions as well as PRW instructions. If the number of GDISM credits is not zero, the POW instruction cannot be executed. In the case where GDISM credits control the transfer of data to and from both external storage, a single GDISM credit is consumed only when all POW instructions and PRW instructions for a given ESP are completed.

[0238] In some examples, a third set of credits manages data transfer from the gateway 52 to the host 63 or remote storage 151. These credits are stored by the GDxSM 142. More specifically, these credits are stored in the GDESM of the GBxSM 142. The third set of credits may be referred to as GDESM credits. Those skilled in the art will appreciate that the term "GDESM credits" is only a name, and the nature of the credits is not limited by the name.

[0239] If the number of GDESM credits is not zero, the gateway 128 will only send data to external storage. In response to the gateway 52 sending data to external storage, the GDESM credits are decremented. In this way, GDESM credits can be used to block POW instructions. If the number of GDESM credits is not zero, the POW instruction cannot be executed. The gateway 52 decrements the number of GDESM credits in response to the completion of the POW instruction.

[0240] The host 63 sends an indication for updating / increasing the GDISM credits, which it sends using RDMA. When the streaming engine 124 is notified of an update to the GDISM credit register via an RDMA write from the host 63, the streaming engine 124 will update the credit register accordingly.

[0241] There is an association between GDISM credits and ESP credits. GDISM credits allow the gateway 52 to transfer data for one superstep from the host memory to the gateway memory 114. When the gateway 52 has loaded the data for the superstep into its memory 114, it will decrement the GDISM credits and add one credit to the ESP credits. Now, the accelerator 51 can perform a pull on this data (including a pull according to any pull model), or the gateway 52 can push this data to the accelerator 51 (a push according to any push model), because when the number of ESP credits>0, the LSPM 117 and / or LSBM118 will confirm the synchronization request.

[0242] refer to Fig. 9 , which shows an example of a system 130 including multiple accelerators 131, multiple gateways 132, and multiple hosts 133. Since the gateways 132 communicate with each other, the gateways 132 together form an Ethernet 134. The communication between the gateways 132 realizes the separation of the accelerators and the hosts. In other words, any host 133 in the system 130 can communicate with any accelerator 131.

[0243] Although Fig. 9 Each gateway 132 is shown associated with a host 133 with which it communicates, but in some embodiments, not each gateway has a host. Fig. 9One of the gateways 132 shown in FIG. 1 can communicate directly with one host 133. The one host 133 can control multiple gateways 134. The gateway coupled to the host can distribute data from the host to the remaining gateways 134. Alternatively, multiple gateways 134 can retrieve data from remote storage 151.

[0244] In the case where only one gateway 134 communicates with the host 133, the one gateway 134 may be the only gateway 134 including a network interface device among the multiple gateways 134. This has the advantage of reducing costs by reducing the number of components required to build the remaining gateways. When the remaining gateways provide data to the host, they may first perform data augmentation operations on the data and then provide the data to the gateway including the network interface device for communication with the host.

[0245] In some embodiments, there is no external host 133 in the system 130, but rather a host system is run on one or more gateways 134. In this case, a compiler is run on the gateway 134.

[0246] In some examples, a gateway 132 receives data from a host 133 and distributes the data to one or more other gateways 132. In other examples, a subset of gateways 132 receives data from one or more hosts 133 and distributes the received data to one or more other gateways. Each of the one or more other gateways 132 can provide the distributed data to its associated accelerator 131. By performing the above method, not all gateways 132 need to receive data from the host 133. This method can reduce costs because in this case, the entire bandwidth does not need to be provided to all gateways. The method can also improve efficiency. In some examples, each accelerator 131 in a group of accelerators receives the same data and processes it. In this case, the data only needs to be obtained once from the host 133. Therefore, the gateway 132 receives the above data from the host 133 and distributes a copy of the data to one or more gateways 132, each gateway being configured to distribute the data to their associated accelerators 131. Therefore, efficiency gains are achieved because the same data does not need to be obtained multiple times from the host 133. Alternatively, this can be combined with using remote storage 151 to retrieve data through the gateway. Using remote storage 151 for retrieval means that cost reduction can be achieved and the gateway can have the entire bandwidth. The host can send storage descriptors to many gateways, which can act in parallel based on these descriptors and pull / push data from remote storage 151 through independent network connections of each gateway. This technique scales I / O as a function of the number of gateways.

[0247] In some cases, data distributed from a gateway 132 to one or more other gateways 132 is modified at the one or more other gateways 132. For example, one or more other gateways 132 apply data augmentation to one or more other gateways 132. Such data augmentation is performed by the DME in the corresponding gateway. When each of the one or more other gateways 132 has modified the data it has received, it pushes the modified data to its associated accelerator 131.

[0248] The precompiled gateway software specifies which accelerator 52 gets which data from which host that the gateway 132 stores in memory 114. The compiler of the accelerator code decides how to distribute data between the accelerators in order to distribute work between them. The gateway 132 load balances I / O traffic on the two PCIe ports it has to each accelerator.

[0249] In this way, the gateway and accelerator layers of the system are replicated to allow for the expansion of the system. Fig.12 , which shows an example of an apparatus 161 including a plurality of accelerators 162 and a plurality of gateways 163. The apparatus 161 is referred to as a machine 161. The machine 161 includes four accelerators 162 and two gateways 163. Each gateway 163 is also coupled to one or more hosts (not shown).

[0250] Reference Fig.13 , which shows an example of a device 170, which includes multiple Fig.12 1. A plurality of machines 161 are arranged in an apparatus 171, which is referred to as a cluster 171. Each cluster 171 includes up to 4 machines 161. A plurality of clusters 171 are arranged in an apparatus 170, which is referred to as a pod 171. Each pod 171 includes up to 32 machines 161. By expanding the system in this manner, the resulting pod 171 includes 128 accelerators, realizing a system with 16PFLop and 8TB DRAM.

[0251] exist Fig.12 and Fig.13 In the model shown, each gateway 163 provides a low-latency bridge between two or more groups of accelerators 162, thereby allowing accelerators 162 attached to different gateways 163 to communicate with each other as if they were connected to the same internal structure. Figure 8 162). Packets targeted to the memory space mapped to the remote accelerator are detected at the XPU port and directed to the appropriate fabric port of the gateway 163 (e.g. Figure 8). Packets received at the appropriate accelerator port will be forwarded to the appropriate gateway. From there, the gateway will forward the packet to the remote accelerator indicated by the memory space that is the destination of the packet.

[0252] Each gateway 163 includes a PCIe port. Four of these PCIe ports are configured to pass packets to and from the accelerator 162. Each PCIe port (eg, Fig.12 The custom gateway layer 163 (shown) can be configured to use different accelerator-specific protocols. The custom gateway transaction layer then translates between that protocol and the gateway internal protocol. The custom gateway layer implements address mapping and provides collection and broadcast / multicast offload support. Each gateway 163 provides an address mapping scheme that presents all participating accelerators 162 in a global address space. Packets received from accelerators 162 at gateway 163 contain a gateway ID that identifies the destination gateway to which the packet will be routed.

[0253] The global address space encompasses the memory resources of all accelerators 162 and all gateways 163 belonging to the cluster 170. An accelerator can dispatch packets that specify addresses in the global address space. Some portions of the address are used to select resources on a target gateway. Some portions of the address are used to identify the gateway being addressed. Other portions are used to identify the address of a memory in a gateway memory or a block memory of an associated accelerator. The block memory of an accelerator can be addressed by a block index and a memory offset. The address can include the block index and the memory offset to identify the location in the accelerator where the data in the packet is to be stored.

[0254] When a packet is received, the identification of the gateway in the address is compared with the global ID of the gateway. If there is a match, the target of the request is a resource belonging to the gateway (local accelerator or local memory). Otherwise, a portion of the address is used to index the routing table. The contents of the routing table indicate the destination port in the system. Certain bits of the address will be matched with the gateway routing table to determine where to route the packet.

[0255] The ingress packet pipeline is intended to be a cut-through pipeline with no buffering other than the pipeline stages required to implement the desired features. Packets are first classified by type: multicast / broadcast, collection, and unicast / memory write. They are then split into separate blocks for processing. The gateway 52 may include a unicast module and a multicast packet table for processing unicast packets. The gateway 52 uses a unicast packet routing table to perform routing of unicast packets (i.e., those directed to a single accelerator). The incoming address is decoded and selected bits are used to determine the destination. This is a two-step process: first, the gateway ID bit is used to determine whether the packet is targeted at the gateway. If not, the gateway ID bit is used to index a routing table, which returns the output structure port for the packet.

[0256] If the packet is destined for the gateway 52, the local address bits in the packet address are used to look up in a set (BARS) of local gateway base address registers (BARs) consisting of multiple regions (i.e., one BAR for the gateway memory and one BAR for each accelerator port). If the local address bits indicate that the packet is for storage in the gateway memory (e.g., memory 114), the packet is stored in the gateway memory according to the address in the BAR of the gateway memory. If the local address bits indicate that the packet is to be delivered to an accelerator, the packet is forwarded to the DME 122 of the gateway 52. ​​From there, the packet may be forwarded to the accelerator according to the address in the BAR for the associated accelerator port.

[0257] Packets specifying multicast / broadcast services are processed in the multicast group table. Each fabric port has its own table with a list of ports that will get a replica for each group (including broadcast). There are three destination sets. First, packets are sent to the local accelerator if and only if the packet belongs to the same virtual fabric as the gateway. Second, all incoming broadcast / multicast packets are checked against the fabric table to determine if they must be forwarded. Third, a replica is sent to the local DRAM. Once the destination port vector is built, the vector and the packet are forwarded to the switch interconnect, which provides the replication service.

Claims

1. A gateway used in a computing system for connecting a host with an accelerator subsystem for acting as a working accelerator of the host, the gateway having: An accelerator interface for connecting to the accelerator subsystem to implement the transmission of batch data between the accelerator subsystem and the gateway; a data connection interface for connecting to an external storage to exchange data between the gateway and the storage; a gateway interface for connecting to at least one second gateway, the second gateway being connected to another work accelerator; a memory interface connected to a local memory associated with the gateway; and A streaming engine for controlling streaming of batch data into and out of a gateway in response to a pre-compiled data exchange synchronization point reached by an accelerator subsystem, the pre-compiled data exchange synchronization point being triggered by the execution of a synchronization instruction as part of a pre-compiled code executable by the accelerator subsystem, wherein the streaming of the batch data is selectively via at least one of an accelerator interface, a data connection interface, a gateway interface, and a memory interface.

2. The gateway according to claim 1, wherein: One or more of the batches of data are transferred from the local memory to the accelerator subsystem in response to a pre-compiled data exchange synchronization point being reached by the accelerator subsystem.

3. The gateway according to claim 2, wherein: The one or more batches of data are made available at indicated storage locations in the local memory to be pulled from the local memory by the accelerator subsystem. 4 . The gateway of claim 2 , wherein the streaming engine is configured to execute instructions to push the one or more batches of data from the local memory to the accelerator subsystem.

5. The gateway according to any one of claims 2 to 4, wherein: The streaming engine is configured to stream the one or more batches of data from at least one second gateway into the local memory prior to a pre-compiled data exchange synchronization point reached by the accelerator subsystem.

6. The gateway according to any one of claims 2 to 4, wherein: The streaming engine is configured to stream the one or more batches of data from the external storage into the local memory prior to a pre-compiled data exchange synchronization point reached by the accelerator subsystem.

7. The gateway according to claim 4, wherein: The streaming engine includes a data mover engine and a memory management engine, the data mover engine and the memory management engine being configured to coordinately execute instructions from a work descriptor, The memory management engine is configured to execute instructions from the work descriptor to stream the one or more batches of data into the local memory before a pre-compiled data exchange synchronization point reached by the accelerator subsystem, The data mover engine is configured to execute instructions from the work descriptor to push the one or more batches of data from the local memory to the accelerator subsystem in response to a pre-compiled data exchange synchronization point reached by the accelerator subsystem.

8. The gateway according to any one of claims 1 to 4, wherein: The gateway is configured as: In response to a pre-compiled data exchange synchronization point being reached by the accelerator subsystem, receiving at the accelerator interface one or more additional batches of data in the batch of data; and storing the additional one or more batches of data in a local memory, Wherein the streaming engine is configured to subsequently stream the additional batch or batches of data from the local memory to the external storage.

9. The gateway according to claim 7, wherein: The gateway is configured as: In response to a pre-compiled data exchange synchronization point being reached by the accelerator subsystem, receiving at the accelerator interface one or more additional batches of data in the batch of data; and storing the additional one or more batches of data in a local memory, wherein the streaming engine is configured to subsequently stream the additional batch or batches of data from the local storage to the external storage, Wherein the memory management engine is configured to execute instructions from the work descriptor to stream the additional batch or batches of data to the external storage.

10. The gateway of any one of claims 1 to 4, wherein the gateway is configured to receive a data packet from the accelerator subsystem in response to a pre-compiled data exchange synchronization point being reached by the accelerator subsystem.

11. The gateway according to claim 10, configured to: checking the address of the data packet to determine whether the data packet is addressed to a resource associated with the gateway; and In response to determining that the data packet is addressed to a resource associated with the gateway, forwarding the data packet to the resource, The resources associated with the gateway include one or more of the following: a second accelerator subsystem for acting as a working accelerator for the host; and The gateway's storage.

12. The gateway according to claim 10, configured to: looking up in the routing table the port of the gateway interface corresponding to the address indicated in the packet; and The data packet is forwarded through the port of the gateway.

13. The gateway of claim 12, wherein the address indicates an address of an accelerator associated with a second gateway, wherein the gateway is configured to forward the data packet to the second gateway through the gateway interface.

14. The gateway according to claim 11, wherein: The address is an address in a global address space covering multiple accelerators and gateways in the extended system.

15. The gateway according to any one of claims 1 to 4, wherein the external storage comprises at least one of: a storage of a host; a local storage and a network attached storage.

16. The gateway according to claim 12, wherein: The gateway interface includes a plurality of ports, each port being connected to a corresponding second gateway among the plurality of gateways.

17. The gateway of claim 14, wherein the accelerator and the gateway form part of a common synchronization area and are each configured to generate a synchronization confirmation in response to receiving an indication of a pre-compiled data synchronization point when a corresponding condition is met.

18. A system on chip, comprising one or more gateways according to any one of claims 1 to 4 and at least one accelerator connected at the accelerator interface.

19. A method of operating a gateway according to any one of claims 1 to 4, the method comprising: The streaming engine executes instructions to selectively transmit the batch data to and from the gateway via the selected one of the interfaces.

Images