Communication between the host and the accelerator over the network

Abstract

A host system compiles a set of local programs that are provided to multiple subsystems over a network. By defining synchronization operations on the host and then providing that information to the subsystems, the host is able to service a large number of subsystems. Defining synchronization operations includes defining synchronization groups where synchronization barriers occur and defining points in time at which data exchange with the host occurs during program execution. Defining synchronization operations between subsystems allows a large number of subsystems to be connected while minimizing the required exchanges with the host.

Description

[Technical field]

[0001] This application relates to a host system and a processing unit communicating over a network. [Background technology]

[0002] In the context of data processing for complex or high-volume applications, a work accelerator may be a subsystem where the processing of certain data is offloaded from a host system. Such a work accelerator may have hardware specialized to perform a particular type of processing.

[0003] As an example, one area of ​​computing where such specialized accelerator subsystems may be useful is found in machine intelligence. As is well known to those skilled in the field of machine intelligence, machine intelligence algorithms are centered around performing iterative updates of a "knowledge model" that may be represented by a graph of multiple interconnected nodes. The implementation of each node involves processing data, and the interconnections of the graph represent data exchanged between the nodes. Typically, at least some of the processing of each node can be performed independently of some or all of the other nodes of the graph, and thus large graphs expose great opportunities for multithreading. Thus, work accelerators specialized for machine intelligence applications may include massive multithreading. One form of parallel processing can be achieved by a processor that includes an array of multiple tiles on the same chip (i.e., the same die), with each tile including a separate processing unit as well as memory (including program memory and data memory). Thus, separate portions of the program code can be executed in parallel on different tiles. The tiles are connected to each other via on-chip interconnects, which allow data to be exchanged between the tiles. Such accelerators may act as subsystems for a host system to perform parallel processing of provided data sets.

[0004] In general, there may be dependencies between parts of a program running on different tiles. Thus, techniques are needed to prevent dependent data of a previously executed piece of code on one tile from being made available to another piece of code on another tile. There are many possible schemes to achieve this, one of which, Bulk Synchronous Parallel ("BSP"), is described as an example herein. According to BSP, each tile performs a computation phase and an exchange phase on alternating cycles. During the computation phase, each tile executes one or more computation tasks locally on the tile, but does not communicate the results of its computations with any other tiles. In the exchange phase, each tile may exchange one or more results of the computation from the preceding computation phase with one or more other tiles in the group, but does not yet proceed to the next computation phase. Furthermore, according to the BSP principle, a barrier synchronization is placed at the junctions where the computation phase transitions to the exchange phase, the exchange phase transitions to the computation phase, or both. That is, either (a) all tiles in a group must complete their respective computation phases before any tile in the group can proceed to the next exchange phase, (b) all tiles in a group must complete their respective exchange phases before any tile in the group can proceed to the next computation phase, or (c) both. In some scenarios, tiles performing computations may communicate with other system resources, such as network cards or storage disks, as long as this does not involve communication with other tiles in the group.

[0005] During the exchange phase, data exchange may occur not only internally within the accelerator (i.e., between tiles), but in some circumstances may also need to occur between the accelerator and further accelerators, or between the accelerator and the host system. When a subsystem acts as a working accelerator, the subsystem is configured to process a data set that is provided to it (e.g., from the host system or from another form of storage in an extended system). Summary of the Invention [Problem to be solved by the invention]

[0006] To provide workload from a host to one or more subsystems, the host can be locally connected to those subsystems over a PCIe interface. This allows data transfer between the host and one or more subsystems with low latency. However, one problem is that the number of subsystems that the host can directly connect to over a PCIe connection is limited. Thus, in trying to scale up the number of subsystems to run larger applications, some challenges are presented because of the problem of how to accomplish data exchange between those subsystems and the host. [Means for solving the problem]

[0007] To process larger applications, it is desirable to interconnect more accelerator subsystems that can act as work accelerators to the host. One proposal for achieving this is to connect the host to the accelerator subsystems over a network. There is a delay between the host device receiving results from operations performed by the subsystems and providing further workload to the subsystem in response. This is especially true when those workloads may depend on results output by other subsystems. In some instances, the host must receive results from one subsystem and provide results to another subsystem.

[0008] According to a first aspect, there is provided a host system configured to be connected over a network to a plurality of subsystems that operate as work accelerators for the host system in accordance with a local program executed in each of the one or more subsystems, the host system including at least one processor and at least one memory storing computer readable instructions, the at least one processor of the host system configured to execute the computer readable instructions to compile a set of local programs, each local program being executed in each of the one or more subsystems, the step of compiling the source code including defining a set of synchronization groups, each synchronization group including at least some of the subsystems, each local program including a computation instruction set for performing a computation operation during one or more of a plurality of computation phases of the subsystem when executed by the at least one processor of the respective subsystem, and a data transfer instruction set for performing a plurality of barrier synchronizations at predefined points in the execution of the local program, each of at least some of the barrier synchronizations being defined between one of the synchronization groups, at least one of the barrier synchronizations being a synchronization requiring host involvement, and a synchronization requiring host involvement followed by a synchronization requiring host involvement, the synchronization requiring host involvement being followed by a synchronization requiring data transfer. Initiator In response to the respective subsystems providing the respective subsystems to the host system, data exchange is performed between the respective subsystems and the host system, and at least one processor of the host system receives a received notification of the data transfer. Initiator In response to each of the identifiers Provided by A host system is provided that is configured to exchange data over the network with one of the subsystems.

[0009] By defining synchronization operations on the host and then providing that information to the subsystems, the host can provide services to a large number of subsystems. Defining synchronization operations includes defining synchronization groups where synchronization barriers occur and defining the points in time where data exchange with the host occurs during program execution. Defining synchronization operations between subsystems allows a large number of subsystems to be connected while minimizing required exchanges with the host.

[0010] In some embodiments, the identifier Provided by The exchange of data over the network with one of the subsystems is performed using remote direct memory access.

[0011] In some embodiments, the network is an Ethernet network.

[0012] In some embodiments, the identifier Provided by The step of exchanging data over the network with one of the subsystems that is a host system includes: Provided by The method includes synchronizing at least a portion of a buffer of the network interface device that can be accessed by one of the subsystems, which are the

[0013] In some embodiments, the received data transfer Initiator Each of the host subsystem and the identifier Provided by the identifier of the stream between one of the subsystems Provided by The step of exchanging data over the network with one of the subsystems comprises sending and / or receiving data belonging to each identified stream.

[0014] In some embodiments, at least a portion of the host system's buffer includes a portion of the buffer associated with the identified stream, and Provided by At least a portion of the buffer of the network interface device that is accessible to one of the subsystems includes a portion of the buffer associated with the identified stream.

[0015] In some embodiments, the identifier Provided by The step of exchanging data over the network with one of the subsystems that is Provided by transmitting data to a network interface device accessible by the subsystem before one of a plurality of barrier synchronizations defined in a local program of the subsystem; Provided by The method includes receiving data from a network interface device accessible by a subsystem, the subsystem being a network interface device, following one of a plurality of barrier synchronizations defined in a local program of the subsystem.

[0016] In some embodiments, at least one processor of the host system is configured to cause each of the local programs to be dispatched over the network to a respective subsystem.

[0017] According to a second aspect, a subsystem configured to be connected to a host system via a network, the subsystem configured to operate as a work accelerator for the host system according to a local program executed on at least one processor of the subsystem, the subsystem including an interface configured to receive the local program from the host system over the network, the local program including: a representation of a set of synchronization groups, at least some of the synchronization groups including the subsystem and one or more further subsystems for operating as accelerators for the host; a computation instruction set for, when executed by the at least one processor, performing a plurality of computational operations during one or more of a plurality of computational phases of the subsystem; and a data transfer instruction set for, when executed by the at least one processor, performing a plurality of barrier synchronizations at predefined points in the execution of the respective local program, at least some of the barrier synchronizations being defined between one subsystem of the synchronization group, at least one of the barrier synchronizations being a synchronization requiring host involvement, and a notification of data transfer by the subsystem to the host system following the synchronization requiring host involvement. Initiator followed by data exchange between the subsystem and a host system over a network.

[0018] In some embodiments, data exchange between the subsystem and the host system is performed using remote direct memory access.

[0019] In some embodiments, the network is an Ethernet network.

[0020] In some embodiments, exchanging data between the subsystem and the host system includes at least one processor causing an exchange of data with a buffer of a network interface device for the subsystem, at least a portion of the buffer being synchronized with at least a portion of a buffer of the host system.

[0021] In some embodiments, the data transfer Initiator includes identifiers of streams between the subsystem and the host subsystem, and data exchange between the subsystem and the host system includes exchanging data belonging to each identified stream.

[0022] In some embodiments, at least a portion of the buffer of the network interface device includes a portion of the buffer associated with the identified stream.

[0023] In some embodiments, data exchange between the subsystem and the host system involves the use of identifiers From transmitting data to a network interface device accessible by the subsystem before one of a plurality of barrier synchronizations defined in a local program of the subsystem; From and receiving data from a network interface device accessible to the subsystem following one of a plurality of barrier synchronizations defined in a local program of the subsystem.

[0024] In some embodiments, the interface is configured to receive a local program from a host system over a network.

[0025] According to a third aspect, a method implemented in a host system includes connecting over a network with a plurality of subsystems that act as work accelerators for the host system in accordance with a local program executed in each of the one or more subsystems, and includes compiling a set of local programs executed in each of the one or more subsystems, the step of compiling the source code includes defining a set of synchronization groups including at least some of the subsystems, each local program including: a computation instruction set for performing a computation operation during one or more of a plurality of computation phases of the subsystem when executed by at least one processor of the subsystem; and a data transfer instruction set for performing a plurality of barrier synchronizations at predefined points in the execution of the local program when executed by at least one processor of the subsystem, at least some of the barrier synchronizations being defined between one subsystem of the synchronization group, at least one of the barrier synchronizations being a synchronization requiring host involvement, and a synchronization requiring host involvement followed by a synchronization requiring host involvement, the synchronization requiring host involvement being followed by a synchronization requiring host involvement followed by a synchronization requiring host involvement. Initiator A data exchange is performed between each of the subsystems and the host system in response to the respective subsystems providing the respective subsystems with the host system, and a received notification of the data transfer is received. Initiator In response to each of the identifiers Provided by and exchanging data over a network with one of the subsystems, said subsystems being:

[0026] According to a fourth aspect, there is provided a computer program product for causing at least one processor of a host system to carry out a method according to the third aspect.

[0027] According to a fifth aspect, there is provided a non-transitory computer readable medium storing a computer program according to the fourth aspect.

[0028] According to a sixth aspect, there is provided a method implemented in a subsystem for a host system, comprising: connecting to the host system over a network; receiving a local program from the host system over the network; operating as a work accelerator for the host system according to the local program, the local program being executed on at least one processor of the subsystem and including an indication of a set of synchronization groups including the subsystem and one or more further subsystems for operating as accelerators for the host; executing computation instructions of the local program to perform computation operations during one or more of a plurality of computation phases of the subsystem; and at predefined points in the execution of the local program, executing data transfer instructions of the local program to perform a plurality of barrier synchronizations defined between one of the subsystems of the synchronization group and requiring host involvement, wherein following the synchronization requiring host involvement, the subsystem notifies the host system of the data transfer. Initiator followed by performing data exchange between each of the subsystems and the host system over the network.

[0029] According to a seventh aspect, there is provided a computer program product adapted to cause at least one processor of a subsystem to carry out a method according to the sixth aspect.

[0030] According to an eighth aspect, there is provided a non-transitory computer readable medium storing a computer program according to the seventh aspect.

[0031] For a better understanding of the present invention and to show how it may be carried into effect, reference will now be made, by way of example, to the accompanying drawings in which: FIG. [Brief description of the drawings]

[0032] [Figure 1] FIG. 1 is a schematic block diagram of a processor chip including multiple tiles. [Diagram 2]1 is a schematic diagram of a bulk synchronous parallel (BSP) computing model. [Diagram 3] FIG. 2 is another schematic diagram of the BSP model. [Figure 4] FIG. 1 is a schematic diagram of a multi-stage BSP scheme. [Diagram 5] FIG. 2 is a schematic diagram of a synchronization request and acknowledgement exchange for external synchronization. [Figure 6A] FIG. 2 is another schematic diagram of different synchronization groups. [Figure 6B] 1 illustrates a schematic of program flow involving a host synchronization proxy. [Figure 7] FIG. 1 is a schematic diagram of two accelerators and a gateway communicating with a host system over a network. [Figure 8] 2 is a schematic diagram of further accelerators and gateways communicating with a host system over a network. [Figure 9] FIG. 2 is a further schematic diagram of a system in which data exchange is performed between an accelerator subsystem and a host. [Figure 10] FIG. 4 is a message sequence diagram showing the exchange of messages between a host and an accelerator via a gateway. [Figure 11] 1 illustrates a method according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] This application relates to a method and system for a host system to compile and transfer a local program set for execution on a processing unit. The processing units synchronize with each other and exchange data with the host at predefined points inserted in the compiled code. An example processing unit on which the embodiments may be implemented is first described.

[0034] Referring to Figure 1, Figure 1 illustrates an exemplary processing unit 2. This exemplary processing unit 2 is a multi-tile processing unit 2. The processing unit 2 may be an intelligent processing unit (IPU) as described in our prior U.S. patent application Ser. No. 16 / 538,980, the contents of which are incorporated by reference. Each of the IPUs is formed on a separate integrated circuit.

[0035] The processing unit 2 includes an array 6 of multiple processor tiles 4 and an interconnect 34 connecting the tiles 4. The processing unit 2 may be implemented alone as one of multiple dies packaged in the same IC package. The interconnect 34 may also be referred to herein as a "switching fabric" 34, since it allows the tiles 4 to exchange data. Each tile 4 includes respective instances of execution units and memory. For example, by way of illustration, the processing unit 2 may include hundreds of tiles 4 or even more than a thousand tiles 4. For completeness, it should also be noted that the "array" referred to herein does not necessarily imply a particular dimensionality or physical layout of the tiles 4.

[0036] In an embodiment, each processing unit 2 also includes one or more external links 8, allowing the processing unit 2 to connect to one or more other processing units (e.g., one or more other instances of the same processing unit 2). These external links 8 may include one or more processing-unit-to-host links for connecting the processing unit 2 to a host system and / or one or more processing-unit-to-processing-unit links for collectively connecting with one or more other instances of the processing unit 2 on the same IC package or card or on different cards. The processing unit 2 receives work from the host in the form of application data that the processing unit 2 processes.

[0037] The interconnect 34 is configured to allow different tiles 4 of the array 6 to communicate with each other. However, just as there can potentially be dependencies between threads on the same tile 4, there can also be dependencies between parts of programs executing on different tiles 4 of the array 6. Thus, a technique is needed to prevent dependent data of a previously executed piece of code on one tile 4 from becoming available to another piece of code on another tile 4. This is achieved using a data consistency model.

[0038] Parallel programming models in AI and data science usually follow a three-phase iterative execution model: compute, barrier, and exchange. What this means is that data transfers to and from processors usually have barrier dependencies to provide data consistency between processors and between each processor and external storage. Typically used data consistency models are bulk synchronous parallel (BSP), stale synchronous parallel (SSP), and asynchronous. While the processing unit 2 described herein uses the BSP model, it will be apparent that other synchronization models can be used as alternatives.

[0039] 2 and 3, which show an implementation of a BSP exchange scheme, in which each tile 4 executes a computation phase 33 and an exchange phase 32 in alternating cycles, the computation phase 33 and the exchange phase 32 being mutually separated by a barrier synchronization 30 between tiles. In the case shown by Figs. 2 and 3, the barrier synchronization is placed between each computation phase 33 and the following exchange phase 32. During the computation phase 33, each tile 4 executes one or more computation tasks locally on the tile, but does not communicate the results of these operations with any other tiles 4. In the exchange phase 32, each tile 4 may exchange one or more results of an operation from a preceding computation phase with one or more other tiles 4, but does not execute a new operation until it receives data from the other tiles 4 on which its task has a dependency. It also does not transmit data other than that computed in the preceding computation phase to the other tiles 4. It is not excluded that other operations, such as internal control-related operations, may be executed in the exchange phase 32. Communication outside the tile group can optionally utilize the BSP mechanism, but alternatively may not utilize the BSP and may instead use some other synchronization mechanism of its own.

[0040] According to the BSP principle, the barrier synchronization 30 is placed at the junction of the transition from the computation phase 33 to the exchange phase 32, or from the exchange phase 32 to the computation phase 33, or both. That is, either (a) all tiles 4 of the group must complete their respective computation phase 33 before any tile of the group can proceed to the next exchange phase 32, or (b) all tiles 4 of the group must complete their respective exchange phase 32 before any tile of the group can proceed to the next computation phase 33, or (c) both of these conditions are implemented. In all three variants, it is the individual tiles that alternate between phases and the whole assembly that is synchronized. The sequence of exchange and computation phases can then be repeated for multiple iterations. In BSP terminology, each iteration of the exchange phase and the computation phase is sometimes referred to as a "superstep" (although it should be noted that the terminology is not always used consistently in the literature; i.e., each individual exchange phase and computation phase is sometimes referred to individually as a superstep, whereas in the terminology adopted in this specification, the exchange phase and the computation phase together are sometimes referred to as a superstep).

[0041] It should also be noted that each of multiple different independent groups of tiles 4 on the same processing unit 2 or different processing units may form a separate respective BSP group that operates asynchronously with respect to each other, and the BSP cycles of calculation, synchronization and exchange are imposed only within each given group, but each group is not precluded from doing so independently from other groups. That is, the multi-tile array 6 may include multiple internal synchronization groups, each operating independently and asynchronously from other such groups (discussed in more detail below). In some embodiments, there is a hierarchical classification of synchronization and exchange, as discussed in more detail below.

[0042] Figure 3 shows (a) the BSP principle as implemented between some or all groups 4i, 4ii, 4iii of tiles of an array 6, in the case where a barrier synchronization (see above) from the computation phase 33 to the exchange phase 32 is imposed. Note that in this arrangement, some tiles 4 can start computation 33 while some other tiles are still performing the exchange.

[0043] The BSP model is used for the exchange of data between tiles 4 on a processing unit 2. As shown in our earlier application, U.S. Patent Application No. 16 / 538,980, communication between tiles 4 of a processing unit 2 occurs deterministically in time and data packets are sent without headers. In addition, the BSP model can also be used for the exchange of data between processing units 2.

[0044] In an embodiment, multiple instances of a processing unit 2 can be connected together to form a larger array of tiles 4 that spans multiple processing units 2. Processing units 2 connected together in this way can participate in barrier synchronization with each other to exchange data. Barrier synchronization that occurs only between tiles 4 on a single processing unit 2 is called internal (on-chip) synchronization. On the other hand, barrier synchronization that occurs between processing units 2 is called external (inter-chip) synchronization.

[0045] Referring to Figure 4, Figure 4 illustrates an example BSP program flow with both internal and external synchronization. As shown, the flow includes an internal exchange 50 (of data between tiles 4 on the same chip 2) and an external exchange 50' (of data between tiles 4 on different chips 2). The program flow in Figure 4 illustrates the program flow for a first processing unit 2i and a second processing unit 2ii.

[0046] As shown in FIG. 4, internal BSP supersteps (including internal exchange 50 of data between tiles 4 on the same chip 2) are kept separate from external synchronization and exchange (including external exchange 50' of data between tiles 4 on different chips 2).

[0047] The program may be arranged to execute a sequence of synchronization, exchange phases and computation phases including, in order, (i) a first computation phase, then (ii) an internal barrier synchronization 30, then (iii) an internal exchange phase 50, then (iv) an external barrier synchronization 80, then (v) an external exchange phase 50'. The external barrier 80 is imposed after the internal exchange phase 50, so that the program proceeds to the external exchange 50' only after the internal exchange 50. Note also that a computation phase may optionally be included between the internal exchange (iii) and the external barrier (iv), as shown for chip 2I in FIG.

[0048] This entire sequence is performed by a program (e.g., generated as such by a compiler). In an embodiment, the program is programmed to do so by a SYNC instruction executed by the tile 4. The internal synchronization and exchange does not extend to any tiles or other entities on another chip 2. The sequence (i)-(v) (with any computation phases described above between iii and iv) can be repeated in a series of full iterations. For each iteration, there can be multiple instances of the internal computation, synchronization and exchange (i)-(iii) before the external synchronization and exchange. That is, multiple instances (preserving the order) of (i)-(iii), i.e. multiple internal BSP supersteps, can be performed before (iv)-(v), i.e. the external synchronization and exchange. Note also that any tile 4 can each perform their own instances of the internal synchronization and exchange (ii)-(iii) in parallel with other tiles 4.

[0049] Thus, for every complete BSP cycle (i)-(v), there may be at least a portion of cycles (ii)-(iii), subject to the constraint that synchronization is performed only internally (i.e., only on-chip).

[0050] External exchange 50'Note that during ,communication is not limited to only external: some tiles may only perform internal exchanges, some tiles may only perform external exchanges, and some tiles may perform a mix of both.

[0051] 4, some tiles 4 may perform local input / output during the computation phase, for example, they may exchange data with a host or other type of external storage device.

[0052] Also note that, in general, any or all tiles may have a null computation phase 52 or a null exchange phase 50 in any given BSP superstep, as shown in FIG.

[0053] Each of the synchronization barriers shown in FIG. 4 is passed by a tile 4 upon completion of an exchange of synchronization request and acknowledgement transmissions. In the case of internal barrier synchronization, each tile 4 of the processing unit 2 sends an internal synchronization request to the internal synchronization controller 36 (see FIG. 1) of the interconnect 34 upon reaching an internal barrier synchronization. When the internal synchronization controller 36 receives synchronization requests from all of the tiles 4 that are part of the processing unit 2, the synchronization controller 36 returns a synchronization acknowledgement to each of the tiles 4. Upon receiving the synchronization acknowledgement, each of the tiles 4 enters an exchange phase and data exchange between the tiles 4 occurs. This technique ensures that each tile 4 must reach the barrier before any of the tiles can proceed to the exchange phase.

[0054] In the case of external barrier synchronization, the exchange of synchronization requests and acknowledgments takes place between a group of processing units 2, called a synchronization group. Following the exchange of synchronization requests and acknowledgments for external barrier synchronization, the processing units 2 exchange data during an exchange phase.

[0055] Each of the tiles 4 on a processing unit 2 issues an external synchronization request to the external synchronization logic (not shown in FIG. 1 ) associated with the processing unit 2 when it reaches the external barrier synchronization. When the external synchronization logic receives synchronization requests from all of the tiles 4 on the processing unit 2, it either acknowledges the synchronization requests or propagates the synchronization requests to the external synchronization logic associated with another processing unit 2. The action taken by the external synchronization logic of another processing unit 2 in response to a synchronization request depends on whether the logic is defined as the master of the synchronization group or as a propagation node of that group. The propagation node propagates the received synchronization requests towards the master defined for the synchronization group. When the synchronization master receives an external synchronization request for each of the processing units 2 that are part of the synchronization group, it returns a synchronization acknowledgement to the external synchronization logic associated with each of the other processing units 2 of the synchronization group. The synchronization master also returns a synchronization acknowledgement to each of the tiles 4 of its own processing unit 2. Upon receiving the synchronization acknowledgement, each external synchronization logic (ie, propagation node) of the other processing units 2 of the synchronization group returns the synchronization acknowledgement to the tile 4 of its own processing unit 2 .

[0056] Referring to FIG. 5, FIG. 5 illustrates an example of an exchange of synchronization requests performed to an external barrier synchronization for the exchange of data between processing units 2. The illustrated system 500 includes three processing units 2a, 2b, 2c (collectively referred to as processing units 2) that are part of a synchronization group. Each of the three processing units 2 belongs to a separate integrated circuit 510a, 510b, 510c (collectively referred to as chips 510). Each of the chips 510 includes external synchronization logic 520, referred to as a global synchronization peripheral (GSP) 520. In some cases, the external synchronization logic 520 may be located off-chip (e.g., on a gateway device).

[0057] When each tile 4 reaches the barrier, it issues a synchronization request to the associated GSP 520. Although in FIG. 5 only one tile 4 of each processing unit 2 is shown sending a synchronization request to the associated GSP 520 for simplicity, in practice all tiles 4 on the chip will issue synchronization requests to the GSP 520. In some embodiments, each synchronization request includes an indication of the synchronization group to be used for the barrier synchronization, and the GSP 520 propagates and / or acknowledges the received synchronization request according to the indication of the request. In other embodiments, the writing of the synchronization request to the GSP 520 is preceded by a writing from the nominated tile 4 of the associated processing unit 2 of the synchronization group to be used for the upcoming barrier synchronization. The indication of the synchronization group is used to select a configuration setting for the GSP 520. When the barrier synchronization is reached, the GSP 520 propagates and / or acknowledges the received synchronization request according to the selected configuration setting.

[0058] In this example, GSPs 520a, 520c are configured as propagation nodes. After receiving synchronization requests from all tiles 4 of processing unit 2, each of GSPs 520a, 520c propagates the synchronization requests upstream towards the synchronization master (i.e., GSP 520b) defined for the synchronization group. After receiving synchronization requests from GSPs 520a, 520c and from all tiles 4 of the associated processing unit 2b, GSP 520b issues a synchronization acknowledgement downstream towards the tile 4 of processing unit 2b and towards GSPs 520a, 520c. GSPs 520a, 520c respond to the received acknowledgement by issuing a synchronization acknowledgement to the tile 4 of the respective processing unit 2a, 2c.

[0059] In response to receiving the synchronization acknowledgement, the tile 4 passes through the barrier synchronization and exchanges data with other processing units 2 of the synchronization group during an exchange phase. This exchange of data between different processing units 2 is performed in a non-time deterministic manner, as described in our earlier application, U.S. Patent Application No. 15 / 886,065.

[0060] A different synchronization group can be defined for the exchange of data at each external barrier synchronization. These synchronization groups are programmed into the local programs running on the processing units. Tile 4 executes a synchronization instruction to generate a synchronization request at the barrier synchronization. The synchronization instruction takes a synchronization group as an operand and controls between which processing units 2 each synchronization occurs.

[0061] Referring to FIG. 6A, FIG. 6A shows a number of different possible external synchronization groups (e.g., group_1 or group_2). In an embodiment, these correspond to different hierarchical levels. That is, each higher hierarchical level 92 (e.g., group_2) encompasses two or more groups 91A, 91B of at least one lower hierarchical level. In an embodiment, there are only two hierarchical levels, but it is not excluded that there are more nesting levels. To initiate barrier synchronization, an execution unit of tile 4 of processing unit 2 executes a SYNC instruction. The operand of the SYNC instruction indicates a synchronization group. If the operand of the SYNC instruction is set to a lower hierarchical level of the external synchronization group (SYNC group_1), the synchronization and aggregation operations described above are performed in relation to tile 4 on chip 2 only in the external synchronization group of the same lower level as the tile for which the SYNC was executed. On the other hand, if the operand of the SYNC instruction is set to a higher hierarchical level of the external synchronization group (SYNC group_2), the synchronization and aggregation operations described above are automatically performed with respect to all tiles 4 on all chips 2 in the same high-level external synchronization group as the tile on which the SYNC was performed.

[0062] In response to a SYNC instruction opcode having an external synchronization group as an operand, each tile 4 in that synchronization group causes the synchronization level specified by the operand to be signaled to the associated GSP 520. In response, the GSP 520 processes synchronization requests (sync_req) and acknowledgments (sync_ack) to occur only among the tiles 4 of the signaled group.

[0063] It should be noted that in other embodiments, the synchronous groups that can be specified by the mode of the SYNC instruction are not necessarily hierarchical in nature. In general, the SYNC instruction can be provided with a mode that accommodates any kind of group. For example, the mode can allow selection only among non-hierarchical groups or among a mixture of hierarchical groups and one or more non-hierarchical groups (where at least one group is not completely nested within another). This advantageously allows the programmer or compiler flexibility to select between different layouts of internally synchronous groups that are asynchronous with each other with minimal code density.

[0064] In addition to external barrier synchronization being provided for exchange between processing units 2, external barrier synchronization may be performed to exchange data between processing units 2 and a host system. Such data may include application data provided by the host system for processing by processing unit 2. The data may include results of processing by processing unit 2 that are provided to the host system. This type of barrier synchronization is referred to as barrier synchronization with host involvement.

[0065] The relationship of the barrier synchronization with host participation to other barrier synchronizations is shown in FIG. 6B. This figure shows a barrier synchronization participated in by an exemplary processing unit 2. The processing unit 2 can participate in P barrier synchronizations 80 before a barrier 90 is imposed, which requires a synchronization acknowledgement from the host. The P barrier synchronizations require synchronization requests from all (non-pausing) tiles 4 of the associated synchronization group. A subsequent barrier synchronization 90 requires synchronization requests from all (non-pausing) tiles 4 of the processing unit 2 and requires that the host has previously indicated permission to pass the particular barrier. After the barrier 90, an exchange 50" can be performed between the host and one or more tiles 4 of the processing unit 2 (e.g., one or more tiles 4 to report computation results to the host).

[0066] This barrier synchronization and associated data exchange is performed using a network interface device for connecting between the processing unit 2 and the host system. The network interface device is described as being a gateway device, but could be another type of network interface device. Gateway devices are described in detail in prior U.S. patent application Ser. No. 16 / 235,109, which is incorporated by reference.

[0067] Referring to FIG. 7, FIG. 7 shows an example of a data processing system 700, in which a processing unit 2 includes: Gateway 710 Via Host System 720 In this example, a single Gateway 710 7 is configured to connect to two processing units 2 via a network 740. However, in other embodiments, Gateway 710 Each of the processing units 740 may be connected to a different number (eg, one) of processing units 2 via a network 740 .

[0068] The host system 720 includes a buffer 725 associated with each processing unit 2. Each of the buffers 725 has an associated buffer 730 on the gateway 710. The buffer 730 on the gateway 710 is a mirror of the associated buffer 725 on the host 720. Read and write requests are issued over the network 740 to synchronize the data in the host buffer 725 with the data in the gateway buffer 730. The read and write requests may be RDMA read and write requests. One or more RDMA controllers 735 are present in the host 720 to issue RDMA reads / writes to the gateway 710 to perform data transfers between the buffers 725 and 730. Similarly, an RDMA controller 736 is present in the gateway 710 to perform data transfers between the buffers 730 and 725. The RDMA exchanges occur over the network 740. In an embodiment, the network 740 is an Ethernet network 740. In this case, data is transmitted over network 740 via RDMA over Converged Ethernet (RoCE).

[0069] While the primary example discussed herein is transmitting data over RoCE, in other examples, different networking technologies can be used. For example, data can be transmitted over network 740 using Data Plane Development Kit (DPDK) over regular Ethernet. However, the use of RDMA has the advantage of lower latency. In embodiments where a different protocol other than RDMA is used, the protocols shown in Figures 7 and 9 may be used. RDMA Controller 735, 736 may be other types of processing circuitry used to perform the data transfer operations described.

[0070] The processing units 2 are configured to initiate a barrier synchronization to exchange data with their associated memory 730 on the gateway 710 at predefined points in their assigned programs. In such a predefined barrier synchronization, one or both of the processing units 2 send a synchronization request to the associated external synchronization logic 745 of the gateway 710. The external synchronization logic 745 is shown as a local synchronization propagation module (LSPM) 745. Upon receiving a synchronization request, each LSPM 745 is configured to acknowledge the synchronization request. Upon receiving such synchronization acknowledgement, the processing unit 2 performs a data exchange with its associated memory 730 on the gateway 710. The memory 730 may be a memory mapped to the memory of the processing unit 2. In this case, the processing unit 2 issues memory mapped I / O (MMIO) reads / writes to send and receive data from the memory 730. During the exchange phase, the processing unit 2 can send results of its processing generated in a previous computation phase to the memory 730, or receive workload from the memory 730 for processing in a subsequent computation phase. Synchronization requests, acknowledgements and data exchanges are performed over the PCIe link between the processing unit 2 and the gateway 710.

[0071] Although only two processing units 2 are shown in FIG. 7 as communicating with the host 720, embodiments of the present invention allow a large number of processing units to communicate with the host 720. Referring to FIG. 8, FIG. 8 shows an example of a system 800 including multiple processing units 2 communicating with a single host 720. The processing units 2 are configured to execute a distributed application together by each executing a local program provided by the host 720. The processing units exchange data with the host via a buffer 730 and with other processing units at predefined synchronization points of the local program. Data exchange between the processing units 2 can occur via a PCIe link between the processing units 2 or via a PCIe link between the gateways 710. Data exchange between the gateways 710 and the host 720 occurs via a connection on the network 740. There is a separate connection for each processing unit 2. Each of these connections can be an RDMA connection.

[0072] Referring now to Figure 9, this shows in further detail the interaction between the host system 720 and one of the processing units 2, describing the compilation and provision of a program to the processing unit 2 and the subsequent data exchange resulting from the execution of the program.

[0073] As shown, the host 720 includes at least one processor 722 configured to compile a set of programs for execution on the processing unit 2. Processor 910 A local program generated from the source code for an application together with the local program constitutes a compiled application. Each of the programs is in the form of an executable image. At least one Processor 910 receives a set of source code and generates a set of local programs from the source code based on a compiler program. The compilation process includes generating separate local programs and allocating them across the processing units 2. As part of the compilation process to generate the local programs, at least one Processor 910 inserts a set of exchange synchronization points into each local program. At each of these points in the execution of the local program, the respective processing unit 2 is configured to participate in a barrier synchronization. Some of the barrier synchronizations are internal barrier synchronizations as described above. Some of the barrier synchronizations are external barrier synchronizations for exchanging data with one or more other processing units 2 involved in executing the application. Some of the barrier synchronizations are external barrier synchronizations for exchanging data with the host 720 via an attached gateway 710.

[0074] After the host 720 compiles the local program for processing unit 2, the program is stored in control program buffer 920. The host has a separate such control program buffer for each processing unit 2. The comments regarding control program buffer 920 apply equally well to these other control program buffers. Buffers 920 and 930 are mirrors of each other, as are buffers 725, 730 used to exchange data between the host 720 and the gateway 710. The local program is transferred from buffer 920 to buffer 930 in one or more writes over the network 740. This transfer can be made by one or more RDMA controllers 736 configured to transfer the local program from buffer 920 to buffer 930 in one or more RDMA writes over the network 740. Processing unit 2 is configured to issue a read to gateway 710 to read the program from control program buffer 930. Processing unit 2 includes a small amount of secondary boot loader code to perform such a read. The read is an MMIO read.

[0075] Once processing unit 2 has loaded its local programs from buffer 930, it executes the programs. To process workloads, the programs must first receive them from host 720. Host 720 stores the workloads required by the programs during their execution. Host 720 stores these workloads in buffer 725 before making them available to processing unit 2. Data (including the workloads) is split into different streams, and processing unit 2 can select which streams to read and write depending on the outcome of different conditions in the execution of its local programs.

[0076] Each stream may include, for example, an image sequence (individual still images or video frames), an audio stream of one or more audio samples, or any other sequence of experiential data (e.g., online shopping habits, etc.) being sent from the host subsystem 720 to the processing unit 2 for use as input to a machine learning or machine intelligence algorithm running on multiple processing units 2. For example, this may be for training a neural network based on the streaming data, or for making inferences from the data based on an already trained neural net. In an example, if the stream is in the direction from processing unit 2 to the host subsystem 720, the stream may include a stream of inferences resulting from a machine intelligence algorithm, such as a trained neural net, running on the processing unit 2.

[0077] The buffers 725, 730 are multi-stream buffers 725, 730 used to store data of multiple different streams in preparation for transfer. The buffers 725, 730 are also bidirectional, and data of each stream can be both written and read from each buffer 725, 730. The processing unit 2 can selectively read / write from different streams by providing the stream's identifier to the gateway. After the local program is loaded into the processing unit 2, the processing unit 2 executes the program and issues a write of the identifier of the stream to be read to the processing unit 2. The stream's identifier includes one or more of the index or address of the buffer 725, 730. Rather than explicitly identifying the stream, the stream's identifier can identify one of the memory areas of the buffers 725, 730 shown in FIG. 9 (each of which is associated with one of the streams). The gateway 710 receives the stream's identifier, which is temporarily stored in the stream ID store 940. In response to receiving the identifier of the stream, the gateway 710 writes the identifier to the host 720 over the network 740. In response to receiving the identifier, the host 720 writes the data of the identified stream from the buffer 725 to the buffer 730 over the network 740. The processing unit 2 then reads the data from the buffer 730.

[0078] After getting the workload from buffer 730, processing unit 2 processes the workload according to its local program. At some point during its execution, processing unit 2 writes a representation of the stream, which is read or written in the exchange phase following the upcoming barrier synchronization. This writing can occur during processing unit 2's computation phase or during a previous exchange phase. Then, during the execution of the program, a synchronization barrier is reached. At this point, processing unit 2 performs the synchronization request / acknowledge procedure described above with respect to FIG. 7. Following this procedure, processing unit 2 reads and writes from buffer 730, whereupon: Processing Unit 2 reads from and writes to the memory area of ​​the buffer corresponding to the stream identified by the provided stream indication. In response to writing data to that stream, gateway 710 writes the data to buffer 725. Thus, during the exchange phase, host 720 receives results of processing of workload data from processing unit 2, and processing unit 2 receives further workload data from host 720.

[0079] Referring to FIG. 10, FIG. 10 shows a message sequence diagram illustrating the exchange of messages between the host 720, the gateway 710, and the processing unit 2. As shown in FIG. 10, in step 1, the processing unit 2 writes an identifier of a stream to the gateway. The gateway receives this stream identifier and writes it to the host 720. In step 2, in response to receiving the stream ID, the host 720 writes a workload to the gateway 710. As shown, a barrier synchronization is reached by the processing unit 2. Following the barrier synchronization, the processing unit 2 reads from the gateway 710 the workload provided by the host 720 in step 2. And in step 3, the processing unit 2 writes the results obtained from processing the previous workload.

[0080] In step 4, processing unit 2 writes an identifier of the stream written by processing unit 2 in step 3 to the gateway 710. In response to receiving the stream identifier, the gateway 710 writes the data written by processing unit 2 in step 3 to the host.

[0081] Following that data exchange, the processing unit 2 proceeds to further computation phases. The process for data exchange shown in Figure 10 can be carried out multiple times.

[0082] In Figure 10, Gateway 710 and Host System 720 In the embodiment shown, the processing unit 2 sends a control message MemWr to initiate a data transfer between the Host System 720 However, this data transfer can be initiated by periodically synchronizing buffers 725 and 730.

[0083] 11, which illustrates an exemplary method 1100 according to an embodiment of the present application. The figure provides an overview of the method 1100 performed by a system including a host, a network interface device (i.e., a gateway as described in the above embodiment), and a processing unit each configured to run an application by executing a local program.

[0084] In S1110, the host transfers the local program to the processing unit.

[0085] At S1120, each processing unit issues a request for workload data from a designated stream. The stream designated by each processing unit depends on the local program assigned to that processing unit. The request, along with the designated stream, is provided to the host via the network interface device.

[0086] At S1130, in response to receiving each of the requests, the host writes data from the identified stream. The host writes this data to the network interface device for transmission to the processing unit. The processing unit reads the data from the network interface device memory.

[0087] At S1140, the processing units execute calculations during a calculation phase to perform calculations using the workload data received at S1130. The calculations to be performed are specified in a local program provided to each processing unit.

[0088] In S1150, the processing units exchange the results of their data processing during S1140. This step is performed using barrier synchronization between synchronous groups of processing units. The time of such barrier synchronization and the definition of the synchronous groups are specified in local programs provided to each of the processing units.

[0089] Following each such external barrier synchronization at S1150, the method returns to S1140 where further computations are performed during further computation phases.

[0090] At a later point in the execution of the local program, the processing unit proceeds to S1160, where the process begins to initiate a data transfer with the host. This step may overlap with S1140 or S1150.

[0091] At S1160, the processing unit provides the stream identifier to the host via the network interface device.

[0092] In S1170, the host writes the data of the identified stream to the network interface device.

[0093] At S1180, following a pre-compiled synchronization barrier, the processing unit reads and writes data for the identified stream from / to the network interface device, which writes the data to the host.

[0094] It is not necessary for each processing unit to execute steps S1140 to S1180 simultaneously, and some processing units 2 may perform calculations in S1140 or exchange data in S1150 while one or more other processing units 2 are exchanging data with the host in S1180. Except when barrier synchronization is performed between groups of processing units 2 in S1150, the processing units operate asynchronously with each other. The order and timing of the steps performed by each processing unit depends on the local program that has been assigned to the processing units by the host.

Claims

1. A host system configured to be connected via a network to a plurality of subsystems each of which operates as a work accelerator for the host system according to a local program executed in each of the plurality of subsystems, the host system includes at least one processor and at least one memory storing computer readable instructions; At least one processor of the host system is configured to execute the computer readable instructions to compile a set of local programs, each local program to be executed in one of a plurality of subsystems, and compiling source code includes defining a set of synchronous groups, each synchronous group including at least some of the subsystems; Each local program: a set of operational instructions for performing operational operations during one or more of a plurality of operational phases of the subsystem, when executed by at least one processor of the subsystem; a set of data transfer instructions, when executed by at least one processor of said subsystem, for performing a plurality of barrier synchronizations at predefined points in the execution of said local program, at least some of said barrier synchronizations being defined between one of said synchronization groups, at least one of said barrier synchronizations being a synchronization requiring host participation, and in response to a data transfer initiator being provided by said subsystem to a host system following a synchronization requiring host participation, data exchange is performed between said subsystem and said host system; at least one processor of a host system is configured to, in response to each received data transfer initiator, exchange data over the network with one of the subsystems from which the initiator originates; Host system.

2. 2. The host system of claim 1, wherein the exchange of data over the network with one of the subsystems from which the initiator originates is performed using remote direct memory access.

3. 3. The host system of claim 1, wherein data transmission between the subsystem and the host system is performed using RDMA.

4. The step of exchanging data over the network with one of the subsystems that is a source of the initiator, comprising: A host system as claimed in any one of claims 1 to 3, comprising synchronising at least a portion of a buffer of the host system with at least a portion of a buffer of a network interface device accessible to one of the subsystems from which the initiator is provided.

5. Each of the received data transfer initiators includes an identifier of a stream between the host system and the one of the subsystems that provided the identifier; A host system as claimed in any one of claims 1 to 4, wherein the step of exchanging data over the network with one of the subsystems from which the identifier is provided comprises sending and / or receiving data belonging to each identified stream.

6. at least a portion of the host system buffer includes a portion of the buffer associated with the identified stream; each of the received data transfer initiators includes an identifier of a stream between the host system and the one of the subsystems from which the identifier was provided; 5. The host system of claim 4, wherein at least a portion of a buffer of a network interface device accessible to one of the subsystems from which the identifier is provided includes a portion of a buffer associated with the identified stream.

7. said step of exchanging data over said network with one of said subsystems which is a source of said initiator, said step comprising: transmitting data to a network interface device accessible by the subsystem providing the initiator prior to one of a plurality of barrier synchronizations defined in the local program of the subsystem; receiving data from the network interface device accessible to the subsystem from which the initiator originates, following the one of a plurality of barrier synchronizations defined in the local program of the subsystem from which the initiator originates; The host system according to any one of claims 1 to 6, comprising:

8. A host system according to any preceding claim, wherein at least one processor of the host system is configured to cause each of the local programs to be dispatched over the network to a respective subsystem.

9. A subsystem configured to be connected to a host system via a network, configured to operate as a work accelerator for the host system according to a local program executing on at least one processor of the subsystem; an interface configured to receive the local program from the host system over the network; The local program is a representation of a set of synchronization groups, at least some of the synchronization groups including the subsystem and one or more further subsystems to act as accelerators for a host; a set of operational instructions for performing a plurality of operational operations during one or more of a plurality of operational phases of the subsystem when executed by at least one processor; a set of data transfer instructions for, when executed by at least one processor, performing a plurality of barrier synchronizations at predefined points in the execution of a respective local program; at least some of the barrier synchronizations are defined between one subsystem of the synchronization group, at least one of the barrier synchronizations is a synchronization requiring host participation, and following the synchronization requiring host participation, a data transfer initiator is sent by the subsystem to the host system, followed by data exchange over the network between the subsystem from which the data transfer initiator was sent and the host system. Subsystem.

10. 10. The subsystem of claim 9, wherein the data exchange between the subsystem and the host system is performed using remote direct memory access.

11. A subsystem according to claim 9 or 10, wherein data exchange between the subsystem and the host system is performed using RDMA.

12. The data exchange between the subsystem and the host system is at least one processor causing data exchange with a buffer of a network interface device for the subsystem; A subsystem according to any one of claims 9 to 11, wherein at least a portion of the buffer is synchronized with at least a portion of the buffer of the host system.

13. the initiator of a data transfer includes an identifier of a stream between the subsystem and the host system; A subsystem according to any one of claims 9 to 12, wherein data exchange between the subsystem and the host system comprises exchanging data belonging to each identified stream.

14. 14. The subsystem of claim 13 when dependent on claim 12, wherein at least a portion of the buffer of the network interface device includes a portion of the buffer associated with the identified stream.

15. data exchange between said subsystem and said host system; transmitting data to a network interface device accessible by the subsystem from which the initiator originates, prior to one of a plurality of barrier synchronizations defined in the local program of the subsystem; receiving data from the network interface device accessible to the subsystem from which the initiator originates following one of a plurality of barrier synchronizations defined in the local program of the subsystem; The subsystem of any one of claims 9 to 14, comprising:

16. The subsystem of any one of claims 9 to 15, wherein the interface is configured to receive the local program from the host system over the network.

17. 1. A method implemented in a host system, comprising: connecting via a network to a plurality of subsystems that operate as work accelerators for the host system according to a local program executed in each of the plurality of subsystems; compiling a set of local programs to be executed in one of the plurality of subsystems; The step of compiling the source code includes defining a set of synchronization groups including at least some of the subsystems; Each local program: a set of operational instructions for performing operational operations during one or more of a plurality of operational phases of the subsystem, when executed by at least one processor of the subsystem; a set of data transfer instructions, when executed by at least one processor of said subsystems, for performing a plurality of barrier synchronizations at predefined points in the execution of said local program, at least some of said barrier synchronizations being defined between one subsystem of said synchronization group, at least one of said barrier synchronizations being a synchronization requiring host participation, and following a synchronization requiring host participation, data exchange is performed between said subsystem and said host system in response to an initiator of a data transfer being provided by said subsystem to said host system; in response to each received data transfer initiator, exchanging data over the network with one of the subsystems from which the initiator originates; method.

18. 20. A computer program product causing at least one processor of a host system to carry out the method of claim 17.

19. 1. A method implemented in a subsystem for a host system, comprising: connecting to a host system via a network; receiving a local program from the host system over the network; operating as a work accelerator for the host system according to a local program executing on at least one processor of the subsystem and including an indication of a set of synchronization groups including the subsystem and one or more further subsystems for operating as accelerators for a host; executing operational instructions of the local program to perform operational operations during one or more of a plurality of operational phases of the subsystem; executing data transfer instructions of the local program at predefined times in the execution of the local program to perform a plurality of barrier synchronizations, the synchronizations being defined between one subsystem of the synchronization group and requiring host involvement; sending a data transfer initiator by said subsystem to said host system following a synchronization requiring host involvement, followed by performing a data exchange over said network between the subsystem from which the data transfer initiator was sent and said host system; A method comprising:

20. 20. A computer program product causing at least one processor of a subsystem to carry out the method of claim 19.

Images