Communication between the host and the accelerator over the network
Patent Information
- Application Number
- JP2023502602
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-07-14
- Filing Date
- 2020-12-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-12-22
AI Technical Summary
The challenge of connecting a large number of accelerator subsystems to a host system over PCIe is limited, leading to difficulties in data exchange and scalability for large applications, especially when workload dependencies require synchronization between subsystems.
A host system connected via a network to multiple accelerator subsystems, implementing synchronization groups and barrier synchronizations at predefined points in the execution of local programs, with data exchange occurring through remote direct memory access (RDMA) over Ethernet networks.
Enables the connection of a large number of subsystems while minimizing host interactions, facilitating efficient data exchange and scalability by defining synchronization groups and using RDMA for data transfer.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] This application relates to host systems and processing units that communicate over a network.
Background Art
[0002] In the context of data processing for complex or large-scale applications, a work accelerator can be a subsystem in which the processing of certain data is offloaded from a host system. Such a work accelerator can have hardware specialized for the execution of a particular type of processing.
[0003] As an example, one area of computing in which such specialized accelerator subsystems can be useful is found in machine intelligence. As is well known to those skilled in the art of machine intelligence, machine intelligence algorithms center around the execution of iterative updates of a "knowledge model" that can be represented by a graph of multiple interconnected nodes. The implementation of each node involves data processing, and the interconnections of the graph correspond to the data exchanged between nodes. Typically, at least some of the processing of each node can be performed independently of some or all of the other nodes of the graph, and thus large-scale graphs present excellent opportunities for multithreading. Thus, a work accelerator specialized for machine intelligence applications can involve massive multithreading. One form of parallel processing can be achieved by a processor that includes an array of multiple tiles on the same chip (i.e., the same die), each tile including a separate processing unit as well as memory (including program memory and data memory). Thus, separate portions of program code can be executed in parallel on different tiles. The tiles are connected to each other via on-chip interconnects, thereby enabling the exchange of data between the tiles. Such an accelerator can function as a subsystem for a host system to perform parallel processing of a provided data set.
[0004] In general, dependencies can exist between parts of a program running on different tiles. Therefore, techniques are needed to prevent dependent data of a code snippet on one tile from becoming available to another code snippet on another tile. Many schemes exist to achieve this, and in this specification, one such scheme, Bulk Synchronous Parallelism ("BSP"), is described as an example. According to BSP, each tile performs an arithmetic phase and an exchange phase in an alternating cycle. During the arithmetic phase, each tile performs one or more arithmetic tasks locally on the tile, but does not communicate the results of its arithmetic with any other tile. In the exchange phase, each tile can exchange one or more results of arithmetic from the preceding arithmetic phase with one or more other tiles in the group, but does not yet proceed to the next arithmetic phase. Furthermore, according to the BSP principle, barrier synchronization is placed at the junctions where the arithmetic phase transitions to the exchange phase, the junctions where the exchange phase transitions to the arithmetic phase, or both. That is, (a) all tiles in a group must complete their respective calculation phases before any tile in the group can proceed to the next exchange phase, (b) all tiles in a group must complete their respective exchange phases before any tile in the group can proceed to the next calculation phase, or (c) both. In some scenarios, a tile performing a calculation may communicate with other system resources, such as a network card or storage disk, as long as this does not involve communication with other tiles in the group.
[0005] During the exchange phase, data exchange may occur not only internally within the accelerator (i.e., between tiles), but also, in some situations, between an accelerator and further accelerators, or between an accelerator and the host system. When a subsystem acts as a work accelerator, it is configured to process datasets provided (for example, from the host system or another form of storage in the expansion system). [Overview of the project] [Problems that the invention aims to solve]
[0006] To provide workloads to one or more subsystems from a host, the host can connect locally to those subsystems over a PCIe interface. This enables low-latency data transfer between the host and one or more subsystems. However, one problem is that there is a limit to the number of subsystems that a host can connect directly over a PCIe connection. Therefore, when attempting to increase the number of subsystems to run larger applications, several challenges arise as there are issues with how to achieve data exchange between those subsystems and the host. [Means for solving the problem]
[0007] To handle larger-scale applications, it is desirable to connect more accelerator subsystems to the host, which can act as work accelerators. One proposed way to achieve this is to connect the host to the accelerator subsystems over a network. There is a delay between the host device receiving the results from calculations performed by subsystems and the host device responding by providing further workloads to those subsystems. This is especially true when those workloads may depend on the results output by other subsystems. In some cases, the host must receive results from one subsystem and provide those results to another subsystem.
[0008] According to a first aspect, a host system is configured to be connected via a network to a plurality of subsystems that act as work accelerators to a host system according to local programs executed in each of the subsystems, the host system comprising at least one processor and at least one memory for storing computer-readable instructions, the at least one processor of the host system being configured to execute computer-readable instructions to compile a set of local programs, each local program being executed in each of the plurality of subsystems, the step of compiling source code comprising defining a set of synchronization groups, each synchronization group comprising at least some subsystems, and each local program being executed by at least one processor of each subsystem, with respect to multiple operations of the subsystems. A host system is provided, comprising an arithmetic instruction set for performing arithmetic operations during one or more of the arithmetic phases, and a data transfer instruction set for performing a plurality of barrier synchronizations at predefined points in the execution of a local program when executed by at least one processor of the subsystem, each of at least some of the barrier synchronizations being defined between one of the synchronization groups, at least one of the barrier synchronizations being a host-involved synchronization, and following the host-involved synchronization, data exchange is performed between each subsystem and the host system in response to a data transfer initiation program being provided to the host system by each subsystem, and at least one processor of the host system is configured to exchange data over the network with one of the subsystems that is the source of the identifier in response to each of the received data transfer initiation programs.
[0009] By defining synchronization behavior on the host and then providing that information to subsystems, the host can service the majority of subsystems. The definition of synchronization behavior includes defining synchronization groups where synchronization barriers occur and defining the points at which data exchange with the host occurs during program execution. By defining synchronization behavior between subsystems, the majority of subsystems can be connected while minimizing the necessary exchanges with the host.
[0010] In some embodiments, data exchange over the network with one of the subsystems that receives the identifier is performed using remote direct memory access.
[0011] In some embodiments, the network is an Ethernet® network.
[0012] In some embodiments, the step of exchanging data over a network with one of the subsystems that receive the identifier includes synchronizing at least a portion of the host system's buffers with at least a portion of the buffers of a network interface device that can be accessed by one of the subsystems that receive the identifier.
[0013] In some embodiments, each received initiation program for data transfer includes a stream identifier between the host subsystem and one of the subsystems that is the source of the identifier, and the step of exchanging data over the network with one of the subsystems that is the source of the identifier includes transmitting and / or receiving data belonging to the respective identified stream.
[0014] In some embodiments, at least a portion of the host system's buffers includes a portion of the buffer associated with the identified stream, and at least a portion of the buffers of a network interface device accessible to one of the subsystems that receive the identifier includes a portion of the buffer associated with the identified stream.
[0015] In some embodiments, the step of exchanging data over a network with one of the subsystems that receive an identifier includes sending data to a network interface device accessible by that subsystem before one of a plurality of barrier synchronizations defined in the local program of that subsystem, and receiving data from the network interface device accessible by that subsystem after one of a plurality of barrier synchronizations defined in the local program of that subsystem.
[0016] In some embodiments, at least one processor in the host system is configured to send each of the local programs to its respective subsystem over the network.
[0017] According to a second aspect, a subsystem is provided which is configured to be connected to a host system via a network, and is configured to act as a work accelerator to the host system according to a local program executed on at least one processor of the subsystem, and includes an interface configured to receive the local program from the host system over the network, wherein the local program includes: a representation of a set of synchronization groups, where at least some of the synchronization groups include one or more further subsystems to act as accelerators to the subsystem and the host; an arithmetic instruction set for performing a plurality of arithmetic operations between one or more of a plurality of arithmetic phases of the subsystem when executed by at least one processor; and a data transfer instruction set for performing a plurality of barrier synchronizations at predefined points in the execution of each local program when executed by at least one processor, where at least some of the barrier synchronizations are defined between one of the subsystems of the synchronization group, and at least one of the barrier synchronizations is a synchronization requiring host involvement, followed by the subsystem sending a data transfer initiation program to the host system, after which data exchange takes place between the subsystem and the host system over the network.
[0018] In some embodiments, data exchange between the subsystem and the host system is performed using remote direct memory access.
[0019] In some embodiments, the network is an Ethernet® network.
[0020] In some embodiments, data exchange between a subsystem and a host system includes causing at least one processor to exchange data with a buffer of a network interface device for the subsystem, with at least a portion of the buffer being synchronized with at least a portion of the host system's buffer.
[0021] In some embodiments, the data transfer initiation program includes an identifier for the stream between the subsystem and the host subsystem, and the data exchange between the subsystem and the host system includes the exchange of data belonging to the respective identified streams.
[0022] In some embodiments, at least a portion of the buffer of the network interface device includes a portion of the buffer associated with the identified stream.
[0023] In some embodiments, data exchange between a subsystem and a host system includes sending data to a network interface device accessible to the subsystem before one of a plurality of barrier synchronizations defined in the subsystem's local program that receives the identifier, and receiving data from the network interface device accessible to the subsystem after one of a plurality of barrier synchronizations defined in the subsystem's local program that receives the identifier.
[0024] In some embodiments, the interface is configured to receive local programs from a host system over a network.
[0025] According to a third aspect, a method implemented in a host system, including connecting a plurality of subsystems that operate as a work accelerator for the host system according to local programs executed in each of the one or more subsystems via a network, including compiling a set of local programs executed in each of the one or more subsystems, the step of compiling the source code includes defining a set of synchronization groups including at least some of the subsystems, each local program includes an instruction set for performing arithmetic operations during one or more of a plurality of arithmetic phases of the subsystem when executed by at least one processor of the subsystem, and a data transfer instruction set for performing a plurality of barrier synchronizations at a predefined point in the execution of the local program when executed by at least one processor of the subsystem, at least some of the barrier synchronizations are defined between subsystems of one of the synchronization groups, at least one of the barrier synchronizations is a synchronization that requires host participation, following the synchronization that requires host participation, in response to the start program for data transfer being provided to the host system by each subsystem, data exchange is performed between each subsystem and the host system, and in response to each of the received start programs for data transfer, it includes exchanging data on the network between one of the subsystems that is the recipient of the identifier. A method is provided.
[0026] According to a fourth aspect, a computer program is provided that causes at least one processor of a host system to execute the method according to the third aspect.
[0027] According to a fifth aspect, a non-temporary computer-readable medium storing the computer program according to the fourth aspect is provided.
[0028] According to a sixth aspect, there is provided a method implemented in a subsystem for a host system, the method comprising: connecting to a host system via a network; receiving a local program from the host system over the network; operating as a work accelerator for the host system according to the local program, the local program including a display of a set of synchronization groups including one or more further subsystems that are executed on at least one processor of the subsystem and operate as accelerators for the subsystem and the host; executing arithmetic instructions of the local program to perform arithmetic operations during one or more of a plurality of arithmetic phases of the subsystem; executing data transfer instructions of the local program to perform a plurality of barrier synchronizations defined between one of the subsystems of the synchronization group and requiring host involvement at a predefined point in the execution of the local program; transmitting a data transfer start program to the host system by the subsystem following the synchronization that requires host involvement; and performing data exchange between each subsystem and the host system over the network.
[0029] According to a seventh aspect, there is provided a computer program for causing at least one processor of a subsystem to execute the method according to the sixth aspect.
[0030] According to an eighth aspect, there is provided a non-transitory computer-readable medium storing the computer program according to the seventh aspect.
[0031] For a better understanding of the present invention and to show how it may be implemented, reference is made herein, by way of example, to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] [Figure 1] It is a schematic block diagram of a processor chip including a plurality of tiles. [Figure 2]This is a schematic diagram of a bulk synchronous parallel (BSP) computing model. [Figure 3] Another schematic diagram of the BSP model. [Figure 4] This is a schematic diagram of a multi-stage BSP scheme. [Figure 5] This is a schematic diagram of the exchange of synchronization requests and acknowledgments for external synchronization. [Figure 6A] This is another schematic diagram of a different synchronization group. [Figure 6B] A schematic diagram of the program flow involving a host-synchronous proxy is shown below. [Figure 7] This is a schematic diagram of two accelerators and one gateway that communicate with the host system over a network. [Figure 8] This is a schematic diagram of more accelerators and gateways that communicate with the host system over the network. [Figure 9] This is a further schematic diagram of the system in which data exchange takes place between the accelerator subsystem and the host. [Figure 10] This is a message sequence diagram showing the exchange of messages between the host and the accelerator via the gateway. [Figure 11] The method according to the embodiment of this application is shown. [Modes for carrying out the invention]
[0033] This application relates to a method and system for a host system to compile and transfer a local program set for execution on a processing unit. The processing units are synchronized with each other and exchange data with the host at predefined points in time that are inserted into the compiled code. An example of a processing unit in which an embodiment is implemented is described first.
[0034] Referring to Figure 1, Figure 1 shows an exemplary processing unit 2. This exemplary processing unit 2 is a multi-tile processing unit 2. Processing unit 2 may be an intelligent processing unit (IPU) as described in our earlier U.S. Patent Application No. 16 / 538980, whose contents are incorporated by reference. Each IPU is formed on a separate integrated circuit.
[0035] The processing unit 2 includes an array 6 of multiple processor tiles 4 and an interconnection section 34 connecting the tiles 4. The processing unit 2 can be implemented independently as one of several dies packaged in the same IC package. The interconnection section 34 may also be referred to herein as an "exchange fabric" 34, as it enables the tiles 4 to exchange data. Each tile 4 contains an instance of an execution unit and memory, respectively. For example, the processing unit 2 may contain hundreds or even thousands of tiles 4. For completeness, it should be noted that the term "array" as used herein does not necessarily imply a specific number of dimensions or physical layout of tiles 4.
[0036] In the embodiment, each processing unit 2 also includes one or more external links 8 that enable the processing unit 2 to connect to one or more other processing units (e.g., one or more other instances of the same processing unit 2). These external links 8 may include one or more processing unit-host links for connecting the processing unit 2 to a host system, and / or one or more processing unit links for connecting together with one or more other instances of the processing unit 2 on the same IC package or card, or on different cards. The processing unit 2 receives work from the host in the form of application data to be processed by the processing unit 2.
[0037] The interconnection unit 34 is configured to allow different tiles 4 of the array 6 to communicate with each other. However, just as dependencies potentially exist between threads on the same tile 4, dependencies may also exist between parts of a program running on different tiles 4 of the array 6. Therefore, a technique is needed to prevent dependency data of a code snippet on one tile 4 that is already running from becoming available to another code snippet on another tile 4. This is achieved using a data consistency model.
[0038] Parallel programming models in AI and data science typically follow a three-phase iterative execution model consisting of computation, barriers, and exchanges. This means that data transfer to and from processors typically has barrier dependencies to provide data consistency between processors and between each processor and external storage devices. Typical data consistency models used are bulk synchronous parallelism (BSP), stale synchronous parallelism (SSP), and asynchronous. While processing unit 2 described herein uses the BSP model, it will become clear that other synchronous models can be used as alternatives.
[0039] Referring to Figures 2 and 3, which illustrate an implementation of the BSP exchange scheme, each tile 4 performs a calculation phase 33 and an exchange phase 32 in an alternating cycle, and the calculation phase 33 and the exchange phase 32 are separated from each other by barrier synchronization 30 between tiles. In the example shown in Figures 2 and 3, barrier synchronization is placed between each calculation phase 33 and the subsequent exchange phase 32. During the calculation phase 33, each tile 4 performs one or more calculation tasks locally on the tile, but does not communicate the results of these calculations with any other tile 4. In the exchange phase 32, each tile 4 can exchange one or more results of calculations from the preceding calculation phase with one or more other tile 4, but does not perform new calculations until it receives data on which its task depends from other tile 4. Nor does it transmit data other than that calculated in the preceding calculation phase to other tile 4. It is not excluded that other operations, such as internal control-related operations, may be performed in the exchange phase 32. Communication with the outside world of a tile group can optionally utilize the BSP mechanism, but as an alternative, it can use some other synchronization mechanism of its own without using BSP.
[0040] According to the BSP principle, the barrier synchronization 30 is located at the junction where the transition is made from the calculation phase 33 to the exchange phase 32, at the junction where the transition is made from the exchange phase 32 to the calculation phase 33, or both. That is, (a) all tiles 4 in a group must complete their respective calculation phases 33 before any tile in the group can proceed to the next exchange phase 32, (b) all tiles 4 in a group must complete their respective exchange phases 32 before any tile in the group can proceed to the next calculation phase 33, or (c) both of these conditions are met. In all three variations, it is the individual tiles that alternate phases, and the entire assembly that synchronizes. The sequence of exchange and calculation phases can then be repeated over multiple iterations. In BSP terminology, each iteration of the exchange phase and calculation phase is sometimes referred to as a "superstep" (however, it should be noted that terminology is not always used consistently in the literature. That is, each individual exchange phase and calculation phase may be called a superstep individually, while in other cases, the exchange phase and calculation phase together may be called a superstep, as is the case with the terminology used herein).
[0041] Furthermore, each of several different independent groups of tiles 4 on the same processing unit 2 or different processing units can form separate BSP groups that operate asynchronously with each other, and while the BSP cycles of computation, synchronization, and exchange are imposed only within each given group, this does not preclude each group from doing so independently of other groups. That is, a multi-tile array 6 may include multiple internal synchronization groups, each operating independently and asynchronously from other such groups (to be discussed in more detail later). In some embodiments, there is a hierarchical classification of synchronization and exchange, as will be discussed in more detail later.
[0042] Figure 3 illustrates the BSP principle as implemented between some or all groups 4i, 4ii, and 4iii of tiles in array 6 in a case where barrier synchronization (see above) is imposed from computation phase 33 to exchange phase 32. Note that in this array, some tiles 4 can start computation 33 while some other tiles are still performing exchanges.
[0043] The BSP model is used for data exchange between tiles 4 on processing unit 2. As described in our earlier application, U.S. Patent Application No. 16 / 538980, communication between tiles 4 on processing unit 2 occurs temporally deterministically, and data packets are transmitted without headers. In addition, the BSP model can also be used for data exchange between processing units 2.
[0044] In this embodiment, multiple instances of processing unit 2 can be connected to each other to form a larger array of tiles 4 that extends across multiple processing units 2. These interconnected processing units 2 can participate in barrier synchronization with each other to exchange data. Barrier synchronization occurring only between tiles 4 on a single processing unit 2 is called internal (on-chip) synchronization. On the other hand, barrier synchronization occurring between processing units 2 is called external (inter-chip) synchronization.
[0045] Referring to Figure 4, Figure 4 shows an exemplary BSP program flow involving both internal and external synchronization. As shown, the flow includes internal exchange 50 (of data between tiles 4 on the same chip 2) and external exchange 50' (of data between tiles 4 on different chips 2). The program flow in Figure 4 shows the program flow for the first processing unit 2i and the second processing unit 2ii.
[0046] As shown in Figure 4, the internal BSP superstep (including the internal exchange of data 50 between tiles 4 on the same chip 2) is kept separate from the external synchronization and exchange (including the external exchange of data 50' between tiles 4 on different chips 2).
[0047] The program can be arranged to execute a sequence of synchronization, exchange phases, and calculation phases, including (i) a first calculation phase, then (ii) internal barrier synchronization 30, then (iii) internal exchange phase 50, then (iv) external barrier synchronization 80, and then (v) external exchange phase 50'. The external barrier 80 is imposed after the internal exchange phase 50, and as a result, the program proceeds to external exchange 50' only after the internal exchange 50. It should also be noted that an calculation phase can optionally be included between the internal exchange (iii) and the external barrier (iv), as shown with respect to chip 2I in Figure 4.
[0048] This entire sequence is performed by a program (for example, generated in this way by a compiler). In embodiments, the program is programmed to perform this function by a SYNC instruction executed by tile 4. Internal synchronization and exchange do not extend to any other tile or other entity on another chip 2. Sequence (i) to (v) (with the aforementioned arbitrary arithmetic phases between iii and iv) can be repeated in a series of iterations. In each iteration, there may be multiple instances of the internal arithmetic, synchronization and exchange (i) to (iii) before the external synchronization and exchange. That is, multiple instances of (i) to (iii) (in their order), i.e., multiple internal BSP supersteps, can be performed before (iv) to (v), i.e., the external synchronization and exchange. It should also be noted that each of the tile 4s can execute its own instances of internal synchronization and exchange (ii) to (iii) in parallel with other tile 4s.
[0049] Therefore, for every BSP cycle (i) to (v), at least a portion of cycles (ii) to (iii) may exist, and synchronization is constrained to be performed only internally (i.e., only on the chip).
[0050] It should be noted that during external exchange 50, communication is not limited to external only. That is, some tiles can simply perform internal exchange, some tiles can perform external exchange only, and some tiles can perform a mix of both.
[0051] Furthermore, as shown in Figure 4, some tiles 4 can perform local input / output during the computation phase. For example, those tiles 4 can exchange data with the host or other types of external storage devices.
[0052] Furthermore, as shown in Figure 4, it should be noted that, in general, any or all tiles can have a null calculation phase 52 or a null exchange phase 50 in any predetermined BSP superstep.
[0053] Each of the synchronization barriers shown in Figure 4 is passed by a tile 4 once the exchange of synchronization requests and acknowledgments is complete. In the case of internal barrier synchronization, each tile 4 of the processing unit 2 sends an internal synchronization request to the internal synchronization controller 36 (see Figure 1) of the interconnection unit 34 when it reaches internal barrier synchronization. When the internal synchronization controller 36 receives synchronization requests from all of the tiles 4 that are part of the processing unit 2, the synchronization controller 36 returns a synchronization acknowledgment to each of the tiles 4. Upon receiving the synchronization acknowledgment, each of the tiles 4 enters the exchange phase, and data exchange between the tiles 4 occurs. This technique ensures that each tile 4 must reach the barrier before any of the tiles can proceed to the exchange phase.
[0054] In the case of external barrier synchronization, the exchange of synchronization requests and acknowledgments takes place between groups of processing units 2 called synchronization groups. Following the exchange of synchronization requests and acknowledgments for external barrier synchronization, the processing units 2 exchange data during the exchange phase.
[0055] Each tile 4 on processing unit 2 issues an external synchronization request to the external synchronization logic (not shown in Figure 1) associated with processing unit 2 when it reaches external barrier synchronization. When the external synchronization logic receives synchronization requests from all tiles 4 on processing unit 2, it either acknowledges those synchronization requests or propagates the synchronization requests to the external synchronization logic associated with another processing unit 2. The action taken by the external synchronization logic of another processing unit 2 in response to the synchronization requests depends on whether the logic is defined as the master of the synchronization group or as the propagation node of that group. The propagation node propagates those received synchronization requests to the master defined for the synchronization group. When the synchronization master receives an external synchronization request for each of the processing units 2 that are part of the synchronization group, it returns a synchronization acknowledgment to the external synchronization logic associated with each of the other processing units 2 in the synchronization group. The synchronization master also returns a synchronization acknowledgment to each of the tiles 4 on its own processing unit 2. Each external synchronization logic (i.e., propagation node) of the other processing unit 2 in the synchronization group returns a synchronization acknowledgment to its processing unit 2's tile 4 as soon as it receives one.
[0056] Referring to Figure 5, Figure 5 shows an example of the exchange of synchronization requests performed against external barrier synchronization for data exchange between processing units 2. The shown system 500 includes three processing units 2a, 2b, and 2c (collectively referred to as processing units 2) which are part of a synchronization group. Each of the three processing units 2 belongs to a separate integrated circuit 510a, 510b, and 510c (collectively referred to as chip 510). Each of chip 510 includes an external synchronization logic 520 called a global synchronization peripheral (GSP) 520. In some cases, the external synchronization logic 520 may be located off-chip (e.g., on a gateway device).
[0057] When each of the tiles 4 reaches the barrier, the tile 4 issues a synchronization request to the associated GSP 520. In Figure 5, for simplicity, only one tile 4 from each processing unit 2 is shown as sending a synchronization request to the associated GSP 520; however, in practice, all tiles 4 on the chip will issue synchronization requests to the GSP 520. In some embodiments, each synchronization request includes an indication of the synchronization group to be used for barrier synchronization, and the GSP 520 propagates and / or acknowledges the received synchronization request according to the indication of the request. In other embodiments, prior to the writing of the synchronization request to the GSP 520, a write occurs from the nominated tile 4 of the associated processing unit 2 of the synchronization group to be used for the upcoming barrier synchronization. The indication of the synchronization group is used to select the configuration setting of the GSP 520. When the GSP 520 reaches barrier synchronization, it propagates and / or acknowledges the received synchronization request according to the selected configuration setting.
[0058] In this example, GSPs 520a and 520c are configured as propagation nodes. After receiving synchronization requests from all tile 4s of processing unit 2, each of GSPs 520a and 520c propagates the synchronization requests upstream to the synchronization master (i.e., GSP 520b) defined for the synchronization group. After receiving synchronization requests from GSPs 520a and 520c and from all tile 4s of the associated processing unit 2b, GSP 520b issues synchronization acknowledgments downstream to the tile 4s of processing unit 2b and to GSPs 520a and 520c. GSPs 520a and 520c respond to the received acknowledgments by issuing synchronization acknowledgments to the tile 4s of their respective processing units 2a and 2c.
[0059] In response to receiving a synchronization acknowledgment, tile 4 passes through barrier synchronization and exchanges data with other processing units 2 in the synchronization group during the exchange phase. This exchange of data between different processing units 2 is carried out non-temporal deterministically, as described in our earlier application, U.S. Patent Application No. 15 / 886065.
[0060] Different synchronization groups can be defined for data exchange in each external barrier synchronization. These synchronization groups are programmed into local programs that run on the processing units. Tile 4 executes a synchronization instruction to generate synchronization requests in barrier synchronization. The synchronization instruction takes a synchronization group as an operand and controls which processing units 2 each synchronization occurs between.
[0061] Referring to Figure 6A, Figure 6A shows several different possible external synchronization groups (e.g., Group 1 or Group 2). In embodiments, these correspond to different hierarchical levels. That is, each of the higher hierarchical levels 92 (e.g., Group 2) includes two or more groups 91A, 91B of at least one lower hierarchical level. In embodiments, only two hierarchical levels exist, but it is not excluded that more nested levels exist. To initiate barrier synchronization, the execution unit of tile 4 of processing unit 2 executes a SYNC instruction. The operand of the SYNC instruction indicates a synchronization group. If the operand of the SYNC instruction is set to a lower hierarchical level of the external synchronization group (SYNC Group 1), the synchronization and aggregation operations described above are performed in relation to tile 4 on chip 2 only in the external synchronization group of the same lower level as the tile on which SYNC is performed. On the other hand, if the operand of the SYNC instruction is set to a higher hierarchy level of the external synchronization group (SYNC group_2), the synchronization and aggregation operations described above are automatically performed in relation to all tiles 4 on all chips 2 in the same high-level external synchronization group as the tile on which SYNC was performed.
[0062] In response to the opcode of the SYNC instruction, which has an external synchronization group as an operand, each tile 4 of that synchronization group signals the synchronization level specified by the operand to the associated GSP 520. In response, the GSP 520 performs the synchronization request (sync_req) and acknowledgment (sync_ack) process, which is performed only among the tile 4 of the signaled group.
[0063] It should be noted that in other embodiments, the synchronization groups that can be specified by the mode of the SYNC instruction are not necessarily hierarchical. In general, the SYNC instruction can provide modes that accommodate any type of group. For example, the mode can allow selection from only non-hierarchical groups or from a mixture of hierarchical groups and one or more non-hierarchical groups (where at least one group is not fully nested within another). This advantageously allows the programmer or compiler flexibility in choosing between different layouts of internal synchronization groups that are asynchronous to one another at the lowest code density.
[0064] In addition to external barrier synchronization provided for exchange between processing units 2, external barrier synchronization can be performed to exchange data between processing units 2 and the host system. Such data may include application data provided by the host system for processing by processing units 2. The data may also include the results of processing by processing units 2 provided to the host system. This type of barrier synchronization is called barrier synchronization with host involvement.
[0065] The relationship between barrier synchronization involving the host and other barrier synchronizations is shown in Figure 6B. This figure shows a barrier synchronization in which an exemplary processing unit 2 participates. Processing unit 2 can participate in P barrier synchronizations 80 before a barrier 90 is imposed that requires a synchronization acknowledgment from the host. P barrier synchronizations require synchronization requests from all (non-paused) tile 4 of the associated synchronization group. Subsequent barrier synchronizations 90 require synchronization requests from all (non-paused) tile 4 of processing unit 2 and require that the host has previously given permission to pass through a particular barrier. After barrier 90, an exchange 50" can be performed between the host and one or more tile 4 of processing unit 2 (e.g., one or more tile 4 for reporting calculation results to the host).
[0066] This barrier synchronization and associated data exchange are performed using a network interface device to connect processing unit 2 and the host system. The network interface device is described as a gateway device, but may be another type of network interface device. The gateway device is described in detail in the aforementioned U.S. Patent Application No. 16 / 235109, incorporated by reference.
[0067] Referring to Figure 7, which shows an example of a data processing system 700, the processing unit 2 is configured to communicate with the host system 710 via a gateway 720. In this example, a single gateway 720 is configured to connect two processing units 2 via a network 740. However, in other embodiments, each of such gateways 720 may connect to a different number (e.g., one) of processing units 2 via the network 740.
[0068] The host system 720 includes a buffer 725 associated with each processing unit 2. Each buffer 725 has an associated buffer 730 on the gateway 710. The buffer 730 on the gateway 710 is a mirror of the associated buffer 725 on the host 720. Read and write requests are issued on the network 740 to synchronize the data in the host buffer 725 with the data in the gateway buffer 730. Read and write requests may be RDMA read and write requests. One or more RDMA controllers 735 are present on the host 720 to issue RDMA reads / writes to the gateway 710 to perform data transfers between buffers 725 and buffer 730. Similarly, an RDMA controller 736 is present on the gateway 710 to perform data transfers between buffer 730 and buffer 725. RDMA exchanges occur on the network 740. In this embodiment, the network 740 is an Ethernet® network 740. In this example, data is transmitted over network 740 via RDMA over Converged Ethernet (RoCE).
[0069] The primary examples discussed herein involve transmitting data via RoCE, but other examples may utilize different networking technologies. For example, data can be transmitted over network 740 using a Data Plane Development Kit (DPDK) over standard Ethernet®. However, the use of RDMA offers the advantage of low latency. In embodiments where a different protocol other than RDMA is used, the RDMA controllers 725, 735, 736, and 740 shown in Figures 7 and 9 may be other types of processing circuits used to perform the data transfer operations described.
[0070] The processing units 2 are configured to initiate barrier synchronization to exchange data with their associated memory 730 on the gateway 710 at predefined points in time for their assigned programs. In such predefined barrier synchronization, one or both of the processing units 2 send a synchronization request to the associated external synchronization logic 745 on the gateway 710. The external synchronization logic 745 is shown as a local synchronization propagation module (LSPM) 745. Upon receiving a synchronization request, each LSPM 745 is configured to make an acknowledgment of the synchronization request. Upon receiving such a synchronization acknowledgment, the processing units 2 perform data exchange with their associated memory 730 on the gateway 710. Memory 730 may be memory mapped to the memory of the processing unit 2. In this example, the processing unit 2 issues memory-mapped I / O (MMIO) read / write to send and receive data from memory 730. During the exchange phase, processing unit 2 can send the results of the processing generated in the preceding calculation phase to memory 730, or receive a workload from memory 730 for processing in the subsequent calculation phase. Synchronization requests, acknowledgments, and data exchange take place over the PCIe link between processing unit 2 and gateway 710.
[0071] In Figure 7, only two processing units 2 are shown communicating with the host 720; however, embodiments of the present invention enable a majority of processing units to communicate with the host 720. Referring to Figure 8, Figure 8 shows an example of a system 800 including a number of processing units 2 communicating with a single host 720. The processing units 2 are configured to run a distributed application together, with each running a local program provided by the host 720. The processing units exchange data with the host via a buffer 730 and with other processing units at predefined synchronization points of the local programs. Data exchange between processing units 2 can occur via PCIe links between the processing units 2 or via PCIe links between gateways 710. Data exchange between gateways 710 and the host 720 occurs via connections on the network 740. A separate connection exists for each processing unit 2. Each of these connections may be an RDMA connection.
[0072] Referring to Figure 9, which illustrates in more detail the interaction between the host system 720 and one of the processing units 2, we will now describe the subsequent data exchange resulting from the compilation and provision of the program to the processing unit 2 and the execution of the program.
[0073] As shown, the host 720 includes at least one processor 810 configured to compile a set of programs for execution on the processing units 2. The local programs generated from the source code for the application, combined with the compiled application, form an executable image. Each program takes the form of an executable image. At least one processor 810 receives the source code set and, based on the compiler program, generates a set of local programs from the source code. The compilation process includes generating separate local programs and allocating them across the processing units 2. As part of the compilation process for generating the local programs, at least one processor 810 inserts a set of exchange synchronization points into each local program. At each of these points in the execution of the local programs, each processing unit 2 is configured to participate in barrier synchronization. Some of the barrier synchronizations are internal barrier synchronizations, as described above. Some of the barrier synchronizations are external barrier synchronizations for exchanging data with one or more other processing units 2 involved in executing the application. Some of the barrier synchronizations are external barrier synchronizations for exchanging data with the host 720 via the accompanying gateway 710.
[0074] After host 720 compiles the local program for processing unit 2, the program is stored in control program buffer 920. The host has a separate control program buffer for each processing unit 2. Comments concerning control program buffer 920 apply equally well to these other control program buffers. Buffers 920 and 930 are mirrors of each other, as are buffers 725 and 730 used to exchange data between host 720 and gateway 710. The local program is transferred from buffer 920 to buffer 930 in one or more writes over network 740. This transfer can be performed by one or more RDMA controllers 736 configured to transfer the local program from buffer 920 to buffer 930 in one or more RDMA writes over network 740. Processing unit 2 is configured to issue a read to gateway 710 to read the program from control program buffer 930. Processing unit 2 includes a small amount of secondary boot loader code to perform such reads. The reads are MMIO reads.
[0075] When processing unit 2 loads the local program from buffer 930, processing unit 2 executes the program. To process the workload, the program must first receive that workload from host 720. Host 720 stores the workload required by the program during its execution. Host 720 stores these workloads in buffer 725 before they become available to processing unit 2. The data (including the workload) is divided into different streams, and processing unit 2 can select which streams to read and write depending on the results of different conditions in the execution of its local program.
[0076] Each stream may include, for example, an image sequence (individual still images or video frames), an audio stream of one or more audio samples, or any other arbitrary experience data sequence (e.g., online shopping habits) that is sent from the host subsystem 720 to the processing unit 2 for use as input to a machine learning or machine intelligence algorithm running on multiple processing units 2. For example, this may be for training a neural network based on streaming data, or for inference from data based on an already trained neural network. In the example, if the stream is in the direction from the processing unit 2 to the host subsystem 720, the stream may include a stream of inferences resulting from a machine intelligence algorithm, such as a trained neural network, running on the processing unit 2.
[0077] Buffers 725 and 730 are multi-stream buffers used to store data for transfer from multiple different streams. Buffers 725 and 730 are also bidirectional, allowing both writing and reading of data from each stream. Processing unit 2 can selectively read / write from different streams by supplying stream identifiers to the gateway. After a local program is loaded into processing unit 2, it executes the program to issue a write to processing unit 2 of the identifier of the stream to be read. The stream identifier includes one or more indices or addresses in buffers 725 and 730. Rather than explicitly identifying a stream, the stream identifier can identify one of the memory areas of buffers 725 and 730 shown in Figure 9 (each associated with one of the streams). Gateway 710 receives the stream identifier, which is temporarily stored in stream ID store 940. In response to receiving the stream identifier, the gateway 710 writes the identifier to the host 720 on the network 740. In response to receiving the identifier, the host 720 writes the identified stream data from buffer 725 to buffer 730 on the network 740. The processing unit 2 then reads the data from buffer 730.
[0078] After obtaining the workload from buffer 730, processing unit 2 processes the workload according to its local program. At some point during its execution, processing unit 2 writes a stream representation, which is read or written in the exchange phase following the upcoming barrier synchronization. This write can be performed during processing unit 2's calculation phase or the preceding exchange phase. Subsequently, during program execution, the synchronization barrier is reached. At this point, processing unit 2 performs the synchronization request / acknowledgment procedure described above with respect to Figure 7. Following this procedure, processing unit 2 reads from and writes to buffer 730. In doing so, processing unit 730 reads from and writes to the memory area of the buffer corresponding to the stream identified by the provided stream representation. In response to the data writing to the stream, gateway 710 writes data to buffer 725. Thus, during the exchange phase, host 720 receives the results of processing the workload data from processing unit 2, and processing unit 2 receives further workload data from host 720.
[0079] Referring to Figure 10, which shows a message sequence diagram illustrating the exchange of messages between host 720, gateway 710, and processing unit 2. As shown in Figure 10, in step 1, processing unit 2 writes a stream identifier to the gateway. The gateway receives this stream identifier and writes it to host 720. In step 2, in response to receiving the stream ID, host 720 writes the workload to gateway 710. As shown, barrier synchronization is achieved by processing unit 2. Following barrier synchronization, processing unit 2 reads the workload provided by host 720 in step 2 from gateway 710. Also in step 3, processing unit 2 writes the results obtained from processing the aforementioned workload.
[0080] In step 4, processing unit 2 writes the stream identifier written by processing unit 2 in step 3 to gateway 710. In response to receiving the stream identifier, gateway 710 writes the data written by processing unit 2 in step 3 to the host.
[0081] Following the data exchange, processing unit 2 proceeds to a further calculation phase. The data exchange process shown in Figure 10 can be executed multiple times.
[0082] In Figure 10, the processing unit 2 sends a control message MemWr to initiate data transfer between the gateway 720 and the host 710. In other embodiments, the host 710 can initiate this data transfer by periodically synchronizing buffers 725 and 730.
[0083] Referring to Figure 11, Figure 11 shows an exemplary method 1100 according to an embodiment of the present application. This figure provides an overview of method 1100 performed by a system including a host, a network interface device (i.e., a gateway as described in the above embodiment), and processing units, each configured to run an application by executing a local program.
[0084] In S1110, the host transfers the local program to the processing unit.
[0085] In S1120, each processing unit issues a request for workload data from a specified stream. The stream specified by each processing unit depends on the local program assigned to that processing unit. The request, along with the specified stream, is provided to the host via the network interface device.
[0086] In S1130, in response to each incoming request, the host writes data from the identified stream. The host writes this data to the network interface device for transmission to the processing unit. The processing unit reads the data from the network interface device memory.
[0087] In S1140, the processing unit performs calculations during the calculation phase to execute calculations using the workload data received in S1130. The calculations to be performed are specified in the local program provided to each processing unit.
[0088] In S1150, the processing units exchange the results of their data processing during S1140. This step is performed using barrier synchronization between the synchronization groups of the processing units. The timing and synchronization group definitions for such barrier synchronization are specified in the local program provided to each processing unit.
[0089] Following each of these external barrier synchronizations in S1150, the method returns to S1140, where further calculations are performed during a further computer phase.
[0090] Later in the execution of the local program, the processing unit proceeds to S1160, where the process to initiate data transfer with the host begins. This step may overlap with S1140 or S1150.
[0091] In S1160, the processing unit provides the stream identifier to the host via the network interface device.
[0092] In S1170, the host writes the data of the identified stream to the network interface device.
[0093] In S1180, following a pre-compiled synchronization barrier, the processing unit reads and writes data from / to the identified stream to / from the network interface device. The network interface device writes the data to the host.
[0094] Each processing unit does not need to execute steps S1140 to S1180 simultaneously. While one or more other processing units 2 are exchanging data with the host in S1180, some processing units 2 can perform the calculations in S1140 or the data exchange in S1150. Except when barrier synchronization is performed between groups of processing units 2 in S1150, the processing units operate asynchronously with respect to each other. The order and timing of the steps executed by each processing unit depend on the local programs assigned to the processing units by the host.
Claims
1. A host system configured to be connected via a network to a plurality of subsystems each of which operates as a work accelerator for the host system according to a local program executed in each of the plurality of subsystems, the host system includes at least one processor and at least one memory storing computer readable instructions; At least one processor of the host system is configured to execute the computer readable instructions to compile a set of local programs, each local program to be executed in one of a plurality of subsystems, and compiling source code includes defining a set of synchronous groups, each synchronous group including at least some of the subsystems; Each local program a set of operational instructions for performing operational operations during one or more of a plurality of operational phases of the subsystem, when executed by at least one processor of the subsystem; a set of data transfer instructions, when executed by at least one processor of said subsystem, for performing a plurality of barrier synchronizations at predefined points in the execution of said local program, at least some of said barrier synchronizations being defined between one of said synchronization groups, at least one of said barrier synchronizations being a synchronization requiring host participation, and in response to a data transfer initiator being provided by said subsystem to a host system following a synchronization requiring host participation, data exchange is performed between said subsystem and said host system; at least one processor of a host system is configured to, in response to each received data transfer initiator, exchange data over the network with one of the subsystems from which the initiator originates; Host system.
2. 2. The host system of claim 1, wherein the exchange of data over the network with one of the subsystems from which the initiator originates is performed using remote direct memory access.
3. 3. The host system of claim 1, wherein data transmission between the subsystem and the host system is performed using RDMA.
4. The step of exchanging data over the network with one of the subsystems that is a source of the initiator, comprising: A host system as claimed in any one of claims 1 to 3, comprising synchronising at least a portion of a buffer of the host system with at least a portion of a buffer of a network interface device accessible to one of the subsystems from which the initiator is provided.
5. Each of the received data transfer initiators includes an identifier of a stream between the host system and the one of the subsystems that provided the identifier; A host system as claimed in any one of claims 1 to 4, wherein the step of exchanging data over the network with one of the subsystems from which the identifier is provided comprises sending and / or receiving data belonging to each identified stream.
6. at least a portion of the host system buffer includes a portion of the buffer associated with the identified stream; each of the received data transfer initiators includes an identifier of a stream between the host system and the one of the subsystems from which the identifier was provided; 5. The host system of claim 4, wherein at least a portion of a buffer of a network interface device accessible to one of the subsystems from which the identifier is provided includes a portion of a buffer associated with the identified stream.
7. said step of exchanging data over said network with one of said subsystems which is a source of said initiator, said step comprising: transmitting data to a network interface device accessible by the subsystem providing the initiator prior to one of a plurality of barrier synchronizations defined in the local program of the subsystem; receiving data from the network interface device accessible to the subsystem from which the initiator originates, following the one of a plurality of barrier synchronizations defined in the local program of the subsystem from which the initiator originates; The host system according to any one of claims 1 to 6, comprising:
8. A host system according to any preceding claim, wherein at least one processor of the host system is configured to cause dispatch of each of the local programs to a respective subsystem over the network.
9. A subsystem configured to be connected to a host system via a network, configured to operate as a work accelerator for the host system according to a local program executing on at least one processor of the subsystem; an interface configured to receive the local program from the host system over the network; The local program is a representation of a set of synchronization groups, at least some of the synchronization groups including one or more further subsystems for acting as accelerators for the subsystem and a host; a set of operational instructions for performing a plurality of operational operations during one or more of a plurality of operational phases of the subsystem when executed by at least one processor; a set of data transfer instructions for, when executed by at least one processor, performing a plurality of barrier synchronizations at predefined points in the execution of a respective local program; at least some of the barrier synchronizations are defined between one subsystem of the synchronization group, at least one of the barrier synchronizations is a synchronization requiring host participation, and following the synchronization requiring host participation, a data transfer initiator is sent by the subsystem to the host system, followed by data exchange over the network between the subsystem from which the data transfer initiator was sent and the host system. Subsystem.
10. 10. The subsystem of claim 9, wherein the data exchange between the subsystem and the host system is performed using remote direct memory access.
11. A subsystem according to claim 9 or 10, wherein data exchange between the subsystem and the host system is performed using RDMA.
12. The data exchange between the subsystem and the host system is at least one processor causing data exchange with a buffer of a network interface device for the subsystem; A subsystem according to any one of claims 9 to 11, wherein at least a portion of the buffer is synchronized with at least a portion of the buffer of the host system.
13. the initiator of a data transfer includes an identifier of a stream between the subsystem and the host system; A subsystem according to any one of claims 9 to 12, wherein data exchange between the subsystem and the host system comprises exchanging data belonging to each identified stream.
14. 14. The subsystem of claim 13 when dependent on claim 12, wherein at least a portion of the buffer of the network interface device includes a portion of the buffer associated with the identified stream.
15. data exchange between said subsystem and said host system; transmitting data to a network interface device accessible by the subsystem from which the initiator originates, prior to one of a plurality of barrier synchronizations defined in the local program of the subsystem; receiving data from the network interface device accessible to the subsystem from which the initiator originates following one of a plurality of barrier synchronizations defined in the local program of the subsystem; The subsystem of any one of claims 9 to 14, comprising:
16. The subsystem of any one of claims 9 to 15, wherein the interface is configured to receive the local program from the host system over the network.
17. 1. A method implemented in a host system, comprising: connecting via a network to a plurality of subsystems that operate as work accelerators for the host system according to a local program executed in each of the plurality of subsystems; compiling a set of local programs to be executed in one of the plurality of subsystems; The step of compiling the source code includes defining a set of synchronization groups including at least some of the subsystems; Each local program a set of operational instructions for performing operational operations during one or more of a plurality of operational phases of the subsystem, when executed by at least one processor of the subsystem; a set of data transfer instructions, when executed by at least one processor of said subsystems, for performing a plurality of barrier synchronizations at predefined points in the execution of said local program, at least some of said barrier synchronizations being defined between one subsystem of said synchronization group, at least one of said barrier synchronizations being a synchronization requiring host participation, and following a synchronization requiring host participation, data exchange is performed between said subsystem and said host system in response to an initiator of a data transfer being provided by said subsystem to said host system; in response to each received data transfer initiator, exchanging data over the network with one of the subsystems from which the initiator originates; method.
18. 20. A computer program product causing at least one processor of a host system to carry out the method of claim 17.
19. 1. A method implemented in a subsystem for a host system, comprising: connecting to a host system via a network; receiving a local program from the host system over the network; operating as a work accelerator for the host system according to a local program executing on at least one processor of the subsystem and including an indication of a set of synchronization groups including the subsystem and one or more further subsystems for operating as accelerators for a host; executing operational instructions of the local program to perform operational operations during one or more of a plurality of operational phases of the subsystem; executing data transfer instructions of the local program at predefined times in the execution of the local program to perform a plurality of barrier synchronizations, the synchronizations being defined between one subsystem of the synchronization group and requiring host involvement; sending a data transfer initiator by said subsystem to said host system following a synchronization requiring host involvement, followed by performing a data exchange over said network between the subsystem from which the data transfer initiator was sent and said host system; A method comprising:
20. 20. A computer program product causing at least one processor of a subsystem to carry out the method of claim 19.
