Communication in computers with multiple processors

Abstract

A computer includes a plurality of processors, each processor configured to perform operations on data during a compute phase for the computer and, following a pre-compiled synchronization barrier, exchange data with at least one other processor during an exchange phase for the computer, wherein the processors in the computer are indexed and the data exchange operation performed by each processor in the exchange phase depends on its index value.

Description

Technical Field

[0001] The present disclosure relates to controlling data transfer operations performed between different processors in a computer including multiple processors. Background Art

[0002] When performing massively parallel operations, multiple processors are provided within a computer. Such a computer may be a system in which each processor is provided on a single chip. Each processor itself may include multiple processing units. In some cases, each processor may be configured to perform the same set of operations on different data sets. Each processor performs its corresponding calculation during a calculation phase and then exchanges the results with one or more of the other processors during an exchange phase. A precompiled barrier is applied between the calculation phase and the exchange phase. The exchange of results allows synchronization between the different processors, each performing a different task within the context of the larger operation.

[0003] An example of the application of computers comprising multiple processors is found in the context of machine learning algorithms, such as deep neural networks. As is familiar to those skilled in the art of machine intelligence, machine intelligence algorithms are based on iteratively updating a "knowledge model," which can be represented by a graph of multiple interconnected nodes. Each node represents a function of its inputs. Some nodes receive inputs to the graph, while some receive inputs from one or more other nodes, while the outputs of some nodes form the inputs of other nodes, and the outputs of some nodes provide the outputs of the graph (and in some cases, a given node may even have all of these: graph inputs, graph outputs, and connections to other nodes). Furthermore, the function at each node is parameterized by one or more corresponding parameters (e.g., weights). During the learning phase, the goal is to find values for various parameters based on a set of empirical input data so that the graph as a whole will generate the desired output for a range of possible inputs. Various algorithms for doing this are known in the art, such as backpropagation algorithms based on stochastic gradient descent. Over multiple iterations based on the input data, the parameters are gradually tuned to reduce their errors, so that the graph converges towards a solution. In subsequent stages, the learned model can be used to predict the output given a specified set of inputs or to infer the input (cause) given a specified set of outputs.

[0004] Typically, at least some of the processing for each node can be performed independently of some or all other nodes in the graph, so large graphs provide large opportunities for concurrency and / or parallelism. Each processing unit is a computer that can be used to perform processing associated with a different node. In this way, multiple processing units can be used together to train a single neural network. This is achieved by applying data parallelism, where each processing unit uses a different data set to derive updates to the weights for each processing unit of the neural network. These updates are then synchronized between the processing units during the exchange phase.

[0005] Therefore, it can be understood that when exchanging data between processors in a computer, each processor is required to exchange different pieces of data at different times. Furthermore, each processor may be configured to perform different operations on the received data, such as storage and reduction of data, depending on the location of the corresponding processor in the computer. Therefore, it is necessary to ensure that each processor in the computer performs its corresponding operation at the appropriate time. Summary of the Invention

[0006] As mentioned above, in order to ensure that data is exchanged between the processors of a computer, each processor must perform the appropriate operation required for data exchange at the appropriate time. A way to achieve this purpose is to compile different executable code sets (executable images) for each processor in the computer. By compiling different code sets for each processor, each processor can be programmed to perform the appropriate operation for data exchange at different times, thereby allowing the appropriate data exchange between the processors. However, there is a problem associated with compiling a separate program for each processor, that is, the compile time significantly increases along with the number of processors, making the compile time become undesirably large.

[0007] The present invention provides a technique for exchanging data between processors that is particularly suitable for use when performing training using machine learning models, but is not limited to this context.

[0008] According to an embodiment of the present application, each processor in a processor set in a computer is provided with a single set of executable computer code, which may be in the form of an executable image. Each processor in the processor set is provided with an index value that determines which operations expressed in the executable computer code set are performed at a particular time during execution. The operations that depend on the index value are operations that control the exchange of data between the processors. Thus, the processors are configured to perform the appropriate exchange operations based on their corresponding index values, without requiring a separate set of executable code to be compiled for each processor.

[0009] According to a first aspect, a computer is provided, comprising a plurality of processors, each processor being configured to perform operations on data during a computation phase of the computer and, after a pre-compiled synchronization barrier, to exchange data with at least one other processor during an exchange phase of the computer, each processor comprising: at least one storage device comprising a program containing an executable instruction set and an index value identifying a corresponding processor among a plurality of processors; at least one data memory storing data for computation specified by the executable instructions; and at least one execution unit arranged to execute one or more instructions in the corresponding executable instruction set to: select, during an exchange phase of the computer, an operation for performing data transfer between the corresponding at least one data memory and the corresponding at least one other processor according to the corresponding index value; and execute the selected operation for performing the data transfer during the exchange phase.

[0010] In some embodiments, a computer is provided wherein, for each of at least some of the processors, selecting an operation for performing a data transfer according to a corresponding index value includes selecting a corresponding at least one other processor to which to transfer the data.

[0011] In some embodiments, for each of at least some of the processors, the operation causes data to be transferred between a region of the corresponding at least one data memory selected according to the corresponding index value and the corresponding at least one other processor.

[0012] In some embodiments, for each processor of at least some of the processors, the corresponding at least one execution unit is configured to calculate an address of the corresponding region of the data memory by performing an arithmetic operation on the corresponding index value.

[0013] In some embodiments, for each of at least some of the processors, the corresponding at least one execution unit is configured to branch to at least one instruction in the corresponding executable instruction set based on the corresponding index value, wherein each of the at least one instruction, when executed by the corresponding at least one execution unit, causes data to be transferred between the corresponding region of the data memory and the corresponding at least one other processor.

[0014] In some embodiments, for each of at least some of the processors, the operations include storing data received from the corresponding at least one other processor in a corresponding region of the data memory.

[0015] In some embodiments, for each of at least some of the processors, the operations include selecting data stored in the corresponding at least one data memory according to the corresponding index value and transmitting the selected data to the corresponding at least one other processor.

[0016] In some embodiments, at least one execution unit of each of at least some of the processors is configured to process the respective selected data to generate a data packet comprising the respective selected data and a header comprising an address of a buffer on the respective at least one other processor.

[0017] In some embodiments, the computer includes routing hardware configured to statically route data being transferred in a data transfer.

[0018] In some embodiments, for each processor of at least some of the processors: the operation includes combining data received from the corresponding at least one other processor with additional data stored in the corresponding at least one data memory; and the corresponding at least one execution unit is configured to select the additional data from the corresponding at least one data memory according to the corresponding index value.

[0019] In some embodiments, for each processor: the corresponding at least one data store includes a plurality of incremental weights for at least one machine learning model; and data transfer between the corresponding at least one data store and the corresponding at least one other processor includes controlling the transfer of one or more incremental weights between the corresponding at least one data store and the corresponding at least one other processor.

[0020] In some embodiments, for each of at least some of the processors, the corresponding at least one execution unit is configured to select one or more delta weights from the corresponding data memory according to the corresponding index value, and cause the selected one or more delta weights to be transmitted to the corresponding at least one other processor.

[0021] In some embodiments, for each processor of at least some of the processors, the corresponding at least one execution unit is configured to: select one or more delta weights from the corresponding data memory based on the corresponding index value; and reduce the selected one or more delta weights with one or more additional delta weights received from the corresponding at least one other processor of the plurality of processors.

[0022] In some embodiments, each processor in the computer is a system on a chip.

[0023] According to a second aspect, there is provided a computer-implemented method for generating a plurality of programs, each program being suitable for execution by an execution unit of a computer comprising a plurality of processors, each processor having at least one memory for holding a corresponding program, at least one execution unit for executing the corresponding program, and a data memory for holding data, the method comprising: compiling a single executable instruction set; determining for each processor an index value associated with the corresponding processor; and generating for each processor a local program comprising the single instruction set and the corresponding index value, wherein each local program, when assigned to its corresponding processor, is scheduled for execution on at least one execution unit of the corresponding processor so as to: perform operations on data during a computation phase for the computer, thereby selecting operations for performing data transfer between the corresponding at least one data memory and the corresponding at least one other processor during an exchange phase for the computer according to the corresponding index value; and performing the selected operations for performing the data transfer during the exchange phase after a pre-compiled synchronization barrier.

[0024] In some embodiments, for each local program, selecting an operation for performing the data transfer according to the corresponding index value includes selecting a corresponding at least one other processor to which to transfer the data.

[0025] In some embodiments, for each local program, the operation causes data to be transferred between the region of the corresponding at least one data memory selected according to the corresponding index value and the corresponding at least one other processor.

[0026] In some embodiments, for each native program, the corresponding at least one execution unit is configured to calculate an address of the corresponding region of the data memory by performing an arithmetic operation on the corresponding index value.

[0027] In some embodiments, for each native program, the corresponding at least one execution unit is configured to branch to at least one instruction in the corresponding executable instruction set based on the corresponding index value, wherein each of the at least one instruction, when executed by the corresponding at least one execution unit, causes data to be transferred between the corresponding region of the data memory and the corresponding at least one other processor.

[0028] In some embodiments, each local program, the operation includes storing data received from the corresponding at least one other processor in a corresponding area of the data memory.

[0029] In some embodiments, each local program operation includes selecting data stored in the corresponding at least one data memory according to the corresponding index value and transmitting the selected data to the corresponding at least one other processor.

[0030] In some embodiments, each local program, when assigned to its corresponding processor, is scheduled to execute on at least one execution unit of the corresponding processor so as to: process the corresponding selected data to generate a data packet including the corresponding selected data and a header including an address of a buffer on the corresponding at least one other processor.

[0031] In some embodiments, the computer includes routing hardware configured to statically route data being transferred in a data transfer.

[0032] In some embodiments, for each local program, the operation includes combining data received from the corresponding at least one other processor with additional data stored in the corresponding at least one data memory; and the corresponding at least one execution unit is configured to select the additional data from the corresponding at least one data memory according to the corresponding index value.

[0033] In some embodiments, for each local program, the corresponding at least one data storage includes a plurality of incremental weights for at least one machine learning model; and data transfer between the corresponding at least one data storage and the corresponding at least one other processor includes controlling the transfer of one or more incremental weights between the corresponding at least one data storage and the corresponding at least one other processor.

[0034] In some embodiments, for each local program, the corresponding at least one execution unit is configured to select one or more delta weights from the corresponding data memory according to the corresponding index value, and cause the selected one or more delta weights to be transmitted to the corresponding at least one other processor.

[0035] In some embodiments, for each local program, the corresponding at least one execution unit is configured to: select one or more delta weights from the corresponding data memory according to the corresponding index value; and reduce the selected one or more delta weights with one or more additional delta weights received from the corresponding at least one other processor of the plurality of processors.

[0036] In some embodiments, each processor in the computer is a system on a chip. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] For a better understanding of the invention and to show how it may be practiced, reference will now be made to the accompanying drawings, by way of example.

[0038] Figure 1 is a diagram illustrating distributed training in a neural network,

[0039] Figure 1Ais a schematic diagram showing a row of processors for implementing a simple "stream" row full reduce algorithm,

[0040] Figure 2 is a schematic diagram of the full reduction operation,

[0041] Figure 3A This is a diagram of the stages of a reduce-scatter operation.

[0042] Figure 3B is a schematic diagram of the timing of operations in the full reduction operation,

[0043] Figure 4 It is a schematic diagram of exchanging data between processors based on the operation selected by the index value.

[0044] Figure 4A is a diagram of how data is exchanged between processors according to operations selected depending on index values.

[0045] Figure 4B This is a schematic diagram of a multi-tile processor.

[0046] Figure 5 is a diagram illustrating the computation and switching stages within a multi-chip processor.

[0047] Figure 6 The diagram shows the data exchange in a massive synchronous parallel system.

[0048] Figure 7 The diagram shows the arrangement of multiple processors in a computer.

[0049] Figure 8 Diagram showing the exchange of data packets between slices in a computer.

[0050] Figure 9 is a simplified diagram of a slice of a multi-chip processor,

[0051] Figure 10 illustrates the arrangement of slices in a multiprocessor computer.

[0052] Figure 11 A diagram illustrates how to compile a program for each processor in a computer, and

[0053] Figure 12 is a diagram of exchanging data between processors according to an operation selected depending on an index value. DETAILED DESCRIPTION

[0054] Aspects of the present invention were developed in the context of a computer including a multi-chip processor designed to act as an accelerator for machine learning workloads. However, the present invention is not limited to the machine learning context. The accelerator includes multiple interconnected processors. In some embodiments, each processor can take the form of a multi-chip processor. Multi-chip processors that can be used to implement embodiments of the present invention are described in U.S. Patent Application No. 15 / 886,315, which is incorporated herein by reference. Alternatively, each processor can simply take the form of a single monolithic processor.

[0055] Example embodiments of the present invention will now be described in more detail with reference to the accompanying drawings.

[0056] refer to Figure 12 , which illustrates an example of a computer 700 according to an embodiment of the present application. Computer 700 includes multiple processors 2i, 2ii, 2iii, 2iv, 2v, 2vi (collectively referred to as processors 2). Each processor 2 is implemented in an integrated circuit, also known as a chip. Therefore, each processor 2 is a system on a chip.

[0057] Each of the plurality of processors 2 includes at least one instruction memory 410 storing instructions executable by at least one execution unit 420 of the corresponding processor 2. Each processor 2 also includes at least one storage device 415 storing index values. Although each of the storage units 405, 410, 415 is Figure 12 405, 410, 415 are shown as separate storage elements, but in some embodiments, two or more of these may form part of a single memory unit. Although each of the memory units 405, 410, 415 is shown as a contiguous memory bank, each of one or more of the memory units 405, 410, 415 may include multiple memory units distributed throughout the corresponding processor 2. Similarly, each of the at least one execution unit 420 may be multiple execution units. The index memory 415 may be a register.

[0058] The set of instructions in instruction memory 410 and the indexes in index storage 415 together constitute a program for the corresponding processor 2. Each processor 2 includes at least one execution unit configured to execute instructions stored in instruction memory 410 that modify data values stored in data memory 405 and perform operations to transfer data to and from other processors 2. The instructions in instruction memory 410 are identical in each processor 2. However, the index values stored in index storage 415 differ between processors 2. Each processor 2 in computer 700 stores a different index value in its corresponding index storage 415. The index value stored by each processor 2 in storage 415 uniquely identifies that processor 2 in computer 700. Each processor 2 is configured to execute a different set of data transfer operations based on its stored index value, thereby appropriately transferring data between processors 2 in computer 700. The index value controls which data from memory 405 is transferred and controls the operations performed on the received data.

[0059] Routing Hardware ( Figure 12 (not shown) is located in the computer 700 and is used to route data between the processors 2. The routing hardware takes the form of an external interconnect 72, as described with respect to FIG. Figure 10 As described in more detail, data is sent between processors 2 in the form of data packets. The routing hardware is configured to statically route the data packets. In static routing, the routing hardware includes a routing table that includes fixed routing information for routing the data packets.

[0060] Computer 700 alternates between a computation phase and an exchange phase. During the computation phase, each processor 2 in the computer performs computations until a pre-compiled synchronization barrier is reached. After the barrier, computer 700 moves into an exchange phase, where processors 2 exchange data with each other.

[0061] Some of the instructions in the instruction memory 410, when executed by the execution unit 420, cause the execution unit 420 to perform calculations using the data held in the data memory 405. Some of the instructions in the instruction memory 410, when executed by the execution unit 420, cause the data held in the data memory 405 to be transferred to another of the processors 2. Some of the instructions in the instruction memory 410, when executed by the execution unit 420, cause data received from another of the processors 2 to be stored in the processor's data memory 405.

[0062] Thus, the instructions are executed by execution unit 420 to control data transfers between processor 2, which includes execution unit 420, and another processor of computer 700. The index stored in index storage 415 is used by execution unit 420 to control data transfers. This can be implemented in various ways. In some embodiments, instructions from instruction memory 410 take an index value as input and, when executed, cause execution unit 420 to determine how to perform a data transfer based on a calculation performed on the index value. In some embodiments, the index value is used to select a specific instruction from instruction memory 410 to perform a specific data transfer operation.

[0063] The index value can be used by execution unit 420 to select a particular processor from the plurality of processors 2 to which data is to be transferred from data memory 405. In one example, processor 2i selects processor 2ii from the plurality of processors based on the index value stored in index storage 415. Execution unit 420 then causes the data to be transferred to the selected processor 2ii. To do so, execution unit 420 processes the data to generate a data packet containing the address of processor 2ii, and then dispatches the packet, provided by the routing hardware of computer 2, to processor 2ii.

[0064] The index value can be used by execution unit 420 to select an address on the destination processor where the data is to be stored. For example, execution unit 420 of processor 2i determines an address in data memory 205 of processor 2ii based on the index value of processor 2i. Execution unit 420 then processes the data to be transmitted to processor 2ii to generate one or more data packets, the data packets including one or more headers containing the address in memory 405 of processor 2ii where the data is to be stored. Execution unit 420 causes these data packets to be sent to processor 2ii, where they are stored in memory 405 at the address indicated in the header of the data packet.

[0065] The index value can be used by execution unit 420 to select specific data to be transferred from data memory 405. Execution unit 420 determines the address of the data to be transferred in data memory 405. Execution unit 420 then causes the selected data to be transferred to one of the processors, such as processor 2 ii. The execution unit can determine the address of the data to be transferred in data memory 405 by performing a calculation using the address of the index value as input or by selecting an instruction from instruction memory 405 for causing the specific data to be transferred.

[0066] The index value can be used by the execution unit 420 to control the handling of the received data at the processor 2. For example, when the processor 2ii receives data from the processor 2i, the execution unit 420 of the processor 2ii can use the corresponding index value to control where the data is stored in the data memory 405. Additionally or alternatively, when the processor 2ii receives data from the processor 2i, the execution unit 420 of the processor 2ii can use the corresponding index value to select the data from the data memory 405 and perform an operation (e.g., combine the data) involving the received data and the data selected from the memory 405. The processor 2ii can then store the result of the operation in the memory 405.

[0067] Although Figure 12 Only data transfer between processors 2i and 2ii is shown, but it should be recognized that data transfer can be performed between other processors 2 in computer 700, and for each processor 2, the index value of the corresponding processor 2 can be used to select a specific data transfer operation to be performed in any manner described above for processors 2i and 2ii.

[0068] When training a machine learning model, embodiments of the present invention can be applied to data exchange between processors of a computer. To explain this example application of the disclosed technology, refer to Figure 1 , which shows a schematic block diagram of a distributed architecture for training a neural network. A source of training data 100 is provided. This can be a database or any other type of data repository capable of holding training data suitable for the neural network model being trained. The processing of the neural network model itself is distributed across a plurality of processors 110a, 110b, 110c, etc. Figure 1 Only three units are shown, but it will be readily appreciated that any number of processors may be used. Each processor 110a, b, c receives batches of training data from a training data source 100. Each processor 110a, b, c maintains a set of parameters 112a, 112b, 112c that define the model. The incoming batches of training data are processed in a computation function 114 with the current set of parameters, and the result of the computation function 114 is used to generate so-called deltas, which represent the difference between the original parameters and the new parameters as a result of applying the computation function to the batch of training data and the current set of parameters. In many neural networks, these parameters are called "weights," and so the delta values are called "delta weights." In Figure 1 In FIG, weights are labeled 112a, b, c, and delta weights are labeled 116a, b, c. It will be appreciated that in practice, the weights and delta weights are stored in a suitable memory bank accessible to the processor. If the weights and delta weights can be kept in local memory, this makes the training process more efficient.

[0069] Figure 1 The goal of the architecture is not to train three separate models, but to train a single model in a distributed fashion. Therefore, the goal is to have the model parameters (or weights) converge to a single common set in each processor. Starting from any particular set of weights, and assuming that the batches of training data received at each processor are not exactly the same, the weight increments produced by each compute function in each processor will vary. Therefore, there needs to be a way to distribute the incremental weights across the processors after each iteration of the batch of training data. This is done in Figure 1 Schematically shown in FIG, wherein a combination function 118 receives incremental weights from each processor and performs a mathematical function, such as an average function, that reduces the incremental weights. The output of the combination function 118 is then fed back to the combination circuits 120a, 120b, and 120c within each processor, respectively. A new set of weights is thus generated as a combination of the original weights and the combined outputs from the combination function 118, and the new weights 118a, 118b, 118c are stored back to the local memory. The next batch of training data is then supplied to each processor and the process is repeated multiple times. If the starting weights of the processors are the same, then after each iteration they will be reset to the same new value again. The incremental weights are supplied to the combination function 118, where they are reduced, and then they are supplied back to each processor in their reduced form, where they can be combined with the original weights.

[0070] One way to efficiently exchange data between processors is through the use of collectives, which are routines commonly used when processing data in a computer. They are routines that enable data to be shared and processed across multiple different processes, which can be running on the same processor or different processors. For example, if a process reads data from a data repository, it can use a "broadcast" process to share that data with other processes. Another example is when the results of a particular function are needed on multiple processes. "Reduction" is the result of applying a calculation function to the data values from each of the multiple processes. "Gather" and "scatter" collectives handle more than one data item. Certain collectives are becoming increasingly important in processing machine learning applications.

[0071] MPI (Message Passing Interface) is a message passing standard that can be applied to a variety of parallel computing architectures. MPI defines a number of collectives suitable for machine learning. One such collective is called a "full reduce". The full reduce operation enables the result of a computational function acting on multiple data values from different source processes to be provided at a receiving process. Note that the receiving process can be one of the source processes, and there can be multiple receiving processes. The full reduce collective reduces the data values from multiple source processes and distributes the results to all source processes (which act as receiving processes for the results of the reduction). According to the MPI standard, a full reduce collective is implemented by reducing the data values from all source processes in the reduction collective (for example, at one of the processes) and then broadcasting the result to each source process.

[0072] Figure 1A is a diagram illustrating how to implement a fully reduced aggregate in a topology of five processors N0...N5 connected by wires. These processors can be connected to Figure 1 The processors are connected in a line configuration, where each processor is connected via a "forward" link L F and the "backward" link L B connected to its neighbors. As shown in the figure, and as the directional phrases imply, the forward link is Figure 1A The processors are connected from left to right in the Figure 1A The processors are connected from right to left in FIG. Each processor has a processing capability designated as 200 and a storage capability designated as 202. Each processor also has a memory capability that enables it to communicate with the processor via a link L. F / L B One or more link interfaces connected to its neighboring processors.

[0073] Figure 2 A full-reduce aggregate is illustrated in , which can be used to exchange delta weights between processors. Figure 2A set (vector) of partial values or "parts" P0, P1, P2, P3 on each of four processors in a starting state S1 is shown. In this context, a processor is a processor in a network of processors. Note that each processor N0, N1, N2, N3 has four "corresponding" parts P0, P1, P2, and P3. That is, each part has a position in the vector such that P0(n) has the same position in its vector on processor n as P0(n+1) has in its vector on processor n+1. The suffix (n) is used to indicate the processor on which the part is located - so P0(0) is part P0 on processor N0. In a reduce-scatter pass, the corresponding parts are reduced and the reduction is provided to one of the processors. For example, parts P0(0), P0(1), P0(2), P0(3) are reduced (to r0) and placed on processor N0. Similarly, parts P1(0), P1(1), P1(2), and P1(3) are reduced (to r1) and placed on processor N1. And so on, in the intermediate state S2, each processor has one of the reductions r0, r1, r2, and r3. As explained, the reductions can be obtained by any combination of functions - It can include independent operators (e.g., max) or associative operators = P1(N0)*P1(N1)*P1(N2)*P1(N3). Then, in an all-gether pass, each reduction is provided to all processors to activate state S3, where each processor now holds all four reductions. Note that in S1, the "corresponding" parts, such as P0(0), P0(1), P0(2), and P0(3), can all be different, while in state S3, each reduction (e.g., r0) is the same at all processors, where r i =f{(P i (0),P i (1),P i (2) and P i (3))}. In machine learning, a set of partial values P0, P1, P2, P3 is a vector. Each pass through the model during training produces a vector of parts (e.g., updated weights). The reduction r0, r1, r2, r3 on each processor in state S3 is a fully reduced vector. In the context of machine learning, each part can be a set of update increments for parameters in the model. Alternatively (in an arrangement not further described herein), it can be an updated parameter.

[0074] Therefore, as described previously, a full reduce operation consists of a reduce-scatter operation followed by a full gather operation. During the reduce-scatter operation, each node exchanges a subset of different elements. When the reduce-scatter operation is complete, all nodes have 1 / n of the final full reduce. During the full gather, each node receives an additional 1 / n of the final full reduce, until after n-1 steps, all nodes have the complete data set.

[0075] Figure 3A and 3B An example of a reduce-scatter / gather using six "virtual" ID rings is illustrated. These are also referred to herein as "logical" rings. Figure 3A is a diagram illustrating the reduction of parts in multiple virtual rings. Each part is split into six segments. Figure 3A In the example, the capital letters R, Y, G, B, P, and L each represent a different fragment of the portion stored at each processor. The letters represent corresponding fragments to be reduced to each other and define a "virtual" or "logical" ring for these fragments. Figure 3A , in each part P0, P1, P2, P3, P4 and P5, “R” fragments are in the result vector The Y, G, B, P, and L segments are similar.

[0076] Figure 3B A timing diagram is shown, where the time on the horizontal axis indicates the data exchange and computation in each step of the full reduction process. Figure 3A In A and B, the full reduction process is completed by a reduce-scatter phase followed by a full-gather phase.

[0077] Figure 3A and 3B The symbols in are as follows. The partials are each denoted P0, P1, P2, P3, P4, and P5. At the start of the process, each part is stored on the corresponding processor N0, N1, N2, N3, N4, N5. Each fragment is labeled according to its fragment number and its position in the virtual ring in which it is considered to be reduced. For example, RA0 represents the R fragment in part P0, because this is the first fragment in the virtual ring formed by processors N0-N1-N2-N3-N4-N5. RA1 represents the R fragment calculated at processor N1, which is in the second position in its virtual ring. YA0 represents the Y fragment calculated at processor N1. The "0" suffix indicates that it is the first fragment in its virtual ring, which is N1-N2-N3-N4-N5-N0. It is important to note that the suffix on A reflects the virtual ring, but does not correspond to a physical processor (or part). Note that, Figure 3A Only the virtual ring on the forward link is shown. Figure 3B An equivalent process is shown occurring on the reverse link, where segment B is indicated.

[0078] In step 1, the first fragment (A0) in each virtual ring is transferred from its processor to the next adjacent processor, where it is reduced with the corresponding fragment at that processor. That is, RA0 moves from N0 to N1, where it is reduced with RA1 to form RA 0,1 The notation 0,1 indicates that the segment is formed by reducing the first and second segments in the virtual ring. Note that in the same step, the A0 segment of each virtual ring is transmitted simultaneously. That is, the link between N1 and N2 is used to transmit YA0, the link between N2 and N3 is used to transmit GA0, and so on. In the next step, the corresponding reduced segments are transmitted to their next neighbor processors via the forward link. For example, RA 0,1 Transfer from N1 to N2, YA 0,1 Transmitted from N2 to N3. Note that for clarity, Figure 3A The complete set of fragments and their numbers are in Figure 3B This process is performed in five steps. After all five steps, all segments on each processor are reduced. At the end of the fifth step, the reduction occurs on the last processor in each corresponding ring for that segment. For example, R reduction occurs on processor N5.

[0079] The full gather phase begins with a transfer from the last processor in each virtual ring to the first processor. Thus, the final reduction for the R segment ends on processor N5, preparing for the first step of the full gather phase. The final reduction for the Y segment correspondingly ends on processor N0. In the next step of the full gather phase, the reduced segments are again transferred to their next neighboring processor. Thus, the fully reduced R segment is now also at N2, the fully reduced Y segment is now also at N3, and so on. In this way, each processor ends up at the end of the full gather phase with a partial copy of all fully reduced segments R, Y, G, B, P, L.

[0080] Example embodiments of the present invention can be applied to control data exchange in the context of machine learning. In particular, example embodiments can be applied to control the above Figure 2 、 3A and data exchange during the reduce-scatter operation described in 3B.

[0081] refer to Figure 4 , which illustrates the Figure 12 An example embodiment of a computer 700 is described. This example embodiment can be applied to control data exchange between processors in the context of machine learning as described above. Figure 4, data is exchanged between processors 2 of computer 700 during certain parts of the exchange phase.

[0082] like Figure 4 As shown in , during a particular portion of the execution, the execution unit 420 of each processor 2 is configured to select data segments / elements from its data memory 405 based on the index value held in its index storage 415. The execution unit 420 of processor N0 selects data labeled RA0 based on the index value held in storage 415 by processor N0 and transmits it to processor N1. The execution unit 420 of processor N1 selects data labeled YA0 based on the index value held by processor N1 and transmits it to processor N2. The execution unit 420 of processor N2 selects data labeled GA0 based on the index value held by processor N2 and transmits it to processor N3. The execution unit 420 of processor N3 selects data labeled BA0 based on the index value held by processor N3 and transmits it to processor N4. The execution unit 420 of processor N4 selects data labeled PA0 based on the index value held by processor N4 and transmits it to processor N5. The execution unit 420 of processor N5 selects data labeled LA0 based on the index value held by processor N5 and transmits it to processor N0.

[0083] In addition to each processor 2 using its index value to select data to transmit, upon receiving a data packet from another processor 2, the execution unit 420 of each processor 2 determines where in memory 405 to store a data segment derived from the received data packet based on its index value stored in index storage 415. Each processor 2's execution unit 420 also selects any other data segments to combine with the received data. For example, the execution unit 420 of processor N0 receives data labeled LA0 from processor N5 and, based on its index, executes an instruction to reduce this data with data LA1 held at location L in memory 405. The execution unit 420 of processor N1 receives data labeled RA0 from processor N0 and, based on its index, executes an instruction to reduce this data with data RA1 held at location R in memory 405. The execution unit 420 of processor N2 receives data labeled YA0 from processor N1 and, based on its index, executes an instruction to reduce this data with data YA1 held at location Y in memory 405. The execution unit 420 of processor N3 receives data labeled GA0 from processor N2 and, based on its index, executes an instruction to reduce this data with data GA1 held at location G in memory 405. The execution unit 420 of processor N4 receives data labeled BA0 from processor N3 and, based on its index, executes an instruction to reduce this data with data BA1 held at location B in memory 405. The execution unit 420 of processor N5 receives data labeled PA0 from processor N4 and, based on its index, executes an instruction to reduce this data with data PA1 held at location P in memory 405. The execution unit 420 of processor N0 receives data labeled LA0 from processor N5 and, based on its index, executes an instruction to reduce this data with data LA1 held at location L in memory 405.

[0084] Each transmitted data segment can be Figure 3A and 3B . In this case, each data segment corresponds to a subset of the incremental values calculated by each processor 2 using its training data. Collectively, the data values held in memory 405 and represented by the letters R, Y, G, B, P, L form a partial vector of incremental values calculated using the training data provided to the corresponding processor 2. Figure 4 Pictured Figure 2, wherein a reduced set of segments comprising one or more delta weights is provided to each processor 2. Each segment of one or more delta weights transmitted to a receiving processor 2 is combined with a corresponding segment of delta weights held in the memory 405 of that receiving processor. For example, segment RA0 is transmitted from processor N0 to processor N1 and is stored in memory 405 of that receiving processor. Figure 4 4. The fragments are combined with the fragment RA1 in the step shown in FIG. A reduce-scatter operation is performed by transmitting the fragments and combining them in subsequent steps. At each step, the fragments exchanged between the processors depend on the index value maintained in the index value storage 415. At least one execution unit of each processor 2 uses this index value to determine which fragment to send and where to store each received fragment in memory.

[0085] refer to Figure 4A , Figure 4A The diagram shows the data exchange during the full gather process, which is performed after the reduce-scatter process, as Figure 2 Part of the full reduction process shown in . Figure 2 In the embodiment, each fragment shown in the memory 405 in the processor is a reduced fragment, that is, the fragment that existed before the reduce-scatter operation began. Figure 4 The combination of corresponding fragments in the memory in .

[0086] At least one execution unit 420 of each processor 2 is configured to select and transmit the reduced fragment according to the index value stored therein. The execution unit 420 of processor N0 selects and transmits to processor N1 the fragment marked as The execution unit 420 of the processor N1 selects and transmits the data marked as The execution unit 420 of the processor N2 selects and transmits the data marked as The execution unit 420 of the processor N3 selects and transmits the data marked as The execution unit 420 of the processor N4 selects and transmits the data marked as The execution unit 420 of the processor N5 selects and transmits the data marked as Each of these data transfers completes the first step of the full gather operation. By executing the subsequent steps, each reduced segment is provided to each processor 2.

[0087] In addition to each processor 2 using its index value to select data to transmit, upon receiving data from another processor 2, the receiving processor determines where the data will be stored in memory based on the index value stored in the index storage device 415. For example, the execution unit 420 of processor N0 receives data labeled , and executes the instruction to store this data at location L in memory 405 according to its index. The execution unit 420 of processor N1 receives the data marked as from processor N0. , and executes the instruction to store this data at the location R in the memory 405 according to its index. The execution unit 420 of the processor N2 receives the data marked as from the processor N1. , and executes the instruction to store this data at location Y in memory 405 according to its index. The execution unit 420 of processor N3 receives the data marked as , and executes the instruction to store this data at location G in memory 405 according to its index. The execution unit 420 of processor N4 receives the data marked as , and executes the instruction to store this data at location B in memory 405 according to its index. The execution unit 420 of processor N5 receives the data marked as from processor N4. , and executes the instruction to store this data at location P in memory 405 according to its index. The execution unit 420 of processor N0 receives the data marked as and executes an instruction to store this data at location L in memory 405 according to its index.

[0088] Each processor includes a processor for transmitting data (such as Figure 4 and 4A The complete instruction set of a processor that transfers data (shown in a memory in FIG. 1 ) to another processor, where an index value controls the data transfer. The index can control the data transfer in different ways.

[0089] In some embodiments, the index value held by each processor determines the set of instructions that are executed by each execution unit 420 to transfer the appropriate data segment from memory 405 to another processor 2. Each processor 2 also includes a complete set of instructions for receiving and storing the segments at appropriate locations in memory 405. The index value held by each processor 2 determines the set of instructions that are executed by at least one execution unit 420 of the processor 2 to store the received segments at appropriate locations. In this case, the execution unit 420 performs a branch operation that depends on the index value to select the specific instruction set to be executed to perform the relevant data transfer.

[0090] In some embodiments, at least one execution unit 402 of each processor 2 performs an arithmetic operation using an index value as input to determine an address in the data memory 405 from which data is to be read or written.

[0091] exist Figure 4 and 4A In FIG, each processor 2 is shown as having a unified data memory 405 and instruction memory 410. However, these memories may be distributed among their respective processors 2. In some embodiments, each processor 2 may include multiple processing units, referred to herein as slices. Such an arrangement is described in U.S. Patent Application No. 15 / 886,315, which is incorporated by reference. Each slice includes a copy of the index of the processor 2 to which it belongs and is configured to execute operations from its instruction set that implement appropriate data transfer operations based on the index value, such as those described above with reference to FIG. Figure 12 、 4 and those operations discussed in 4A.

[0092] refer to Figure 4B , which illustrates an example of a multi-chip processor 2. Processor 2 includes an array 6 of multiple processor slices 4 and interconnects 34 connecting the slices 4. Processor 2 can be implemented separately as one of multiple dies packaged in the same IC package. Interconnect 34 may also be referred to herein as a "switch fabric" 34 because it enables the slices 4 to exchange data with each other. Each slice 4 includes a respective instance of a processor and memory. For example, by way of illustration, processor 2 may include on the order of hundreds or even over a thousand slices 4. For the sake of completeness, it is also noted that reference herein to an "array" does not necessarily imply any particular number of dimensions or physical layout of the slices 4.

[0093] In an embodiment, each processor 2 also includes one or more external links 8, thereby enabling the processor 2 to connect to one or more other processors (e.g., one or more other instances of the same processor 2). These external links 8 may include any one or more of the following: one or more processor-to-host links for connecting the processor 2 to a host processor, and / or one or more processor-to-processor links for connecting together with one or more other instances of the processor 2 on the same IC package or card or on different cards. In one example arrangement, the processor 2 receives work in the form of input data to be processed by the processor 2 from a host processor (not shown) connected to the processor via one of the processor-to-host links. Multiple instances of the processor 2 can be connected together into a card via the processor-to-processor links. Thus, the host accesses a computer having multiple processors 2, each of which is built as a multi-chip system on a chip, depending on the workload required by the host application.

[0094] Interconnect 34 is configured to enable different slices 4 in array 6 to communicate with each other. However, in addition to dependencies that may exist between threads on the same slice 4, dependencies may also exist between portions of programs running on different slices 4 in array 6. Therefore, a technique is needed to prevent a piece of code on one slice 4 from running before the data it depends on becomes available through another piece of code on another slice 4.

[0095] Each slice 4 is itself a processor capable of executing instructions (code) from a local instruction memory and processing data in a local data memory. A slice 4 may include respective instances of barrel-threaded processors and memories. For example, as an illustration, a processor 2 may include on the order of hundreds or even more than a thousand slices 4. For the sake of completeness, it is also noted that references herein to an "array" do not necessarily imply any particular number of dimensions or physical layout of slices 4.

[0096] Communication between slices 4 on processor 2 occurs in a time-deterministic manner. However, other forms of inter-slice communication are possible. Dependencies can exist between parts of a program running on different slices 4 in array 6. That is, processing data on one slice can depend on results from another slice, for example, providing results that another slice depends on. Therefore, a technique is needed to prevent a piece of code on one slice 4 from running before the data it depends on becomes available to another piece of code on another slice 4.

[0097] Parallel programming models for AI and data science typically follow a three-phase iterative execution model: compute, barrier, and exchange. This means that data transfers to and from the processor typically rely on barriers to provide data consistency between processors and between each processor and the host. Commonly used data consistency models are massive synchronous parallelism (BSP), stale synchronous parallelism (SSP), and asynchronous. The embodiments described herein use the BSP model, but other synchronization models can obviously be used as alternatives.

[0098] refer to Figure 5 and 6 , which illustrate an implementation of a BSP exchange scheme, where each slice 4 performs a computation phase 33 and an exchange phase 32 in alternating cycles, separated from each other by barrier synchronization 30 between the slices. Figure 5 and 6In the case shown, a barrier synchronization is placed between each calculation phase 33 and the subsequent exchange phase 32. During the calculation phase 33, each slice 4 performs one or more calculation tasks locally on the slice, but does not communicate any results of these calculations with any other slice in the slice 4. In the exchange phase 32, each slice 4 is allowed to exchange one or more results of calculations from the previous calculation phase to and / or from one or more other slices, but does not perform any new calculations until it receives any data from the other slices 4 on which its task(s) have dependencies. It also does not send any data to any other slice other than the data calculated in the previous calculation phase. It is not excluded that other operations, such as internal control related operations, may be performed in the exchange phase 32. Communication outside the slice group may optionally use the BSP mechanism, but may alternatively not use the BSP and may instead use some other synchronization mechanism of its own.

[0099] According to the BSP principle, barrier synchronization 30 is placed at the juncture of the transition from the computation phase 33 to the exchange phase 32, or at the juncture of the transition from the exchange phase 32 to the computation phase 33, or both. That is, either: (a) all slices 4 in the group are required to complete their corresponding computation phase 33 before any slice in the group is allowed to proceed to the next exchange phase 32, or (b) all slices 4 in the group are required to complete their corresponding exchange phase 32 before any slice in the group is allowed to proceed to the next computation phase 33, or (c) both conditions are enforced. In all three variations, it is the individual slices that alternate between phases, and the entire assembly is synchronized. The sequence of exchange and computation phases can then be repeated multiple times. In BSP terminology, each repetition of the exchange and computation phases is sometimes referred to as a "superstep" (note, however, that this terminology is not always used consistently in the literature: sometimes each individual exchange and computation phase is referred to as a superstep, while elsewhere, as in the terminology adopted herein, the exchange and computation phases are collectively referred to as a superstep).

[0100] It is also noted that it is not excluded that multiple different independent groups of slices 4 on the same processor 2 or different processors can each form a separate respective BSP group that operates asynchronously with each other, wherein the BSP cycle of computation, synchronization, and exchange is imposed only within each given group, but each group does so independently of the other groups. That is, the multi-slice array 6 can include multiple internal synchronization groups, each group operating independently and asynchronously with respect to the other such groups (discussed in more detail later). In some embodiments, there is a hierarchical grouping of synchronization and exchange, as will be discussed in more detail later.

[0101] Figure 6The diagram illustrates the BSP principle implemented among groups 4i, 4ii, 4iii of some or all tiles in the array 6, while imposing: (a) barrier synchronization (see above) from the computation phase 33 to the exchange phase 32. Note that in this arrangement, some tiles 4 are allowed to start computation 33 while other tiles are still exchanging.

[0102] Communication between slices 4 on processor 2 occurs in a time-deterministic manner, where data packets are transmitted without headers. This is explained in our earlier US patent application No. 15 / 886,315.

[0103] In an embodiment, multiple instances of processor 2 are connected together to form an even larger array of slices 4 that span multiple processors 2. Figure 7 Processor 2 is connected to external interconnect 72 (via Figure 7 The external interconnect 72 is connected together by an external link 8 shown in FIG. This can connect between processors 2 on the same IC package, different IC packages on the same card, and / or different IC packages on different cards. In addition to providing a conduit for exchanging data between slices 4 on different processors, the external interconnect 72 also provides hardware support for performing barrier synchronization between slices 4 on different processors 2 and aggregating local exit states of slices 4 on different processors 2.

[0104] Figure 8 The diagram illustrates an exemplary mechanism for communicating (external switching) between processors 2. This mechanism is non-time-deterministic. It is implemented in dedicated hardware logic within external interconnect 72. Data is sent via external interconnect 72 in packets. Unlike packets sent via internal interconnect 34, these packets have headers: because the order of transmission can change, they require the destination address to be present in the packet header. External interconnect 72 includes a routing table for statically routing data packets between different processors based on their headers.

[0105] At the physical layer, the interconnect mechanism is lossy, but at the transaction layer, due to the architecture of the link layer, the mechanism is not lossy: if a packet is not acknowledged, it will be automatically retransmitted by the hardware in the interconnect 72. However, the possibility of loss and retransmission at the data link layer means that the delivery of data packets through the external interconnect 72 is not time-deterministic. In addition, all packets for a given exchange can arrive together or separately in any order in time, so the external interconnect requires flow control and queuing. In addition, the interconnect can use clock data recovery (CDR) techniques to infer a clock from the received data stream with sufficient data signal transitions to maintain bit lock. The phase relationship of this inferred clock to the transmit clock is unknown, and therefore represents an additional source of non-determinism.

[0106] As shown, the external interconnect 72 includes an external switch block (XB) 78. The compiler specifies that one of the slices 4 sends an external switch request (XREQ) to the switch block 78 (operation S1). The XREQ is a message including one or more control packets indicating which of the slices 4 have data packets (content) to be sent to another one or more slices 4 on another processor 2. Figure 8 This is schematically illustrated in FIG4 by ticks and crosses: as an example scenario, those marked with ticks have data packets to be sent externally, while those marked with crosses do not. In operation S2, switch block 78 sends a switch-on (XON) control packet to the first of the slices 4 that has data to be sent externally. This causes the first slice to begin sending its packets to the relevant destination via external interconnect 78 (operation S3). Data packets received from the first slice at the external interconnect are statically routed to the destination using the routing table in external interconnect 78. If at any time XB 78 is unable to continue sending packets to the interconnect (e.g., due to previous packet loss and retransmissions within the interconnect, or due to oversubscription of the external interconnect by many other XBs and slices), XB 78 will send a switch-off (XOFF) to that slice before the XB queue overflows. Once the congestion has cleared and XB 78 has sufficient space in its queue again, it will send an XON to the slice, allowing it to continue transmitting its content. Once the slice has sent its last data packet, the switch block 78 sends a switch off (XOFF) control packet to the slice in operation S4, and then sends another XON and the data packet to be sent to the next slice 4 in operation S5, etc. The signaling of XON and XOFF is implemented as a hardware mechanism in dedicated hardware logic in the form of external switch block 78.

[0107] refer to Figure 9 , which illustrates an example of a slice 4 as part of a processor 2. The slice 4 includes at least one execution unit 905, which can perform arithmetic and logical operations, address calculations, load and store operations, and other operations, as defined by instructions stored in an instruction memory 910. The execution unit 905 is configured to execute instructions that operate on data held in a data memory 915. The instructions held in the instruction memory 910 of each slice 4 can be considered to be held in memory for Figure 4 and 4A The instruction memory 410 of the entire processor 2 is shown as a subset of the instruction set. The instruction memory 410 can be considered as a distributed memory including the instruction memory 910 of each slice 4.

[0108] The instructions held in the instruction memory 910 may be identical in the corresponding slice 4 of each processor 2. Figure 10, which illustrates how the different slices 4 of each processor 2 correspond to one another. The first processor 2i includes slice 4ia. The second processor 2ii includes slice 4iia. The third processor 2iii includes slice 4iiia. Each of slices 4ia, 4iia, and 4iiia is loaded with the same instructions. The operations performed by the execution unit 905 of each slice 4ia, 4iia, and 4iiia differ depending on the index of the corresponding processor 2 to which they belong. Similarly, the first processor 2i includes slice 4ib. The second processor 2ii includes slice 4iib. The third processor 2iii includes slice 4iiib. Each of slices 4ib, 4iib, and 4iiib is loaded with the same instructions. The operations performed by the execution unit 905 of each slice 4ib, 4iib, and 4iiib differ depending on the index of the processor 2 to which they belong. Therefore, it will be appreciated that because each corresponding slice 4 in the processor 2 is loaded with exactly the same instruction set, each processor 2 in the computer 700 includes the same overall instruction set.

[0109] Return to reference Figure 9 During the computation phase, certain operations are performed on the data held in the data memory 915 to change the values of the data. Because the instructions held in the instruction memory 910 are the same across corresponding blocks 4 of different processors 2, the operations performed during the computation phase to manipulate the data held in the data memory 915 can be the same across corresponding blocks of processors 2. However, in some embodiments, the instructions held in the instruction memory 910 can be data-dependent, such that different operations are performed during the computation phase based on different sets of data held in the data memory 915.

[0110] Slice 4 includes index values held in index value store 920. Operations performed by execution unit 905 during the computation phase to manipulate data held in data store 915 are independent of the index values held in index value store 920. However, operations performed by execution unit 905 during the exchange phase with other processors 2 depend on the index values. Although index store 920 is shown as separate from instruction memory 910, in some embodiments, index store 920 and instruction memory 910 may form part of a single memory array.

[0111] As shown, the data held in the data memory 915 is divided into different parts / fragments (shown as slices). Before exchanging data with other slices, at least one execution unit 905 is configured to execute instructions to transfer data from one or more parts to a send buffer 925. The execution unit 905 selects the data to be transferred based on the index value held in the index memory 920. The execution unit 905 then passes the selected data to the send buffer 925. During the exchange phase, the execution unit 905 executes instructions to send data via the interface 8. Sending data via the interface 8 includes appending a header to the data packet with a destination address for the data in the other processor 2. According to the above description of Figure 8 In the scenario in question, the data packet is sent to that processor 2.

[0112] During the exchange phase, slice 4 is configured to receive one or more data packets. These data packets are received from slices on other processors 2. After receiving the one or more data packets, the data packets are stored in receive buffer 930. At least execution unit 905 executes instructions for processing the received data based on index values stored in storage device 920. At least one execution unit 905 is configured to store data derived from the data packets at a location in data memory 915 based on the index values stored in storage device 920. At least one execution unit 905 may also perform operations, such as reduction operations, using data from the received data packets and data stored in memory 915 before storing the results of the operations in memory 915.

[0113] Thus, the index value maintained in the memory 920 is used by at least one execution unit 905 for at least one of: selecting data at certain addresses from the memory 915 for transmission, selecting data for performing an operation (e.g., a reduction operation) on the received data, and storing a result derived from the received data at a specific address in the memory 915 depending on the index value.

[0114] There are different ways that index values can be used by execution unit 905 to select an address in memory 915 for storing data.

[0115] In some embodiments, the different data portions are arranged contiguously in memory 915. Execution unit 905 is configured to calculate the address of the data to be read or written in memory 915 based on the index value. Execution unit 905 calculates the address by performing the operation defined in the instruction in instruction memory 910. The operation is an arithmetic operation.

[0116] In some embodiments, instruction memory 910 stores multiple sections of code, each configured to control data transfer differently. For example, one section of code may cause data at a certain memory location in memory 915 to be transferred to another processor 2, while another section of code may cause execution unit 905 to cause data at a different memory location in memory 915 to be transferred to another processor 2. Another section of code may cause received data to be stored at a certain location in memory 915 or cause a certain operation to be performed on the received data. Execution unit 905 executes code from instruction memory 915, and at a point in the execution sequence where data is to be read from or written to memory 915, execution unit 905 executes a branch operation that selects a section of code for performing the read or write operation on the data. The section of code is selected based on an index value.

[0117] According to an exemplary application of the technology disclosed herein, each processor 2 is provided with a different training data set for generating incremental weights for training a machine learning model. In this case, each slice 4 is provided with a different set of training data for generating one or more incremental weights. All slices 4 of each processor 2 together generate a complete set of incremental weights, which are averaged with the incremental weights generated on other processors 2.

[0118] In some embodiments, execution unit 905 is configured to switch between processing different worker threads. In this case, execution unit 905 is part of the barrel-threaded processor described in U.S. Patent Application No. 15 / 886,315. In this case, each worker thread is programmed to perform computations associated with a corresponding processor in the machine intelligence graph. In this case, at least some of the edges between the processors correspond to data exchanges between threads. The threads exchanging data between them can be threads running on the same execution unit 905, or they can be threads running on execution units of different slices 4. Some exchanges may involve exchanges between different slices of processor 2. The slices shown in memory 915 can each correspond to a delta value associated with a particular edge between the processors, which delta value was calculated by execution unit 905 during training. Memory 915 is also shown as including additional data. This additional data may include data used to generate the delta value, such as training data, current values of weights, and any additional data defining the machine learning model (such as activation functions, the number of processors in each layer, etc.).

[0119] refer to Figure 11 , which illustrates a computer (such as Figure 12700) in the computer 700 shown in FIG. The instruction set and index values discussed above together constitute a program for execution on each processor 2. The compiled instruction set for each processor 2 is the same, and different index values for each processor 2 are added to the instruction set.

[0120] The method 1100 is performed by a compiler that may be executed on any suitable computing device including at least one execution unit and at least one memory for holding computer code for execution by the at least one execution unit.

[0121] At step S1110 , the compiler compiles a single executable instruction set to provide to each processor 2 . The single executable instruction set is in the form of an executable image. The executable instruction set may include multiple instruction subsets, each subset being executed by a different slice 4 of the processor 2 .

[0122] At step S1120 , the compiler determines an index value associated with each processor 2 in the computer 700 . Each determined index value uniquely identifies a different processor 2 within the computer 700 .

[0123] At step S1130, the compiler generates a native program for each processor 2, the native program comprising a single instruction set and an index value associated with processor 2. The compiler does this by taking the compiled instruction set generated in S1110 for each processor 2 and patching this instruction set with the index value of the processor determined in S1120.

[0124] It will be appreciated that the above embodiments are described by way of example only. While specific embodiments have been described, other applications and variations of the disclosed technology may become apparent to those skilled in the art once given the disclosure herein.

Claims

1. A computer comprising a plurality of processors, each processor configured to perform operations on data during a compute phase of the computer and, after a synchronization barrier, exchange data with at least one other processor during an exchange phase of the computer, each processor comprising: multiple processing units; at least one storage device comprising a program including an identical set of executable instructions and an index value identifying a corresponding processor among the plurality of processors, wherein the corresponding at least one storage device comprises at least one memory in each processing unit of the processor, each at least one memory storing a copy of the index value for the processor and a subset of the set of executable instructions; a plurality of data memories storing data for computations specified by the executable instruction set, wherein the plurality of data memories comprises a data memory in each of the plurality of processing units of a processor; a plurality of execution units, including an execution unit in each of the plurality of processing units of the processor, each execution unit in at least some of the execution units being arranged to execute one or more of a respective subset of executable instructions to: selecting, during an exchange phase of the computer, an operation for performing a data transfer between a data memory of the corresponding processing unit and at least one other processor according to the corresponding index value; as well as The selected operations for performing the data transfer are performed during the exchange phase.

2. The computer according to claim 1, wherein For each of the at least some processing units of each of the at least some processors, selecting an operation for performing the data transfer according to the corresponding index value includes selecting a corresponding at least one other processor to which to transfer the data.

3. The computer according to claim 1, wherein: For each of at least some of the processing units of each of at least some of the processors, the operation causes data to be transferred between the region of the corresponding data memory selected according to the corresponding index value and the corresponding at least one other processor.

4. The computer according to claim 3, wherein: For each of the at least some processing units of each of the at least some processors, the corresponding execution unit is configured to calculate an address of the corresponding region of the data memory by performing an arithmetic operation on the corresponding index value.

5. The computer according to claim 3, wherein: For each of the at least some processing units of each of the at least some processors, the respective execution unit is configured to branch to at least one instruction in the respective subset of executable instructions based on the respective index value, wherein the at least one instruction, when executed by the respective execution unit, causes data to be transferred between the respective region of the data memory and the respective at least one other processor.

6. The computer according to claim 3, wherein: The operations include, for each of the at least some of the processing units of each of the at least some of the processors, storing data received from the corresponding at least one other processor in a corresponding region of the data memory.

7. The computer according to claim 1, wherein: For each of the at least some processing units of each of the at least some processors, the operations include selecting data stored in a corresponding data memory according to a corresponding index value and transmitting the selected data to the corresponding at least one other processor.

8. The computer of claim 7 , wherein the execution unit of each of the at least some processing units of each of the at least some processors is configured to process the corresponding selected data to generate a data packet comprising the corresponding selected data and a header comprising an address of a buffer on the corresponding at least one other processor.

9. The computer of claim 1, comprising routing hardware configured to statically route data being transferred in a data transfer.

10. The computer according to claim 1, wherein For each processing unit of at least some of the processing units of each of the at least some processors: The operations include combining data received from the respective at least one other processor with further data stored in the respective data store; and The respective execution unit is configured to select the further data from the respective data memory according to the respective index value.

11. The computer according to claim 1, wherein For each processing unit of at least some of the processing units of each of the at least some processors: a corresponding data store including a plurality of delta weights for at least one machine learning model; as well as The transfer of data between the respective data store and the respective at least one other processor includes controlling the transfer of one or more delta weights between the respective data store and the respective at least one other processor.

12. The computer according to claim 11, wherein For each processing unit in at least some processing units of each of the at least some processors, the corresponding at least one execution unit is configured to select one or more incremental weights from the corresponding data memory according to the corresponding index value, and to cause the selected one or more incremental weights to be transmitted to the corresponding at least one other processor.

13. The computer according to claim 11 or claim 12, wherein: For each processing unit in at least some of the processing units of each of the at least some processors, the corresponding execution unit is configured to: selecting the one or more incremental weights from a corresponding data storage according to a corresponding index value; as well as The selected one or more delta weights are reduced with one or more additional delta weights received from the corresponding at least one other processor.

14. The computer of claim 1, wherein each processor is a system on a chip.

15. A computer-implemented method of generating a plurality of programs, each program being adapted for execution by a processor of a computer comprising a plurality of processors, each processor having a plurality of processing units, each processing unit comprising at least one memory for holding a subset of the programs for the respective processor, an execution unit for executing the respective subset of the respective programs, and a data memory for holding data, the method comprising: Compile a single executable instruction set; determining, for each processor, an index value associated with the corresponding processor; as well as Generate a native program for each processor consisting of a single instruction set and corresponding index values, wherein each subset of each local program is scheduled for execution on one of the execution units of the corresponding processor when the local program is assigned to the corresponding processor of the local program such that: Performing operations on data during the computational phase of a computer; selecting, in accordance with the corresponding index value, an operation for performing a data transfer between a data memory of the corresponding processing unit and at least one other processor during an exchange phase of the computer; as well as Following the synchronization barrier, the selected operations for performing the data transfer are performed during the exchange phase.

Images