Parallel processing of reduction and broadcast operations on large datasets of non-scalar data

Abstract

The present application relates to methods, systems and apparatus for parallel processing of reduction and broadcast operations of large data sets of non-scalar data, including instructions encoded on a storage medium for performing reduction of gradient vectors and similar structured data, which are generated in parallel on nodes organized in a grid or torus topology defined by connections in at least two dimensions between the nodes. The method provides parallel computation and communication between nodes in the topology.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application 201810153965.2, filed on February 22, 2018. Technical Field

[0003] This specification relates to parallel processing of large data sets of non-scalar data. Background Art

[0004] Machine learning models receive input and generate output based on the received input and the values ​​of the model parameters. These models usually need to be trained using the received input, which is a very large data set. Summary of the invention

[0005] This specification describes techniques related generally to parallel processing of large data sets, and specifically to parallel processing of large data sets in various topologies for faster reduce, map, and broadcast operations.

[0006] In general, one innovative aspect of the subject matter described in this specification can be implemented in a method for processing training data. The method includes training a respective copy of a machine learning model on each of a plurality of nodes organized in a torus topology comprising rows and columns of nodes, wherein each node is trained in parallel on a respective batch of training data, whereby after training each node holds a respective gradient vector resulting from the training. The respective gradient vectors in the nodes can be combined to generate a final gradient vector by performing an operation, the operation comprising: performing, by code executed on the nodes, a modified rotating clamp algorithm on the rows of the torus in parallel without broadcasting to generate combined data for each respective non-intersecting field of gradient vectors in each row, whereby when the row is processed, each column contains all combined data for the respective same non-intersecting fields of gradient vector data; and then performing, by code executed on the nodes, a second torus algorithm in each of the columns in the torus, including broadcasting the respective final column result in the column to all nodes in the column; and then replicating the final column result across each row of the torus, whereby each node of the torus holds the final gradient vector.

[0007] These and other embodiments may optionally include one or more of the following features. The second ring algorithm may be an improved rotating clamp algorithm. Multiple nodes may be implemented on a single module. Each node may be implemented on a separate integrated circuit. The module may be an integrated circuit. Executing the improved rotating clamp algorithm may include executing the improved rotating clamp algorithm to generate summation data for each corresponding non-intersecting field of the gradient vector. The torus may be composed of rows and columns of nodes. The torus topology may also include node layers, each layer consisting of corresponding rows and columns of nodes. Executing the improved rotating clamp algorithm for each row may include: for each node in each row, using the node as a starting node for instantiation of the improved clamp algorithm, the improved clamp algorithm sends half of the non-intersecting field of the node to each of the two adjacent nodes in the ring of the row; and executing the improved clamp algorithm starting from the node.

[0008] For each starting node in the row, each ending node opposite the starting node may broadcast the final field gradient vector determined by the ending node to the other nodes in the row. The combining may be performed along the columns in parallel with the combining along the rows of the annulus, wherein each node in each row is a starting node in the row, and each node in each column is a starting node in the column, and each starting node processes a different corresponding non-intersecting field of the gradient vector.

[0009] In some embodiments, a method for processing training data may include: training a corresponding copy of a machine learning model on each of a plurality of nodes organized in a torus topology comprising rows and columns of nodes, wherein each node is trained in parallel on a corresponding batch of training data, whereby after the training each node retains a corresponding gradient vector resulting from the training; combining the corresponding gradient vectors in the nodes by performing an operation to generate a final gradient vector, the operation comprising: performing a corresponding toroidal reduction along each row of the torus by code executed on the node, so that each node in each row has a reduced vector for each gradient vector originally in the node in the row; and performing a corresponding toroidal reduction along each column of the torus by code executed on the node, whereby each node retains the same final gradient vector at the end of the torus.

[0010] These and other embodiments may optionally include one or more of the following features: The ring reduction may be performed using a spinning clamp algorithm.

[0011] In some embodiments, a method for processing training data may include: training a corresponding copy of a machine learning model on each node on a node module, wherein the node module is organized in a topology consisting of circular rows with surrounding links and non-circular columns of nodes without surrounding links, wherein each node is trained in parallel on a corresponding batch of training data, whereby after the training each node retains a corresponding gradient vector obtained by the training; combining the corresponding gradient vectors in the nodes by performing operations to generate a final gradient vector, the operations including: performing a corresponding circular reduction along each row of the topology through code executed on the node, so that each node in each row has a reduced vector for each gradient vector originally in the node in the row; and performing a corresponding linear reduction along each column of the torus through code executed on the node, whereby each node retains the same final gradient vector at the end of the linear reduction.

[0012] In some embodiments, a method for processing training data includes: storing a different batch of data at each of a plurality of nodes organized in a torus topology comprising rows and columns of nodes; combining corresponding batches of data in the nodes to generate a final batch of data by performing operations, the operations including: performing, by code executed at the nodes, a corresponding toroidal reduction along each row of the torus such that each node in each row has a combined data batch vector for each batch of data originally in the node in the row; and performing, by code executed at the nodes, a corresponding toroidal reduction along each column of the torus, each node retaining the same final combined batch of data at the end of the toroidal reduction.

[0013] These and other embodiments may optionally include one or more of the following features. Each annular reduction of rows and columns along the annulus may be performed by code implementing a modified spinning clamp algorithm. The combining may be a reduction of a batch of data.

[0014] The details of one or more embodiments of the present invention are set forth in the accompanying drawings and the following description, and the accompanying drawings are given as illustrations only. Other features, aspects and advantages of the present invention will become apparent from the specification, drawings, and claims. Similar reference numerals and names in the various drawings indicate similar elements.

[0015] Another embodiment of the present disclosure includes a method for performing reduction of gradient vectors and similar structured data, such as generated in parallel on nodes organized in a mesh or torus topology defined by connections in at least two dimensions between the nodes. The method provides parallel computation and communication between nodes in the topology.

[0016] In some embodiments, a machine-readable storage medium, such as a non-transitory machine-readable storage medium, stores instructions that, when executed, cause a processing device to perform operations including a method according to any embodiment or aspect described herein.

[0017] In some embodiments, a system includes: a memory; and a processing device operably coupled with the memory to perform operations including the method according to any embodiment or aspect described herein.

[0018] The above-described embodiments form an alternative solution for improving the training of machine learning models by processing the training data in a corresponding manner based on the topology in which the plurality of nodes are organized. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 An example system illustrating machine learning model replicas.

[0020] Figure 2 An example of processing units connected in a ring topology on an example module is illustrated.

[0021] Figure 3 Diagram showing a torus topology of replicas and processing units.

[0022] Figure 4A is a flowchart illustrating processing performed by an intermediate processing unit.

[0023] Figure 4B is a flowchart illustrating the processing performed by the final processing unit.

[0024] Figure 5A is a flow chart illustrating processing performed by a processing unit in a two-dimensional (2D) grid or torus topology.

[0025] Figure 5B is a flow chart illustrating processing performed by a processing unit in a three-dimensional (3D) grid or torus.

[0026] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0027] A common strategy for training machine learning models is to process batches of training data in parallel across multiple processing units. Figure 1 An example system 100 of machine learning model replicas AD is illustrated, each of which is trained on a corresponding processing unit 110. For convenience, in the drawings, both the replica and the processing unit where it resides may be referenced by the same name.

[0028] When training, a machine learning model is defined by the values ​​of model parameters. Model parameters are typically organized as non-scalar data, such as vectors, two-dimensional (2D) matrices, three-dimensional (3D) matrices, or higher-level matrices, whose elements are typically scalar values, such as integers or floating-point numbers.

[0029] In the system, each replica is trained on a unique batch of training data. Figure 1 In , replicas AD are trained on batches 1-4 105 respectively. When a replica has processed its batch of training data, it has a set of gradients for the model parameter values. The structure of the gradient values ​​in each replica is the same and generally corresponds to the structure of the parameter values. For convenience, both are called vectors.

[0030] Because the replicas are trained on different data, the gradient vectors of the replicas are combined to generate a final gradient vector, which is used to update the parameter values ​​of the model, i.e., the parameter vector. One way to combine the gradient vectors is to generate an element-wise average. The updated parameter values ​​are typically transmitted to all machine learning model replicas in anticipation of processing a batch of training data and combining, i.e., reducing, the gradient vectors of each replica into a reduced, i.e., final gradient vector and updating another iteration of the model's parameter values.

[0031] Some machine learning systems have a central parameter computation subsystem that receives gradient vectors from each replica and combines them at a central location. This specification describes techniques that can be used in machine learning systems that do not have a central parameter computation subsystem but instead reduce gradient vectors and update parameter values ​​in a distributed manner among the replicas.

[0032] Figure 2 The topology of a high-speed connection connecting an example aggregation of processing units AH (202a-202h) connected in a ring topology is illustrated. Advantageously, all processing units are on a single module 200. The lines between the processing units shown in this figure represent high-speed data communication links. The processing units are advantageously manufactured on one integrated circuit ("chip"), but they can also be manufactured on multiple chips. Links across chip boundaries are inter-chip network links; processing units on the same chip communicate via intra-chip interface links. The link can be a half-duplex link on which only one processing unit can transmit data at a time, or it can be a full-duplex link on which data can be transmitted in both directions simultaneously.

[0033] A method for performing a reduction in this topology will now be described with reference to processing units AH. The reduction has a summing step, a computing step and a broadcasting step.

[0034] Single Path Algorithm

[0035] The single path algorithm causes one of the processing units, for example unit A, to send data to a neighboring unit, for example unit B. B adds the data from A to its own data and sends the sum to C, and so on. This sending and processing of data is advantageously performed in a streaming manner so that, for example, B can start adding before it has received all the data from A. The final combination occurs in unit H. The final gradient vector is then transmitted to the other units. This can be achieved by reversing the streaming data path. Alternatively, if a broadcast function is available for a processing unit that allows one unit to broadcast to multiple units, the broadcast function can be used to complete the transmission of the final gradient vector. For convenience, this part of the process will be referred to as broadcasting, even though each processing unit can only transmit the final output to one other processing unit.

[0036] Clamp Algorithm

[0037] To achieve lower latency than a single-path algorithm, the clamp algorithm sends data across two paths in opposite directions on the ring. So, for example, unit A and unit B start communicating at the same time, so that F and E end up with the sum across the paths leading to them from A and B, respectively. If the number of units is even, the data must be transmitted from F to E, or vice versa, and combined there before being broadcast to all units.

[0038] Rotating Clamp Algorithm

[0039] The rotating clamp algorithm described below further reduces latency.

[0040] In the rotating clamp algorithm, independent instantiations of the clamp algorithm are executed simultaneously starting from each pair of adjacent processing units. Thus, when the clamp algorithm is run starting from units A and B, the same algorithm is run starting from units B and C, and from C and D, and so on.

[0041] The gradient vectors are split into disjoint subsets of equal or substantially equal size where possible, and each instantiation of the algorithm processes one of these subsets. This may change the way streaming works, because for example now while C is sending one subset of data to D, B will send another subset of data to C. Therefore C will not forward the data from B until it has sent its own data. For this purpose, the data transmitted from B to C is temporarily stored on C. Alternatively, the processing of data arriving at C from B and D can be interleaved.

[0042] Improved Rotary Clamp Algorithm

[0043] If the number of nodes is even, the improved rotating clamp algorithm described below further reduces the delay.

[0044] In the improved rotating clamp algorithm, unit A first sends half of its data to unit H and the other half to B, and then the clamp algorithm continues to work as usual. This makes the data path symmetrical in both directions. For broadcasts in this algorithm, for example, if unit E wants to broadcast to all other nodes, it sends the data in the opposite direction, and as a last step, units B and H send disjoint halves of the data to A so that A still receives all the data.

[0045] Gradient reduction on torus

[0046] Figure 3 A torus topology of replicas and processing units is illustrated. For convenience, 16 replicas and processing units A1 to D4 are illustrated, each of which is part of two rings of four units - a horizontal ring and a vertical ring, and each of which has direct links to four other units. For a topology for processing large machine learning models - such as models with 10, 25, or 50 million parameter values ​​- a suitable number of processing units would be larger, for example, 16 units in each ring and 256 units total. Greater or fewer numbers would be appropriate for larger or smaller models.

[0047] Iterative Reduction Algorithm

[0048] The iterative reduction algorithm on the torus simplifies the reduction on the torus into a series of reductions on the ring. The first step is to perform a ring reduction along each row, using only links A1-A2, A2-A3, A3-A4, and A4-A1 for row A.

[0049] First, any of the ring reduction algorithms described above are performed in each row so that each cell in a row has a sum vector for each gradient vector originally in the cells of the row.

[0050] Then, a ring reduction is performed along each column, at the end of which the same final gradient vector is in each cell.

[0051] As disclosed above, the reduction has a summation step, a computation step, and a broadcasting step. The initial row-ring reduction should not perform a full computation step, because even if the computation on the ring has been completed, the gradients have not actually been summed over the ring surface.

[0052] Rotation Iterative Reduction Algorithm

[0053] An improvement to the iterative reduction algorithm is the rotated iterative reduction algorithm, where the same algorithm is instantiated twice, rotated 90 degrees in parallel on half the data. One instantiation processes the rows and then the columns, and the rotated instantiation processes the columns and then the rows. The two instantiations never use the same communication link at the same time, so this doubles the bandwidth utilization of the torus's links.

[0054] Recursive reduction algorithm

[0055] A further improvement is achieved in the recursive reduction algorithm. In this algorithm, a modified rotating clamp algorithm is first performed on the rows of the torus without broadcasting, where the summed data is updated and sent to other cells in the row. In this way, each cell has summed the data of a disjoint subset of the gradient vectors. Therefore, when the row processing is completed, each column contains all the data of the same subset of the gradient vector data.

[0056] Then, a second ring reduction is performed along the columns, which involves updating the weights and broadcasting the results to all cells in the column. To complete the reduction, the recursive reduction algorithm process copies the data across each row. This provides optimal broadcasting over the row-ring.

[0057] In terms of the data amount N and the number C of processing units in a row of the torus, the recursive reduction algorithm actually runs row reduction on N data and column reduction on N / C data, compared with the column reduction on N data of the iterative algorithm.

[0058] Rotation recursive reduction algorithm

[0059] The recursive reduce algorithm can be rotated and replicated in the same manner as described above for the rotated iterative reduce algorithm to double the use of link bandwidth to a theoretically optimal amount. This is because the algorithm only sends the data that needs to be sent, and it saturates all links in both directions until the algorithm completes.

[0060] Alternative topologies

[0061] The algorithm described in this specification can also be used in other topologies in principle, for example, in a 16x4 topology where there are wrap-around links in the dimension with 16 cells and no wrap-around links in the dimension with 4 cells. The algorithm would be modified to use circular reduction in the dimension with wrap-around links and some form of linear reduction in the dimension without wrap-around links, such as the single path algorithm described above. Similarly, the algorithm can be modified to work on a mesh topology without wrap-around links.

[0062] Implementation of the algorithm

[0063] Each algorithm may be implemented in a distributed manner by processes or threads running on processing units.

[0064] The process may be configured via control messages that indicate, for example, for each reduction instantiation which subset of gradient vectors the process is to process, whether it is the starting cell, final cell, or intermediate cell in the reduction, and in which direction the data or combined data and broadcast should be sent, as described above.

[0065] Programming of gradient reduction process operations

[0066] Figure 4A is a flow chart illustrating the process performed by each intermediate processing unit. Each intermediate processing unit combines its gradient vector or a portion of its gradient vector with the input gradient vector it received when it received a portion of the input gradient vector from the previous processing unit (410). The combining operation performed by each processing unit may be a simple summation or some other calculation of the combined gradient vectors. The intermediate processing unit then sends the combined gradient vector to the next processing unit in the direction (425).

[0067] Figure 4B 4 is a flow chart illustrating the process performed by each final processing unit. The final processing unit combines its gradient vector with the input gradient vector it receives when it receives the input gradient vector from the previous processing unit in the data path direction, and generates a final reduced gradient vector (430).

[0068] The final processing unit generally combines the reduced gradient vector with the values ​​of the machine learning model parameters to produce an updated set of parameter values ​​(432). Assuming the parameter vector is x and the gradient is dx, a simple update has the form: x += -learning_rate*dx, where learning_rate is a scalar term. The update rules can be arbitrarily complex, for example, they can depend on previous gradients. After computing the updated parameters, the final processing unit initiates a broadcast operation that provides the updated parameter values ​​to all other processing units by reversing the data flow and sending the final output through the processing units in a direction all the way back to the root processing unit (435). As a result, each processing unit will have updated parameters for processing the next batch of machine learning training data.

[0069] The process will be referred to as a broadcast process even though each processing unit transmits the final output to only one other processing unit.

[0070] Figure 5A 1 is a flow chart illustrating processing performed by processing units in a two-dimensional (2D) grid or torus topology. The processing units simultaneously perform gradient reduction processes in multiple first dimension data paths (510). The processing units of the 2D grid or torus then perform at least one gradient reduction process in a second dimension data path using the results of the gradient reduction processes from the first dimension (525).

[0071] Figure 5Bis a flow chart illustrating processing performed by processing units in a three-dimensional (3D) grid or torus topology. The processing units simultaneously perform gradient reduction processes in multiple first-dimensional data paths (530). The processing units of the 3D grid or torus then perform multiple gradient reduction processes in a second-dimensional data path using the results of the gradient reduction processes from the first dimension (540). Finally, the processing units of the 3D grid or torus perform at least one gradient reduction process in a third-dimensional data path using the results of the gradient reduction processes in the second dimension (1050).

[0072] Computer Program - which may also be called or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0073] The processes and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on instance data and generating outputs. Although this specification specifically relates to processing units processing gradient vectors, the processing units may perform reduction processing on any non-scalar instance data, i.e., tensors, matrices, arrays, or any other data structure that can hold data that can be split and processed in parallel. The processes and logic flows may also be performed by dedicated logic circuits, such as FPGAs or ASICs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0074] Computers suitable for executing computer programs can be based on general or special microprocessors or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated into the special logic circuit. Generally speaking, a computer will also include one or more large-capacity storage devices for storing data, such as disks, magneto-optical disks or optical disks, or operably coupled to receive data from it or to transmit data to it, or both. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device, such as a universal serial bus (USB) flash drive, just to name a few examples.

[0075] Embodiment 1 is a method for processing training data, comprising:

[0076] training a respective copy of the machine learning model on each of a plurality of nodes organized in a torus topology comprising rows and columns of nodes, wherein each node is trained in parallel on a respective batch of training data, whereby after the training each node retains a respective gradient vector resulting from the training;

[0077] The corresponding gradient vectors in the nodes are combined by performing an operation to generate a final gradient vector, the operation comprising:

[0078] performing, by code executed on the nodes, a modified rotating clamp algorithm in parallel on the rows of the torus without broadcasting to generate combined data for each corresponding non-intersecting field of gradient vectors in each row, whereby when processing of the row is complete, each column contains all combined data for corresponding identical non-intersecting fields of the gradient vector data;

[0079] performing, by code executed on the nodes, in each of the columns of the torus, a second toroidal algorithm, including broadcasting a corresponding final column result in each column to all nodes in the column; and then

[0080] The final column result is replicated across each row of the torus, so that each node of the torus holds the final gradient vector.

[0081] Embodiment 2 is a method according to embodiment 1, wherein the second ring algorithm is an improved rotating clamp algorithm.

[0082] Embodiment 3 is a method according to any one of embodiments 1-2, wherein the multiple nodes are implemented on a single module.

[0083] Embodiment 4 is a method according to any one of embodiments 1-3, wherein each node is implemented on a separate integrated circuit.

[0084] Embodiment 5 is a method according to any one of embodiments 1-4, wherein the module is an integrated circuit.

[0085] Embodiment 6 is a method according to any one of embodiments 1-5, wherein executing the improved spin clamp algorithm includes executing the improved spin clamp algorithm to generate summation data for each corresponding non-intersecting field of the gradient vector.

[0086] Embodiment 7 is a method according to any one of embodiments 1-6, wherein the annulus is composed of rows and columns of nodes.

[0087] Embodiment 8 is a method according to any one of embodiments 1-7, wherein the torus topology further includes node layers, each layer consisting of corresponding node rows and columns.

[0088] Embodiment 9 is a method according to any one of embodiments 1-8, wherein performing the improved rotation clamp algorithm on each row comprises:

[0089] For each node in each row:

[0090] using the node as a starting node for an instantiation of a modified clamp algorithm that sends half of the node's disjoint fields to each of two neighboring nodes in the ring of the row; and

[0091] The improved clamp algorithm starts to be executed at this node.

[0092] Embodiment 10 is a method according to any one of embodiments 1 to 9, further comprising:

[0093] For each starting node in the row, each ending node opposite to the starting node broadcasts the final field gradient vector determined by the ending node to other nodes in the row.

[0094] Embodiment 11 is a method according to any one of embodiments 1-10, further comprising:

[0095] The combining is performed along columns in parallel with the combining along rows of the torus, where each node in each row is a starting node in the row and each node in each column is a starting node in the column, and each starting node processes a different respective non-intersecting field of the gradient vector.

[0096] Embodiment 12 is a method for processing training data, the method comprising:

[0097] training a respective copy of the machine learning model on each of a plurality of nodes organized in a torus topology comprising rows and columns of nodes, wherein each node is trained in parallel on a respective batch of training data, whereby after the training each node retains a respective gradient vector resulting from the training;

[0098] The corresponding gradient vectors in the nodes are combined to generate a final gradient vector by performing operations including:

[0099] performing, by code executed at the nodes, a corresponding toroidal reduction along each row of the torus such that each node in each row has a reduced vector for each gradient vector originally in the node in the row; and

[0100] Through the code executed at the node, a corresponding ring reduction is performed along each column of the torus, with each node retaining the same final gradient vector at the end of the ring reduction.

[0101] Embodiment 13 is a method according to embodiment 12, wherein the ring reduction is performed using a rotating clamp algorithm.

[0102] Embodiment 14 is a method for processing training data, the method comprising:

[0103] training a respective copy of the machine learning model on each node on a node module, the node module being organized in a topology consisting of circular rows with surrounding links and non-circular columns of nodes without surrounding links, wherein each node is trained in parallel on a respective batch of training data, whereby after the training each node retains a respective gradient vector resulting from the training;

[0104] The corresponding gradient vectors in the nodes are combined to generate a final gradient vector by performing operations including:

[0105] performing, by code executed at the nodes, a corresponding ring reduction along each row of the topology such that each node in each row has a reduced vector for each gradient vector originally in the node in the row; and

[0106] Through code executed at the nodes, a corresponding linear reduction is performed along each column of the torus, at the end of which each node retains the same final gradient vector.

[0107] Embodiment 15 is a method for processing training data, the method comprising:

[0108] storing a different batch of data at each of a plurality of nodes organized in a torus topology comprising rows and columns of nodes;

[0109] The corresponding data batches in the nodes are combined by performing operations to generate a final data batch, the operations comprising:

[0110] performing, by code executed at the nodes, a corresponding toroidal reduction along each row of the torus such that each node in each row has a combined data batch vector for each data batch originally in the node in the row; and

[0111] By code executed at the nodes, a corresponding ring reduction is performed along each column of the torus, with each node retaining the same final combined data batch at the end of the ring reduction.

[0112] Embodiment 16 is a method according to embodiment 15, wherein each toroidal reduction is performed along rows and columns of the torus by code implementing a modified spin clamp algorithm.

[0113] Embodiment 17 is a method according to any one of embodiments 15 or 16, wherein the combination is a simplification of data batches.

[0114] Embodiment 18 is a system comprising:

[0115] A plurality of nodes organized into a topology of rows and columns and one or more storage devices storing instructions thereon, the instructions being operable when executed by the nodes to cause the nodes to perform the method described in any one of embodiments 1-17.

[0116] Embodiment 19 is a system according to embodiment 18, wherein the nodes are organized into a torus topology.

[0117] Embodiment 20 is a system according to any one of embodiments 18 or 19, wherein the nodes are organized in a torus topology on the modules.

[0118] Embodiment 21 is a storage medium encoded with instructions, which, when executed by nodes organized in a row and column topology, are operable to cause the nodes to perform the method described in any one of embodiments 1-17.

[0119] Embodiment 22 is the storage medium of embodiment 21, wherein the storage medium is one or more non-transitory storage devices.

[0120] Embodiment 23 is a storage medium according to any one of embodiments 21 or 22, wherein the nodes are organized in a torus topology.

[0121] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be unique to a particular embodiment of a particular invention. In this specification, the terms "vector", "vector data", and "vector element" are used broadly to refer to any non-scalar data. In addition to vectors, examples of non-scalar data are strings, arrays, structures, matrices, and tensors.

[0122] The reduce and update functions described in this specification are merely examples and are completely independent of the present invention itself. Furthermore, the present invention is described for machine learning, but can be used for any purpose involving reducing data distributed across a network.

[0123] Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, individual features described in the context of a single embodiment may also be implemented in multiple embodiments, either individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed as such, in some cases one or more features from a claimed combination may be removed from the combination, and a claimed combination may involve subcombinations or variations of subcombinations.

[0124] Similarly, although the operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order or that all of the operations shown be performed to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0125] Specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes described in the accompanying drawings do not necessarily require the specific order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A system for parallel processing of training data with low latency, the system comprising: A plurality of nodes organized into a topology of node rows and node columns, and one or more storage devices storing instructions that, when executed by the plurality of nodes, are operable to cause the plurality of nodes to perform: training a respective copy of the machine learning model on each of the plurality of nodes, wherein the plurality of nodes are organized in a torus topology comprising torus rows of nodes having surrounding links and torus columns of nodes having surrounding links, each node being a separate processing unit, and the respective copies on each node are trained in parallel on different batches of training data, whereby after the training the respective copy of each node maintains a respective gradient vector resulting from the training; The corresponding gradient vectors in the plurality of nodes are combined by performing an operation to generate a final gradient vector, the operation comprising: performing, by code executed at each of the plurality of nodes, a respective toroidal reduction along each toroidal node row of the torus topology such that each node in each toroidal node row has a reduced vector for each gradient vector originally in each node in the toroidal node row; and performing, by code executed at each of the plurality of nodes, a corresponding ring reduction along each ring node column of the torus topology, wherein each node retains the same final gradient vector at the end of the ring reduction, The ring reduction is performed along the ring node row or the ring node column, including: The intermediate node in the ring node row or the ring node column combines the input gradient vector received from the previous node in the ring node row or the ring node column with the corresponding gradient vector of the intermediate node, and transmits the combined gradient vector to the next node in the ring node row or the ring node column.

2. A non-transitory computer-readable storage medium having encoded thereon instructions operable when executed by a plurality of nodes organized in a topology of node rows and node columns to cause the plurality of nodes to perform: training a respective copy of the machine learning model on each of the plurality of nodes, wherein the plurality of nodes are organized in a torus topology comprising torus rows of nodes having surrounding links and torus columns of nodes having surrounding links, each node being a separate processing unit, and the respective copies on each node are trained in parallel on different batches of training data, whereby after the training the respective copy of each node maintains a respective gradient vector resulting from the training; The corresponding gradient vectors in the plurality of nodes are combined by performing an operation to generate a final gradient vector, the operation comprising: performing, by code executed at each of the plurality of nodes, a respective toroidal reduction along each toroidal node row of the torus topology such that each node in each toroidal node row has a reduced vector for each gradient vector originally in each node in the toroidal node row; and performing, by code executed at each of the plurality of nodes, a corresponding ring reduction along each ring node column of the torus topology, wherein each node retains the same final gradient vector at the end of the ring reduction, The performing of ring reduction along a ring node row or a ring node column comprises: causing an intermediate node in the ring node row or the ring node column to combine an input gradient vector received from a previous node in the ring node row or the ring node column with a corresponding gradient vector of the intermediate node, and transmitting the combined gradient vector to a next node in the ring node row or the ring node column.

3. A system for parallel processing of training data with low latency, the system comprising: A plurality of nodes organized into a topology of node rows and node columns, and one or more storage devices storing instructions that, when executed by the plurality of nodes, are operable to cause the plurality of nodes to perform: Training a corresponding replica of the machine learning model on each of the plurality of nodes, the plurality of nodes being organized in a topology consisting of circular node rows with surrounding links and non-circular node columns without surrounding links, wherein each node is a separate processing unit, and the corresponding replica on each node is trained in parallel on different batches of training data, whereby after the training, the corresponding replica of each node maintains a corresponding gradient vector obtained by the training; The corresponding gradient vectors in the plurality of nodes are combined by performing an operation to generate a final gradient vector, the operation comprising: Performing, by code executed on each of the plurality of nodes, a corresponding ring reduction along each ring node row of the topology so that each node in each ring node row has a reduced vector for each gradient vector originally in each node in the ring node row, wherein performing the ring reduction along each ring node row comprises: causing an intermediate node in the ring node row to combine an input gradient vector received from a previous node in the ring node row with a corresponding gradient vector of the intermediate node and transmitting the combined gradient vector to a next node in the ring node row; and A corresponding linear reduction is performed along each non-cyclic node column of the topology by code executed at each node of the plurality of nodes, with each node retaining the same final gradient vector at the end of the linear reduction.

4. A non-transitory computer-readable storage medium having encoded thereon instructions operable when executed by a plurality of nodes organized in a topology of node rows and node columns to cause the plurality of nodes to perform: Training a corresponding replica of the machine learning model on each of the plurality of nodes, the plurality of nodes being organized in a topology consisting of circular node rows with surrounding links and non-circular node columns without surrounding links, wherein each node is a separate processing unit, and the corresponding replica on each node is trained in parallel on different batches of training data, whereby after the training, the corresponding replica of each node maintains a corresponding gradient vector obtained by the training; The corresponding gradient vectors in the plurality of nodes are combined by performing an operation to generate a final gradient vector, the operation comprising: Performing, by code executed on each of the plurality of nodes, a corresponding ring reduction along each ring node row of the topology so that each node in each ring node row has a reduced vector for each gradient vector originally in each node in the ring node row, wherein performing the ring reduction along each ring node row comprises: causing an intermediate node in the ring node row to combine an input gradient vector received from a previous node in the ring node row with a corresponding gradient vector of the intermediate node and transmitting the combined gradient vector to a next node in the ring node row; and A corresponding linear reduction is performed along each non-cyclic node column of the topology by code executed at each node of the plurality of nodes, with each node retaining the same final gradient vector at the end of the linear reduction.

5. A system for parallel processing of training data with low latency, the system comprising: A plurality of nodes organized into a topology of node rows and node columns, and one or more storage devices storing instructions that, when executed by the plurality of nodes, are operable to cause the plurality of nodes to perform: storing a different batch of data on each of the plurality of nodes, the plurality of nodes being organized in a torus topology comprising torus rows of nodes having surrounding links and torus columns of nodes having surrounding links; The corresponding data batches in the plurality of nodes are combined by performing operations to generate a final data batch, the operations comprising: performing, by code executed at each of the plurality of nodes, a respective toroidal reduction along each toroidal node row of the toroidal topology such that each node in each toroidal node row has a combined batch of data for each batch of data originally in each node in the toroidal node row; and performing, by code executed at each of the plurality of nodes, a corresponding ring reduction along each ring node column of the torus topology, each node retaining the same final combined data batch at the end of the ring reduction, The ring reduction is performed along the ring node row or the ring node column, including: The intermediate node in the ring node row or the ring node column combines the input gradient vector received from the previous node in the ring node row or the ring node column with the corresponding gradient vector of the intermediate node, and transmits the combined gradient vector to the next node in the ring node row or the ring node column.

6. A non-transitory computer-readable storage medium having encoded thereon instructions operable when executed by a plurality of nodes organized in a topology of node rows and node columns to cause the plurality of nodes to perform: storing a different batch of data on each of the plurality of nodes, the plurality of nodes being organized in a torus topology comprising torus rows of nodes having surrounding links and torus columns of nodes having surrounding links; The corresponding data batches in the plurality of nodes are combined by performing operations to generate a final data batch, the operations comprising: performing, by code executed at each of the plurality of nodes, a respective toroidal reduction along each toroidal node row of the toroidal topology such that each node in each toroidal node row has a combined batch of data for each batch of data originally in each node in the toroidal node row; and performing, by code executed at each of the plurality of nodes, a corresponding ring reduction along each ring node column of the torus topology, each node retaining the same final combined data batch at the end of the ring reduction, The performing of ring reduction along a ring node row or a ring node column comprises: causing an intermediate node in the ring node row or the ring node column to combine an input gradient vector received from a previous node in the ring node row or the ring node column with a corresponding gradient vector of the intermediate node, and transmitting the combined gradient vector to a next node in the ring node row or the ring node column.

7. A method comprising: training a machine learning model on batches of training data by processing each batch on a corresponding computing node of a plurality of nodes interconnected according to a network topology comprising node rows and node columns, wherein each node is in exactly one of the rows and exactly one of the columns, wherein one node in each node row is designated as a start node for the row and one node in each node row is designated as an end node for the row, wherein each row has an even number of nodes, wherein each node other than the start node and the end node in each row has a corresponding previous neighbor node in the row and a corresponding next neighbor node in the row, wherein the end node in each row is a next neighbor node of two nodes adjacent to the end node in the row, wherein training the machine learning model comprises executing code by each node to cause the node to process a corresponding batch of training data through a corresponding replica of the machine learning model to obtain corresponding replica gradient data; Generating final gradient data according to the corresponding replica gradient data of each node in the plurality of nodes includes: For each row in the network topology: The corresponding starting node of the row sends (i) a first half of the replica gradient data obtained by the corresponding starting node to a first adjacent node in the row connected to the starting node, and (ii) the rest of the replica gradient data obtained by the starting node to a second adjacent node in the row connected to the starting node, For each node in the row except the starting node, combine the gradient data from the previous neighboring node of the node with the replica gradient data obtained by the node; Each node except the end node sends the corresponding combined gradient data on the node to the next neighboring node of the node, and determining, by the end node of the row, that the end node has combined the gradient data received from two neighboring nodes of the end node in the row, and in response, broadcasting, by the end node, the corresponding intermediate gradient data to each other node in the row, and then Calculating the final gradient data by each column in the network topology according to the corresponding intermediate gradient data of each node in the column; and The final gradient data is broadcasted to each node of the plurality of nodes.

8. The method according to claim 7, wherein one node in each node column is designated as a start node of the column and one node in each node column is designated as an end node of the column, wherein each column has an even number of nodes, wherein each node other than the start node and the end node in each column has a corresponding previous neighbor node in the column and a corresponding next neighbor node in the column, wherein the end node in each column is a next neighbor node of two nodes adjacent to the end node in the column, and The step of calculating the final gradient data by each column in the network topology according to the corresponding intermediate gradient data of each node in the column comprises: The starting node of the column sends (i) the first half of the intermediate gradient data obtained by the corresponding starting node to the first adjacent node in the column connected to the starting node, and (ii) the remaining part of the intermediate gradient data obtained by the starting node to the second adjacent node in the column connected to the starting node, By each node in the column except the starting node, combining the gradient data from the previous adjacent node of the node with the intermediate gradient data obtained by the node; Each node except the end node sends the corresponding combined gradient data on the node to the next neighboring node of the node, and Wherein broadcasting the final gradient data to each of the plurality of nodes comprises: determining, by the end node of each column, that the end node has combined gradient data received from two neighboring nodes of the end node in the column, and in response, broadcasting, by the end node, the combined intermediate gradient data to each other node in the column, wherein the combined intermediate gradient data is the final gradient data.

9. The method according to claim 7, The corresponding replica gradient data of each node includes a corresponding gradient vector, wherein the corresponding gradient vector includes a plurality of elements, and Wherein, for each node in the row except the starting node, combining the gradient data from the previous adjacent node of the node with the replica gradient data obtained by the node comprises: The elements of the gradient vector of the replica gradient data obtained by the node are used to perform element-by-element averaging on the gradient vectors of the gradient data from the preceding neighboring nodes of the node.

10. The method according to claim 7, The step of sending, by the starting node of the row, (i) a first half of the replica gradient data obtained by the corresponding starting node to a first adjacent node in the row connected to the starting node, and (ii) the rest of the replica gradient data obtained by the starting node to a second adjacent node in the row connected to the starting node comprises: The starting node of the row sends the replica gradient data stream to the first adjacent node and the second adjacent node; and The sending of the corresponding combined gradient data on the node by each node except the end node to the next adjacent node of the node includes: sending the corresponding combined gradient data stream on the node by each node to the next adjacent node of the node.

11. The method according to claim 10, wherein combining, by each node in the row except the start node, the gradient data from a preceding neighboring node of the node with the replica gradient data obtained by the node comprises: When the replica gradient data is streamed to the node, the gradient data from the node's preceding neighboring nodes are combined.

12. The method of claim 7, wherein the plurality of nodes are implemented on a single module.

13. The method according to claim 12, wherein: The single module is an integrated circuit.

14. The method of claim 7, wherein each node comprises one or more processing units and one or more non-transitory computer-readable storage media.

15. The method of claim 7, wherein the network topology is a three-dimensional torus topology.

16. A system, comprising: A node and one or more storage devices implemented on one or more processing units, wherein the plurality of nodes are interconnected according to a network topology comprising node rows and node columns, wherein the rows include wrap-around links, wherein each node is in exactly one of the rows and exactly one of the columns, wherein one node in each node row is designated as a start node for the row and one node in each node row is designated as an end node for the row, wherein each node in each row other than the start node and the end node has a corresponding previous neighbor node in the row and a corresponding next neighbor node in the row, wherein the end node in each row is a next neighbor node of two nodes adjacent to the end node in the row, and The one or more storage devices store media encoded with instructions, which, when executed by one or more computers, cause the one or more computers to perform operations, including: Training a machine learning model on the batches of training data by processing each batch on a corresponding computing node among the plurality of nodes, training the machine learning model on the batches of training data by processing each batch on a corresponding computing node among the plurality of nodes comprises: executing code by each node to cause the node to process the corresponding batch of training data through a corresponding replica of the machine learning model to obtain corresponding replica gradient data; Generating final gradient data according to the corresponding replica gradient data of each node in the plurality of nodes includes: For each row in the network topology: The corresponding starting node of the row sends (i) a first half of the replica gradient data obtained by the corresponding starting node to a first adjacent node in the row connected to the starting node, and (ii) the rest of the replica gradient data obtained by the starting node to a second adjacent node in the row connected to the starting node, For each node in the row except the starting node, combine the gradient data from the previous neighboring node of the node with the replica gradient data obtained by the node; Each node except the end node sends the corresponding combined gradient data on the node to the next neighboring node of the node, and determining, by the end node of the row, that the end node has combined the gradient data received from two neighboring nodes of the end node in the row, and in response, broadcasting, by the end node, the corresponding intermediate gradient data to each other node in the row, and then Calculating the final gradient data by each column in the network topology according to the corresponding intermediate gradient data of each node in the column; and The final gradient data is broadcasted to each node of the plurality of nodes. The system of claim 16 , wherein the columns include wraparound links.

18. The system of claim 17, wherein one node in each node column is designated as a start node of the column and one node in each node column is designated as an end node of the column, wherein each column has an even number of nodes, wherein each node other than the start node and the end node in each column has a corresponding previous neighbor node in the column and a corresponding next neighbor node in the column, wherein the end node in each column is a next neighbor node of two nodes adjacent to the end node in the column, and The step of calculating the final gradient data by each column in the network topology according to the corresponding intermediate gradient data of each node in the column comprises: The starting node of the column sends (i) the first half of the intermediate gradient data obtained by the corresponding starting node to the first adjacent node in the column connected to the starting node, and (ii) the remaining part of the intermediate gradient data obtained by the starting node to the second adjacent node in the column connected to the starting node, By each node in the column except the starting node, combining the gradient data from the previous adjacent node of the node with the intermediate gradient data obtained by the node; Each node except the end node sends the corresponding combined gradient data on the node to the next neighboring node of the node, and Wherein broadcasting the final gradient data to each of the plurality of nodes comprises: determining, by the end node of each column, that the end node has combined gradient data received from two neighboring nodes of the end node in the column, and in response, broadcasting, by the end node, the combined intermediate gradient data to each other node in the column, wherein the combined intermediate gradient data is the final gradient data.

19. The system of claim 16, wherein the node rows and the node columns are rows and columns of layers in a plurality of layers interconnected according to the network topology.

20. The system of claim 16, wherein the network topology is a three-dimensional torus topology.

21. The system of claim 16, wherein the training data batch is a first training data batch, and wherein the operations further comprise, after broadcasting the final gradient data to each of the plurality of nodes: Each node uses the final gradient data to update the model parameter values ​​of the machine learning model, and The machine learning model is trained on the second batch of training data.

22. The system of claim 16, wherein the plurality of nodes are implemented on a single module.

23. The system of claim 22, wherein the single module is an integrated circuit.

24. The system according to claim 16, The step of sending, by the starting node of the row, (i) a first half of the replica gradient data obtained by the corresponding starting node to a first adjacent node in the row connected to the starting node, and (ii) the rest of the replica gradient data obtained by the starting node to a second adjacent node in the row connected to the starting node comprises: The starting node of the row sends the replica gradient data stream to the first adjacent node and the second adjacent node; and The sending of the corresponding combined gradient data on the node by each node except the end node to the next adjacent node of the node includes: sending the corresponding combined gradient data stream on the node by each node to the next adjacent node of the node.

25. The system of claim 24, wherein combining, by each node in the row except the start node, the gradient data from a preceding neighboring node of the node with the replica gradient data obtained by the node comprises: When the replica gradient data is streamed to the node, the gradient data from the node's preceding neighboring nodes are combined.

26. One or more computer-readable storage media encoded with instructions for execution by one or more processing units implementing a plurality of nodes, wherein the plurality of nodes are interconnected according to a network topology comprising node rows and node columns, wherein the rows comprise wrap-around links, wherein each node is in exactly one of the rows and exactly one of the columns, wherein one node in each node row is designated as a start node for the row and one node in each node row is designated as an end node for the row, wherein each node in each row other than the start node and the end node has a corresponding previous neighbor node in the row and a corresponding next neighbor node in the row, wherein the end node in each row is a next neighbor node of two nodes adjacent to the end node in the row, and When the instructions are executed by one or more processing units, the one or more processing units are caused to perform operations, the operations comprising: Training a machine learning model on the batches of training data by processing each batch on a corresponding computing node among the plurality of nodes, training the machine learning model on the batches of training data by processing each batch on a corresponding computing node among the plurality of nodes comprises: executing code by each node to cause the node to process the corresponding batch of training data through a corresponding replica of the machine learning model to obtain corresponding replica gradient data; Generating final gradient data according to the corresponding replica gradient data of each node in the plurality of nodes includes: For each row in the network topology: The corresponding starting node of the row sends (i) a first half of the replica gradient data obtained by the corresponding starting node to a first adjacent node in the row connected to the starting node, and (ii) the rest of the replica gradient data obtained by the starting node to a second adjacent node in the row connected to the starting node, For each node in the row except the starting node, combine the gradient data from the previous neighboring node of the node with the replica gradient data obtained by the node; Each node except the end node sends the corresponding combined gradient data on the node to the next neighboring node of the node, and determining, by the end node of the row, that the end node has combined the gradient data received from two neighboring nodes of the end node in the row, and in response, broadcasting, by the end node, the corresponding intermediate gradient data to each other node in the row, and then Calculating the final gradient data by each column in the network topology according to the corresponding intermediate gradient data of each node in the column; and The final gradient data is broadcasted to each node of the plurality of nodes.

Images