Crossed copy reduction on networks with degenerate nodes

Abstract

Methods, systems, and apparatus, including instructions encoded on storage media, for performing a reduction of gradient vectors for a network having one or more degenerate nodes. One method includes training a respective copy of a machine learning model at each node of a plurality of nodes organized in an n-dimensional network topology, combining respective individual gradient vectors in the nodes to generate a final gradient vector by performing operations including designating each set of nodes along a dimension as either a forwarding group or a key group, updating the respective individual gradient vectors with an intermediate gradient vector for each receiving node, performing a reduction on each key node group along a dimension to generate a respective partial final gradient vector for the key group, and updating the individual gradient vector for a representative node with the respective partial final gradient vector for each key node group.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 543,410, filed August 16, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This specification pertains to the parallel processing of large datasets of nonscalar data. Background Technology

[0004] Machine learning models receive input and generate output based on the received input and the values ​​of the model parameters. These models typically require input from very large datasets for training.

[0005] Machine learning models can be trained on a distributed network configured to train multiple copies of the model on different batches of training data. The distributed network can define various topologies, such as grids or tori, and can have arbitrary dimensions. Each copy can be trained individually and in parallel, and the model parameter values ​​of the machine learning model can be updated based on the training results. Summary of the Invention

[0006] This specification generally describes techniques involving parallel processing of large datasets, and specifically relates to parallel processing of large datasets on a machine learning model replica network having one or more degraded nodes. The machine learning replica network (“replica network”) may have multiple nodes, wherein each node has a corresponding processing unit configured to train a corresponding replica of a machine learning model on a batch of training data. The training system for the replica network can train the machine learning model regardless of the one or more degraded nodes of the replica network, which may degrade, for example, due to hardware failure or communication failure.

[0007] Specific implementations of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. The training system can train the machine learning model even when one or more nodes in the replica network of the node training replicas of the machine learning model degenerate. The training system can route and distribute computation, enabling efficient use of the computational resources of non-degenerate nodes during training. Training systems implementing the techniques described in this specification can avoid further degradation or performance impairment caused by overheating in areas of the replica network when data traffic is naively rerouted around degenerate nodes, as can be done using conventional methods. The training system can train replica networks without physically modifying the replica network. The training system can train replica networks with mesh or toroidal topologies and arbitrary dimensions.

[0008] The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the application will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1 An exemplary training system is illustrated.

[0010] Figure 2A A replica network with a two-dimensional grid topology is illustrated.

[0011] Figure 2B A replica network with a two-dimensional torus topology is illustrated.

[0012] Figure 3A A replica network with a three-dimensional grid topology is illustrated.

[0013] Figure 3B A subnetwork of a replica network with a two-dimensional grid topology of representative nodes storing partial final gradient vectors after a first reduction is illustrated.

[0014] Figure 3C A subnetwork of a replica network with a one-dimensional grid topology of representative nodes storing updated partial final gradient vectors after a second reduction is illustrated.

[0015] Figure 4 An exemplary replica network with degenerate nodes is illustrated.

[0016] Figure 5A An exemplary replica network with partially degenerate nodes is illustrated.

[0017] Figure 5B An exemplary replica network with degenerate nodes in a corner of the replica network is illustrated.

[0018] Figure 5C An exemplary replica network with multiple non-contiguous degenerate nodes is illustrated.

[0019] Figure 6 An exemplary replica network with contiguous degenerate nodes is illustrated.

[0020] Figure 7 An exemplary replica network with a torus topology and degenerate nodes is illustrated.

[0021] Figure 8 is a flowchart of an exemplary process for computing final gradient vectors for a replica network with degenerate nodes.

[0022] Figure 9 is a flowchart of an exemplary process for performing an internal reduction on a subnetwork of non-degenerate nodes.

[0023] The same reference numerals and names in the various figures indicate the same elements. Detailed Implementation

[0024] Figure 1 An exemplary training system 100 is illustrated. The training system 100 can receive training data 105 and output data for a specified machine learning model 110. The training system 100 may include a partitioning engine 115 and a machine learning replica network 120.

[0025] Training system 100 can receive training data in any suitable manner. For example, training system 100 can receive training data as an upload from a remote user of the system via a data communication network, for example, using an application programming interface (“API”) provided by system 100. In some implementations, the training system can maintain a repository of training data, and system 100 can receive from the user specifying which data in the repository should be used as input to training data 105.

[0026] Specifying the data for the trained machine learning model 110 can specify the architecture of the machine learning model and the learning parameter values ​​of model 110. For example, the trained machine learning model 110 can be a neural network with multiple neural network layers including an input layer and an output layer. Each neural network layer can include one or more operation nodes configured to receive inputs and perform operations to generate output activations based on one or more learning parameter values ​​called weights. The output activations can be inputs to operation nodes in subsequent neural network layers, or they can be part of the output of the neural network, for example, because the operation nodes are located at the output layer of the neural network. It should be noted that neural networks and the operation nodes of neural networks—which are examples of machine learning models and their components—should not be confused with replica networks of machine learning and their associated nodes—which can be examples of computer networks.

[0027] The data specifying the trained machine learning model 110 may include learning parameter values ​​used for the machine learning model. In some implementations where the machine learning model is a neural network, the data defining the training model may specify training weights for each layer of the neural network. Model parameter values ​​are typically organized as non-scalar data, such as vectors, two-dimensional matrices or tensors, three-dimensional matrices or tensors, or higher-order matrices or tensors.

[0028] The data specifying the trained machine learning model 110 can be used in processing new input data. Alternatively, or in addition to generating data defining the trained machine learning model 110, the training system 100 can instantiate an instance of the trained machine learning model having the same architecture and learned parameter values as the machine learning model 110. The instance of the trained machine learning model can be executed to generate new outputs to perform the machine learning task.

[0029] The partition engine 115 can be configured to receive the training data 105 and generate batches of data 130. Each batch of training data in the batches of data 130 can be unique with respect to every other batch generated by the partition engine 115. In other words, in some implementations, no two batches of training data are the same. The batches of training data can be disjoint, i.e., none of the training data used in a batch is present in any other batch. Alternatively, there can be some overlap between the training data in some batches. The partition engine can send a respective batch of training data to each node 125a-n of the machine learning replica network 120 for processing by a respective machine learning model replica.

[0030] The machine learning replica network 120 (“replica network”) can include a plurality of nodes 125a-n. Each node 125a-n can include a processing unit configured to receive a batch of training data and process the batch of training data through a respective replica of a machine learning model. The processing unit can be implemented on one or more integrated circuits (“chips”) of one or more computing devices.

[0031] Each node 125a-n can be configured to train the respective replica through any appropriate machine learning training technique, e.g., through any appropriate supervised learning technique. In this specification, the term training is used to describe the process by which the training system 100 updates the parameter values of the model replicas in the replica network 120. For example, if the machine learning model is a neural network, each node 125a-n can process the training examples of an input batch through forward passes of each neural network layer of the respective model replica, and then update the weights at each neural network layer, e.g., by computing the gradient of a loss function using backpropagation of batch gradient descent.

[0032] The machine learning replica network 120 can also include one or more communication links between the plurality of nodes 125a-n. The communication links can be high-speed communication links. In some implementations in which the nodes are implemented on multiple chips, the links across chip boundaries are inter-chip network links. In some implementations in which the processing units of the nodes 125a-n are on the same chip, each node can communicate with other nodes through an intra-chip interface link.

[0033] The one or more links can be half-duplex links where only one processing unit can transmit data at a time. The one or more links can be full-duplex links that can transmit data in both directions simultaneously. In some implementations, the machine-learned replica network 120 includes one or more half-duplex links and one or more full-duplex links that connect together a plurality of nodes with processing units on the same chip and one or more other nodes with processing units implemented across multiple chips.

[0034] The machine-learned replica network 120 can define a network topology. The network topology refers to the configuration of nodes and links in the replica network. The network can also have one or more dimensions. Each link between nodes belongs to a respective dimension of the network topology. Each dimension can have one or more sets of nodes along the dimension.

[0035] Figure 2A A replica network with a two-dimensional grid topology 200a is illustrated. The two-dimensional grid consists of k1*k0nodes, where k i represents the number of nodes along one dimension of the grid. For example, the replica network 200a has a dimension of 4*4, for a total of 16 nodes. In general, an n-dimensional grid will have k n-1 *k n-2 *...k0nodes, where k i represents the number of nodes along the i-th dimension of the grid. In Figure 2A , the links indicated by solid lines belong to one dimension of the replica network, the “row” dimension, while the links indicated by dashed lines belong to another dimension of the replica network, the “column” dimension.

[0036] Also in Figure 2A , the replica network topology defines four sets of nodes along the row dimension, nodes A1-A4; B1-B4; C1-C4; and D1-D4, and four sets of nodes along the column dimension, nodes A1-D1; A2-D2; A3-D3; and A4-D4. For a replica network with a two-dimensional topology, each set of nodes in the row dimension will be referred to as a row, and each set of nodes in the column dimension will be referred to as a column.

[0037] Figure 2B A replica network with a two-dimensional torus topology 200b is illustrated. The two-dimensional torus consists of k1*k0nodes, where k i represents the number of nodes along one dimension of the torus. The two-dimensional torus consists of n 2 links, where n is the number of nodes of the torus. In addition to connecting nodes to each neighboring node in the replica network 200b, the torus also connects nodes on opposite sides of the torus. In general, an m-dimensional torus with n nodes will have n m links. In Figure 2BIn particular, the links indicated by solid lines belong to one dimension of the replica network, the "row" dimension, while the links indicated by dashed lines belong to another dimension of the replica network, the "column" dimension.

[0038] Also in Figure 2B and similarly to Figure 2A , the network topology defines four rows of nodes, namely, nodes A1-A4; B1-B4; C1-C4; and D1-D4 and four columns of nodes along the column dimension, namely, nodes A1-D1; A2-D2; A3-D3; and A4-D4.

[0039] Referring back to Figure 1 , the training system 100 can train each model replica 125a-n on a unique batch of training data partitioned by the partition engine 115. When a node has completed processing a batch of training data through the corresponding model replica, the node can generate a respective gradient vector for the values of the model parameters. Because the replicas are trained on different batches of training examples, the training system 100 can combine all the respective gradient vectors generated by all the replicas 125a-n and generate a final gradient vector. For example, the training system 100 can compute the average of each element in the respective gradient vectors to generate a final gradient vector representing an average gradient vector. The training system 100 can use a "reduction" algorithm as described below to generate a single gradient vector from multiple respective gradient vectors. This process can be iterated until a final gradient vector is obtained.

[0040] The training system 100 can use the final gradient vector to update the parameter values of the machine learning model. For example, a gradient descent algorithm can be used to determine the parameter updates, with the goal of minimizing a loss function. Other examples of optimization routines that can be used to determine updates to the parameter values of a machine learning model will be familiar to those skilled in the art. The training system 100 can then broadcast the new parameter values to each node 125a-n. If the training system 100 continues to train the machine learning replica network 120, each node 125a-n will use the updated parameter values to process subsequent training examples through the corresponding model replica for that node.

[0041] The training system 100 can reduce the respective gradient vectors generated by the nodes for each respective replica 125a-n to generate a final gradient vector by any appropriate reduction algorithm that, when executed on the processing units 125a-n, causes the training system 100 to:

[0042] 1. sum each respective gradient vector;

[0043] 2. compute updated model parameter values from the final gradient vector computed from the sum of the respective gradient vectors; and

[0044] 3. broadcast the updated model parameter values for the machine learning model to each node in the replica network 120.

[0045] These operations can be collectively summarized as performing a reduction on the individual gradient vectors at each node in the replica network. The training system can perform the reduction of the nodes in the replica network 120. In this specification, the period during which the training system performs the reduction along the nodes in a dimension is referred to as the “critical phase” for that dimension.

[0046] The reduction of the nodes refers to performing one or more operations on the set of nodes that take as input the respective individual gradient vectors of each node in the set and generate as output a single gradient vector, thereby “reducing” the multiple individual gradient vectors into a single vector. For example, the training system can perform a reduction on a set of nodes to obtain a partial final gradient vector for the set. As discussed below, the training system can iteratively reduce the partial final gradient vectors until reducing to the final gradient vector.

[0047] As discussed in more detail below, when the training system 100 trains a machine learning model with a replica network having one or more degenerate nodes, the training system 100 can perform operations in both the “critical phase” and the “forward phase” for each set of nodes along each dimension.

[0048] One exemplary class of reduction algorithms for reducing individual gradient vectors is a cyclic reduction algorithm. In some implementations, the training system 100 can be configured to perform a cyclic reduction algorithm on the machine learning replica network 120. A detailed discussion of the cyclic algorithm is provided in U.S. Patent No. 10,055,692, filed September 18, 2017, entitled “PARALLEL PROCESSING OF REDUCTION AND BROADCAST OPERATIONS ON LARGE DATASETS OF NON-SCALAR DATA,” which is incorporated by reference in its entirety.

[0049] Figure 3A -C illustrates an exemplary reduction of a replica network having a three-dimensional grid topology. In some implementations, the exemplary reduction is a cyclic reduction according to a cyclic reduction algorithm. Generally, for an n-dimensional replica network, the training system can perform n reductions to generate the final gradient vector.

[0050] At each dimension and for each set of nodes in the dimension, the training system can generate a respective partial final gradient vector by performing a reduction.

[0051] A representative node can be selected to store a partial final gradient vector for the group, and the representative node for each group becomes part of a subnetwork along the next dimension reduction. Specifically, the training system can reduce the subnetwork of representative nodes by repeating the reduction process to obtain updated partial final gradient vectors and a smaller subnetwork of representative nodes. The training system can repeatedly reduce the replica network until only one group of representative nodes remains. The training system can then reduce the last group of representative nodes and obtain a final gradient vector for the replica network. One example follows.

[0052] Figure 3A A replica network 300a with a three-dimensional grid topology is illustrated. Planes 305a-d in the replica network can be represented by replica subnetworks with a two-dimensional grid topology. Similarly, each row or column of planes 305a-d can be represented by replica subnetworks with a one-dimensional grid topology.

[0053] The training system— e.g., training system 100— can perform a first reduction on each group of nodes along the x-dimension, as illustrated by vectors 315a. The training system can perform the first reduction on each group in each plane 305a-d in parallel. After each first reduction, each group in replica network 300a will have a representative node with a partial final gradient vector for the group, as illustrated by the diagonally shaded nodes— e.g., nodes 310a. Figure 3A

[0054] Figure 3B A subnetwork 300b of replica network 300a with a two-dimensional grid topology of representative nodes storing partial final gradient vectors after the first reduction is illustrated. The training system can perform a second reduction on each group in subnetwork 300b to obtain an updated partial final gradient vector from the partial final gradient vectors of each representative node in the group. The training system can perform the second reduction on each group in subnetwork 300b in parallel. After each second reduction, each group in subnetwork 300b can have a representative node storing an updated partial final gradient vector for the group, as illustrated by the shaded nodes— e.g., nodes 305b.

[0055] Figure 3C ​A subnetwork 300c of the replica network 300a is illustrated, which has a one-dimensional grid topology of representative nodes that store updated partial final gradient vectors after a second reduction. The training system can perform a third and final reduction on the groups in the subnetwork 300c by reducing the respective updated partial gradient vectors stored by each representative node in the subnetwork 300c to obtain a final gradient vector. After the third reduction, the representative nodes can store the final gradient vector for the replica network 300a. For example, the node 305c can store the final gradient vector and update model parameters of a machine learning model trained by the replica network 300a. The node 305c can then broadcast the updated model parameters to each node in the replica network 300a.

[0056] Although described in terms of updating the representative nodes that store the partial final gradient vectors for the groups of nodes to which the representative nodes belong Figure 3A -C, in some implementations, each node in the group of nodes is updated with the partial final gradient vector.

[0057] Referring back to Figure 1 Some of the nodes 125a-n can be degraded. A degraded node is a node that is unable to generate or communicate individual gradient vectors to the training system 100 during a reduction. A node can be degraded due to a hardware issue—e.g., a physical failure of a respective processing unit, hardware having been removed from the node for replacement or maintenance. A node can also be degraded because a link between the node and another node is broken. A node can also be degraded due to an issue with software installed in memory at the node—e.g., a software failure, a software update currently in progress, or software maintenance at the node.

[0058] If a link at a node is broken, the node is referred to as being fully degraded or partially degraded. If all links to the node are broken such that data cannot be communicated to or from the node, the node is considered to be fully degraded. If some but not all links to the node are broken, the node is referred to as being partially degraded. When a node is partially degraded, the node is unable to communicate data along the dimension to which the broken link belongs. A partially degraded node is only considered to be degraded along the dimension with the broken link. Thus, a partially degraded node can be classified as being degraded along one dimension but not along another dimension.

[0059] The training system 100 can also be configured to classify a node as being degraded when the node is only adjacent to degraded nodes along each dimension of the replica network. In these cases, the node is considered to be degraded even if the node is able to generate individual gradient vectors for a batch of training data.

[0060] Figure 4An example machine learning replica network 400 with degenerate nodes is illustrated. Replica network 400 has a two-dimensional grid topology. Node B3 is a degenerate node, as indicated by the thatch mark in Figure 4 Nodes C3, C4, and D3 are also degenerate, as at least one of the links connecting to the nodes is broken, indicated by broken links 408 and 410. Specifically, node C4 is partially degenerate: node C4 is degenerate along the row dimension of replica network 400, as the row dimension has a broken link 408, but node C4 is not considered degenerate along the column dimension.

[0061] On the other hand, node C3 is considered a fully degenerate node: node C3 is degenerate along the row dimension due to broken link 408, and is also degenerate along the column dimension due to broken link 410 connecting to node C4.

[0062] Degenerate nodes can be contiguous or non-contiguous. A contiguous degenerate node is a neighbor of one or more other degenerate nodes in a group along a dimension. Training system 100 can be configured to identify a node as degenerate when it only has degenerate nodes as neighbors, such that a contiguous path from any one degenerate node to another degenerate node can be drawn using only links from degenerate nodes. On the other hand, a replica network is considered to have non-contiguous degenerate nodes if there is any degenerate node that cannot be reached from any other degenerate node in the replica network using only links from degenerate nodes.

[0063] For example and with reference to replica network 400 of Figure 4 Nodes B3, C3, C4, and D3 are contiguous, as a path from any one degenerate node to another degenerate node can be drawn while only visiting other degenerate nodes. However, for the sake of example, if node Cl were also degenerate, the degenerate nodes would be considered non-contiguous, as a path to Cl cannot be drawn without visiting a non-degenerate node. As discussed in detail below, how training system 100 trains a replica network with one or more degenerate nodes can depend on whether the degenerate nodes are contiguous.

[0064] In general, training system 100 can train a replica network with one or more degenerate nodes. Training system 100 can classify degenerate nodes from non-degenerate nodes. Training system 100 can then perform a series of “forward phase” operations and “key phase” operations for each group of nodes in each dimension. As noted above, key phase operations include reduction operations. For example, each of the first, second, and third reductions discussed with respect to Figure 3A form part of a respective key phase along each dimension.

[0065] On the other hand, the forward phase operations include operations that, when executed by a node, cause the node to forward respective individual gradient vectors to non-degenerate nodes. The receiving nodes can be configured to receive and compute new individual gradient vectors reflecting the individual gradient vectors forwarded by the forwarding nodes, referred to in this specification as intermediate gradient vectors. As described below, the order and specific operations performed by the training system during the key phase and the forward phase vary from implementation to implementation.

[0066] To determine which groups of nodes should perform the "key phase" operations and which groups of nodes should perform the "forward phase" operations, the training system can be configured to designate each group of nodes along a dimension as either a "forward group" or a "key group." A "forward group" is a group of nodes that does not participate in the key phase of the reduction along a dimension.

[0067] In contrast, in some implementations, each non-degenerate node in a forward group forwards its respective individual gradient vector to a non-degenerate node in a key group, i.e., a group of nodes designated to participate in the key phase along a dimension. In some implementations and as described below, the training system can form "subnetworks" of non-degenerate nodes in a forward group and compute a final gradient vector (referred to as an "internal" gradient vector) for each subnetwork before forwarding the internal gradient vector to a non-degenerate node in a key group.

[0068] Because partially degenerate nodes are defined as degenerate or non-degenerate on a dimension-by-dimension basis, a partially degenerate node can be, for example, part of a forward group along one dimension but also part of a key group along another dimension.

[0069] In some implementations, to designate the group in which a node resides as either a "forward group" or a "key group," the system can identify a partially degenerate node as fully degenerate. This means that the system can identify a partially degenerate node as a fully degenerate node regardless of whether the node is degenerate along the respective dimension. Because a partially degenerate node can be susceptible to further degeneration, the system can be configured to identify the node as fully degenerate as a precaution against including the node to perform key operations. By routing around degenerate nodes, the training system can exploit different levels of degeneration to improve processing by the replica network. The training system can distribute performing operations in both the forward phase and the key phase to exploit the distributed nature of the replica network. Additionally, by distributing gradient vectors and subsequent processing across nodes in the replica network, the training system can mitigate further degeneration caused by "hot spots" on physical components of the replica network caused by uneven distribution of computational tasks. As described below, the training system can be configured to perform reductions on replica networks for arbitrary dimensions.

[0070] The training system 100 can classify a node as degraded by testing the quality of the components of each node and link in any appropriate manner. For example, each node can be configured to receive and respond to status requests sent by the training system 100. If the training system 100 does not receive a response to a status request within a predetermined time period, the training system 100 can classify the node as degraded.

[0071] As another example, the training system can send a status request along each link connecting respective adjacent nodes along a dimension. Similarly, if the training system does not receive a response to a status request sent to a node along a particular link, the training system 100 can classify the node as degraded along the dimension having that particular link. In this example, the training system can receive a response to a status request for a node for requests sent along some links but not others connecting the node. Thus, the training system 100 classifies the node as partially degraded along the dimension having the impaired or non-responsive link.

[0072] Next, the training system can designate each node group as a forwarding group or a critical node group. As described above, the designation of each group can vary depending on the implementation.

[0073] In the forwarding phase of some implementations, the training system designates each node group along a dimension and having a degraded node as a forwarding node group. In implementations where the forwarding node group does include a degraded node, the training system designates all other groups as critical node groups.

[0074] Each non-degraded node in the forwarding group computes a respective individual gradient vector for a batch of training data and forwards the respective individual gradient vector to a non-degraded node in the critical node group. The non-degraded node receiving the forwarded individual gradient vector then computes an intermediate gradient vector from the individual gradient vector of the receiving node and the forwarded individual gradient vector. The receiving node updates its individual gradient vector to the intermediate gradient vector. The training system can then compute a reduction along each critical node group for the dimension, as discussed above with reference to Figure 3A -C.

[0075] Because the non-degraded nodes in the forwarding group have already forwarded their respective individual gradient vectors, the reduction along the dimension is performed on groups having only non-degraded nodes. As discussed above with reference to Figure 3A -C, each critical group can have a representative node storing a partial final gradient vector for the critical group. The training system 100 can be configured to broadcast the respective partial final gradient vector for each critical group to each node in the group. The training system 100 can then perform subsequent reductions by repeating the operations of the forwarding and critical phases for the groups along subsequent dimensions.

[0076] After performing the forward and critical stage operations along each dimension, the training system can compute updated model parameter values from the final gradient vectors and broadcast the updated model parameter values to each non-degenerate node in the replica network, including the nodes for each forward group of each dimension.

[0077] Figure 5A - Figure C illustrates a training replica network 500a-c with a two-dimensional grid topology and with degenerate nodes. In some implementations, when the group includes one or more degenerate nodes, the training system trains the replica network 500a-c with degenerate nodes by designating each group of nodes along a dimension as a forward group.

[0078] Figure 5A Figure C illustrates an example replica network 500a with partially degenerate nodes. For example, in Figure 5A As a result of degenerate nodes B2, B3, C2, and C3, rows “B” and “C” are forward groups. Thus, non-degenerate nodes Bl, Cl, B4, and C4 can compute respective individual gradient vectors and forward the respective individual gradient vectors to receiving nodes in the critical group. In replica network 500a, rows “A” and “D” are considered the critical group. Cl forwards its respective individual gradient vector to Dl; Bl forwards its respective individual gradient vector to Al; B4 forwards its respective individual gradient vector to A4; and C4 forwards its respective individual gradient vector to D4. In some cases, a forwarding node will have to forward an individual gradient vector through more than one node before reaching a node in the critical group. In other cases, multiple nodes can forward their respective individual gradient vectors to the same receiving node. Finally, in some cases, a forwarding node can not be able to forward its individual gradient vector to any receiving node.

[0079] Once each forwarding node has forwarded its respective individual gradient vector to a receiving node, each receiving node can compute an intermediate gradient vector.

[0080] Note that in Figure 5A In some implementations, the training system can be configured to determine which dimension has the fewest degenerate nodes from the multiple dimensions. Then, the training system can perform a reduction along that dimension.

[0081] Figure 5B Figure B illustrates an example replica network 500b with degenerate nodes in a corner of the network. Figure 5BThe degenerated nodes Al, A2, Bl, and B2 are illustrated, and the training system is also illustrated computing a reduction along the row dimension of the replica network 500b, indicated by the horizontal arrow 510. During the forwarding phase, the nodes A3 and B3 forward their respective individual gradient vectors to node C3, while the nodes A4 and B4 forward their respective gradient vectors to node C4. As described above, the nodes A3, B3, A4, and B4 are forwarding nodes because these nodes are non-degenerated and belong to respective node groups that have degenerated nodes.

[0082] The nodes C3 and C4 of the replica network 500b can then compute respective intermediate gradient vectors. Specifically, node C3 computes an intermediate gradient vector between its own individual gradient vector and the forwarded gradient vectors of A3 and B3. Similarly, node C4 can compute an intermediate gradient vector from its own individual gradient vector and the forwarded gradient vectors of A4 and B4. The training system can then perform operations in the key phase for the key groups “C” and “D” and obtain respective partial final gradient vectors for each row.

[0083] The training system can also train a replica network with multiple degenerated nodes that are not contiguous.

[0084] Figure 5C An example replica network 500c with multiple non-contiguous degenerated nodes is illustrated. Figure 5C The degenerated nodes Al, B2, and D4 are illustrated, and the training system is also illustrated computing a reduction along the row dimension of the replica network 500c, indicated by the horizontal arrow 515. First, the training system 100 can determine whether the replica network 500c includes non-contiguous degenerated nodes.

[0085] If the replica network 500c does include non-contiguous degenerated nodes, the training system 100 can identify whether there are groups along the row dimension that do not have any degenerated nodes. In the replica network 500c, the row “C” is a group that does not have any degenerated nodes. In some implementations, the training system can determine along which dimension to reduce first based on the presence of a group with only non-degenerated nodes. For example, if the training system 100 does not identify a group of non-degenerated nodes along one dimension, the training system 100 can analyze groups along other dimensions of the replica network until a group of non-degenerated nodes is found.

[0086] During the forwarding phase, the nodes Bl and Dl of the replica network 500c forward their respective individual gradient vectors to node Cl; the nodes A3, B3, and D3 forward their respective individual gradient vectors to node C3; and the nodes A4 and B4 forward their respective individual gradient vectors to node C4. The nodes Cl, C3, and C4 then compute respective intermediate gradient vectors, as described above with reference to the replica network 500b. Figure 5AThe training system 100 can then perform a key phase on row "C" to generate a final gradient vector for the replica network 500c.

[0087] As described above, the training system 100 can be configured to compute an internal reduction of the subnetworks of non-degenerate nodes in the forwarding group. Each node of a subnetwork can forward a respective internal gradient vector to a respective receiving non-degenerate node in the key group. Each receiving node can use its own individual gradient vector and the received internal gradient vector of the subnetwork to generate an intermediate gradient vector. Each receiving node can then update its respective individual gradient vector with the computed intermediate gradient vector, and the training system 100 can perform a reduction along the dimension for each key node group.

[0088] Figure 6 An example replica network 600 with consecutive degenerate nodes is illustrated. Figure 6 Degenerate nodes B2, A3, and B3 are illustrated, and the training system is also illustrated performing a reduction along the column dimension of the replica network 500c, indicated by the vertical arrows 515. First, the training system can classify each group along the column dimension as having degenerate nodes or not having degenerate nodes. In the replica network 600a, the "2" column and the "3" column include degenerate nodes, so are designated as forwarding groups.

[0089] Next, the training system 100 can perform an internal reduction of the subnetworks of consecutive non-degenerate nodes in each forwarding group. To do so, the training system 100 can identify subnetworks of nodes. Each node subnetwork contains only non-degenerate nodes across one or more forwarding groups.

[0090] For example, in Figure 6 The training system 100 can form a subnetwork 610 including consecutive nodes C2, C3, D2, and D3. Each node in the subnetwork 610 is a non-degenerate node in a forwarding group, namely column "2" and column "3." Because the nodes C2, C3, D2, and D3 are consecutive, the training system 100 can form a subnetwork from all four nodes. In some implementations, the training system 100 is configured to generate subnetworks of different sizes. For example, for the four nodes C2, C3, D2, and D3, the training system can form four subnetworks of size 1, two subnetworks of size 2, or one subnetwork of size 1 and one subnetwork of size 3.

[0091] Also in Figure 6In particular, training system 100 can form a second subnetwork 615 from nodes A2. Training system forms the subnetwork 615 including only node A2 because A2 is not contiguous with any non-degenerate nodes in the forwarding group. Training system 100 can repeat this process of forming subnetworks until every non-degenerate node in each forwarding group is part of a respective subnetwork.

[0092] After forming the subnetworks, training system 100 can cause the nodes to compute an internal gradient vector for each subnetwork. The internal gradient vector is the final gradient vector for the subnetwork. Training system 100 can generate the internal gradient vector using any of the techniques described for the reduction replica network, as discussed above with reference to Figure 3A In replica network 600a, training system 100 can compute internal gradient vectors for subnetworks 610 and 615.

[0093] After computing the internal gradient vectors for each subnetwork, training system can cause the nodes of each subnetwork to forward the respective internal gradient vector to the non-degenerate nodes in the critical group. In replica network 600a, nodes of subnetwork 610 forward the internal gradient vector for subnetwork 610 to nodes CI, DI, C4, and D4. Similarly, node A2 forwards the internal gradient vector for subnetwork 615 to node Al.

[0094] After forwarding the internal gradient vectors, the nodes that receive the internal gradient vectors generate an intermediate gradient vector for the node from the respective individual gradient vector and the received internal gradient vector. Each receiving node then updates the respective individual gradient vector with the generated intermediate gradient vector. In replica network 600a, nodes Al, CI, DI, C4, and D4 update the respective individual gradient vectors with the computed intermediate gradient vectors.

[0095] Finally, training system can compute the partial final gradient vectors for each critical group along a dimension. In replica network 600a, training system 100 computes the reduction for column "1" and column "4." The respective partial final gradient vectors for the group can be used to update a single representative node, and training system 100 can repeat the reduction process along the next dimension.

[0096] Although Figure 5A While the example replica networks in -C and 6 have been replica networks defining a two-dimensional grid topology, training system 100 can also be configured to perform reduction on replica networks having a torus topology. When a replica network defines a torus topology, training system additionally considers wrap-around links between nodes on opposite ends of the replica network in determining contiguous degenerate or non-degenerate nodes.

[0097] Figure 7An example replica network 700 with torus topology and degenerate nodes is illustrated. Figure 7 Degenerate nodes Al, A2, Bl, and B2 are illustrated, and the training system is also illustrated computing a reduction along the column dimension of the replica network 700, shown by vertical arrows 705. The training system 100 can compute the final gradient vector by first designating columns “1” and “2” as a forwarding group. Then, the non-degenerate nodes of the forwarding group, namely nodes Cl, C2, Dl, and D2, can forward individual gradient vectors for the non-degenerate nodes in the group with degenerate nodes and compute intermediate gradient vectors at receiving nodes C3 and D3.

[0098] In some implementations, and as discussed above with reference to Figure 6 the training system can identify a subnetwork of nodes in the replica network 700 and first compute an internal gradient vector for the subnetwork. In replica network 700, subnetwork 710 can include four non-degenerate nodes Cl, C2, Dl, and D2. The training system can compute an internal gradient vector for subnetwork 710, and nodes C2 and D2 can forward the internal gradient vector to adjacent nodes C3 and D3.

[0099] Figure 8 FIG. 8 is a flowchart of an example process 800 for computing a final gradient vector for a replica network with degenerate nodes. For convenience, the example process will be described as being performed by a system of one or more computers, located in one or more locations, programmed appropriately according to the present specification. For example, a training system, such as the training system 100 of Figure 1 , appropriately programmed, can perform the example process of Figure 8 . One iteration of the example process 800 along a dimension is described. As described above, the training system can generate a partial final gradient vector for each dimension of the replica network until the final gradient vector is computed.

[0100] The training system designates each group of nodes along the dimension as either a forwarding group or a critical group based on whether the group includes any degenerate nodes (810). The training system can designate the group as either a forwarding group or a critical group depending on the presence or absence of degenerate nodes in the group. As discussed above with reference to Figure 6 In some implementations, the training system designates one or more subnetworks of non-degenerate nodes across one or more forwarding groups.

[0101] The training system forwards, for each non-degenerate node in a forwarding group of nodes along the dimension, an individual gradient vector for the nodes along the dimension until the respective individual gradient vector is forwarded to a respective receiving node in a respective critical group of nodes along the dimension (820). As discussed above with reference to Figure 5A-C discussed, non-degenerate nodes in the forwarding group can forward their respective individual gradient vectors to non-degenerate nodes that are not in the forwarding group.

[0102] The training system updates the respective individual gradient vector for the representative node with the respective partial final gradient vector for each key node group (850). As discussed above with respect to Figure 5A -C and 6 discussed, in some implementations, the training system can compute an internal reduction for the subnetwork of the non-degenerate node before the non-degenerate node forwards the respective internal gradient vector for the subnetwork to the receiving nodes.

[0103] The training system performs a reduction along the dimensions for each key node group to generate a respective partial final gradient vector (840). If the training system is performing a reduction for a replica network having only one dimension, the partial final gradient vector is the final gradient vector for the replica network.

[0104] The training system updates the individual gradient vector for the representative node with the respective partial final gradient vector for each key node group (850). As discussed above with respect to Figure 3A -C discussed, the subnetwork representing all nodes having the respective partial final gradient vector has one less dimension than the replica network in which the partial final gradient vector was computed. For example, if the replica network defines a three-dimensional grid topology, the partial final gradient vector is represented by nodes defining a two-dimensional grid topology.

[0105] The training system repeats the operation for the next dimension on the subnetwork including only the representative nodes until a final gradient vector is generated (860). Using the final gradient vector, the training system can update the model parameter values of the machine learning model with the final gradient vector before broadcasting the updated model parameter values to each non-degenerate node.

[0106] Figure 9 is a flowchart of an example process for performing an internal reduction on a subnetwork of non-degenerate nodes. For convenience, the example process will be described as performed by a system of one or more computers located in one or more locations and programmed appropriately in accordance with this specification. For example, a training system— e.g., training system 100 of FIG. 1— appropriately programmed can perform the example process of FIG. 8. Figure 1 The training system 100 of FIG. 1— appropriately programmed can perform the example process of FIG. 8. Figure 9 The training system 100 of FIG. 1— appropriately programmed can perform the example process of FIG. 8.

[0107] The training system determines that the group of nodes includes one or more degenerate nodes and, in response, designates the group of nodes as part of one or more node subnetworks, where the one or more node subnetworks do not include the one or more degenerate nodes (step 910). As discussed above with reference to Figure 6 The subnetworks include consecutive non-degenerate nodes of the one or more forwarding groups.

[0108] The training system, for each node subnetwork along the dimension, performs a reduction on the subnetwork of nodes along the dimension in the replica network to generate a respective internal gradient vector (step 920). As discussed above with reference to Figure 6 The internal gradient vector is the final gradient vector for the node subnetwork.

[0109] The training system causes the nodes in each subnetwork to forward the respective internal gradient vector for the node subnetwork until the respective internal gradient vector is forwarded to a node in the critical group of nodes along the dimension, where the node is not in the node subnetwork (step 930). The receiving nodes can update their respective gradient vectors with the respective intermediate gradient vectors.

[0110] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. Although this specification is specifically directed to processing units that process gradient vectors, the processing units can perform reduction processes on any non-scalar instance data, i.e., tensors, matrices, arrays, or any other data structure that can hold data that can be split and processed in parallel. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0111] Embodiments of the subject matter and the actions and operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more modules of computer program instructions, for execution on a data processing apparatus. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and

[0112] The term“data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The data processing apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit) or GPU (graphics processing unit). The apparatus can also include, in addition to hardware, code that creates an execution environment for computer programs in the form of computer-executable instructions.

[0113] A computer program can also be referred to or described as a program, software, a software application, an app, a module, a software module, an engine, a script, or code. A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment, which can include one or more computers under the control of one or more operating systems.

[0114] The computer program can be, but need not be, stored in a file system of a computer. The program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code.

[0115] The processes and logic flows described in this specification can be performed by one or more computers executing one or more computer programs to perform operations by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA, an ASIC, or a GPU, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0116] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0117] Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data. Mass storage devices can be, for example, magnetic, magneto-optical disks, optical disks or solid-state drives. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0118] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a LCD (liquid crystal display) monitor, for displaying information to the user and a input device, e.g., a keyboard and a pointing device, e.g., a mouse, a trackball, and a touchpad, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser, or by interacting with an app running on a user device, e.g., a smart phone or an electronic tablet, in response to requests received from the app. Further, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smart phone that is running a messaging application, and receiving responsive messages from the user in return.

[0119] The term "configured to" is used in this specification to mean that the system, device, or computer program in question has been designed, programmed, or constructed to perform the function in question. For one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. For a particular logic circuit to be configured to perform particular operations or actions means that the circuit has components that perform the operations or actions.

[0120] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0121] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a client device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the client device, e.g., a result of the user interaction, can be received from the device at the server.

[0122] In addition to the embodiments of the appended claims and the embodiments described above, the following numbered embodiments are also inventive:

[0123] Example 1 is a method for processing training data in parallel, the method comprising: training a respective copy of a machine learning model on each node of a plurality of nodes organized in an n-dimensional network topology, where n is an integer greater than or equal to 1, where a plurality of nodes of the plurality of nodes are trained in parallel on a respective batch of training data, where one or more nodes of the plurality of nodes have been classified as degenerate, and where each non-degenerate node stores a respective individual gradient vector resulting from training the respective copy of the node on the respective batch of training data; and combining the respective individual gradient vectors of the nodes to generate a final gradient vector by performing operations on a dimension of the n dimensions in the network topology comprising: designating groups of nodes along the dimension as either a forwarding group or a critical group based on whether each group of nodes along the dimension includes any degenerate nodes, for each non-degenerate node in a forwarding group of nodes along the dimension, forwarding the respective individual gradient vector for the nodes along the dimension until the respective individual gradient vector is forwarded to a respective receiving node in a respective critical group of nodes along the dimension, for each receiving node, updating the respective individual gradient vector with an intermediate gradient vector computed from the respective individual gradient vector and one or more received individual gradient vectors, performing a reduction on each critical group of nodes along the dimension to generate a respective partial final gradient vector for the critical group, and for each critical group of nodes, updating an individual gradient vector for a representative node with the respective partial final gradient vector.

[0124] Example 2 is the method of Example 1, wherein the dimension is a first dimension, wherein combining the respective individual gradient vectors of the nodes to generate the final gradient vector comprises performing the operations on each dimension of the n-dimensional network topology including the first dimension, and wherein the method further comprises repeating the operations on a next dimension on a sub-network including only the representative nodes until the final gradient vector is generated.

[0125] Example 3 is the method of any of Examples 1-2, further comprising: updating model parameter values of the machine learning model with the final gradient vector; and broadcasting the updated model parameter values to each non-degenerate node.

[0126] Example 4 is the method of any of Examples 1-3, wherein designating each group of nodes in a plurality of groups of nodes in a dimension as either a forwarding group of nodes or a critical group of nodes based on the presence of one or more degenerate nodes in the group of nodes comprises: determining that the group of nodes includes one or more degenerate nodes, and in response, designating the group of nodes as a forwarding group of nodes.

[0127] Example 5 is the method of any of Examples 1-4, wherein designating each node group of the plurality of node groups in each dimension as a forwarding node group or a key node group based on the presence of one or more degenerate nodes in the node group comprises determining that the node group includes one or more degenerate nodes and, in response, designating the node group as part of one or more node subnetworks that do not include the one or more degenerate nodes.

[0128] Example 6 is the method of any of Examples 1-5, wherein the reduction is a first reduction, and the method further comprises, for each node subnetwork: for each node subnetwork along the dimension, performing a second reduction on the node subnetwork to generate a respective internal gradient vector; and forwarding the respective internal gradient vector for each node subnetwork until the internal gradient vector is forwarded to a node in a key node group along the dimension.

[0129] Example 7 is the method of any of Examples 1-6, wherein the network topology is a mesh topology.

[0130] Example 8 is the method of any of Examples 1-6, wherein the network topology is a torus topology.

[0131] Example 9 is the method of any of Examples 1-8, further comprising determining that a particular node is not degenerate and that each neighbor of the particular node is degenerate along a particular dimension, and in response, instructing the particular node to degenerate along the particular dimension.

[0132] Example 10 is the method of any of Examples 1-9, wherein performing a reduction on each key node group along the dimension to generate a respective partial final gradient vector comprises performing a cyclic reduction on each key node group.

[0133] Example 11 is a system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform the method of any of Examples 1-10.

[0134] Example 12 is a computer storage medium encoded with a computer program, the program comprising instructions operable to cause data processing apparatus to perform the method of any of Examples 1-10 when executed by the data processing apparatus.

[0135] Although this description contains many specifics, these should not be construed as limiting the scope of any invention or potentially patentable claim in which this description might serve as a basis. No portion of this description should be taken as a specific or exhaustive description of any single invention or potentially patentable claim. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claim can be directed to a subcombination or variation of a subcombination.

[0136] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring such order nor

[0137] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve the desired result. As one example, the diagrams depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some instances, multitasking and parallel processing can be advantageous.

Claims

1. A method for parallel processing of training data, the method comprising: A corresponding copy of a machine learning model is trained on each of a plurality of nodes organized in an n-dimensional network topology, each node having a corresponding processing unit configured to train the corresponding copy of the machine learning model in parallel on a corresponding batch of training data, where n is an integer greater than or equal to 1. Among these, one or more of the plurality of nodes have been classified as degenerate, and Each non-degenerate node stores a corresponding gradient vector obtained by training a corresponding copy of the non-degenerate node on the corresponding batch of training data; and The final gradient vector is generated by combining the respective gradient vectors from the plurality of nodes by performing operations on the dimensions of the n dimensions in the network topology, including the following: Based on whether each node group along the said dimension includes any degenerate nodes, the node group is designated as a forwarding node group or a critical node group. For each non-degenerate node in the forwarding node group along the said dimension, the corresponding gradient vector for that non-degenerate node along the said dimension is forwarded until the corresponding gradient vector is forwarded to the corresponding receiving node in the corresponding critical node group along the said dimension. For each receiving node, the corresponding gradient vector is updated using an intermediate gradient vector, wherein the intermediate gradient vector is calculated from the corresponding gradient vector and one or more received gradient vectors. Perform a reduction for each key node group along the said dimension to generate a corresponding partial final gradient vector for the key node group, and For each key node group, the corresponding gradient vector for the representative node is updated using the corresponding partial final gradient vector.

2. The method according to claim 1, in, The dimension mentioned is the first dimension. The process of combining the respective gradient vectors from the plurality of nodes to generate the final gradient vector includes performing an operation on each dimension of the n-dimensional network topology, including the first dimension, and... The method further includes repeating the operation for the next dimension on a subnetwork that includes only the representative node, until the final gradient vector is generated.

3. The method according to claim 2, further comprising: The final gradient vector is used to update the model parameter values ​​of the machine learning model; as well as The updated model parameter values ​​are broadcast to each non-degenerate node.

4. The method according to claim 1, wherein, Designating a node group as a forwarding node group or a critical node group based on whether each node group along the said dimension includes any degenerate nodes includes: The node group is determined to include one or more degraded nodes, and in response, the node group is designated as a forwarding node group.

5. The method according to claim 1, wherein, Designating a node group as a forwarding node group or a critical node group based on whether each node group along the said dimension includes any degenerate nodes includes: The node group is determined to include one or more degenerate nodes, and in response, the node group is designated as part of one or more node subnetworks, wherein the one or more node subnetworks do not include the one or more degenerate nodes.

6. The method according to claim 5, wherein, The reduction is a first reduction, and the method further includes, for each node subnetwork: For each node subnetwork along the said dimension, a second reduction is performed on the node subnetwork to generate the corresponding internal gradient vector; as well as The corresponding internal gradient vector for each node subnetwork is forwarded until the internal gradient vector is forwarded to a node in the key node group along the dimension.

7. The method according to claim 1, wherein, The network topology is a mesh topology.

8. The method according to claim 1, wherein, The network topology is a toroidal topology.

9. The method according to claim 1, wherein, Performing a reduction for each key node group along the said dimension to generate the corresponding partial final gradient vector includes performing a cyclic reduction for each key node group.

10. The method according to any one of claims 1 to 9, further comprising: Determine that a specific node is not degenerate, and that each of the specific node's neighboring nodes degenerates along a specific dimension; as well as In response, the specific node is instructed to degenerate along the specific dimension.

11. A system for parallel processing of training data, comprising: One or more computers and one or more storage devices, wherein instructions are stored on the one or more storage devices, the instructions, when executed by the one or more computers, being operable to cause the one or more computers to perform a method for parallel processing of training data, the method comprising: A corresponding copy of a machine learning model is trained on each of a plurality of nodes organized in an n-dimensional network topology, each node having a corresponding processing unit configured to train the corresponding copy of the machine learning model in parallel on a corresponding batch of training data, where n is an integer greater than or equal to 1. Among these, one or more of the plurality of nodes have been classified as degenerate, and Each non-degenerate node stores a corresponding gradient vector obtained by training a corresponding copy of the node on the corresponding batch of training data; and The final gradient vector is generated by combining the respective gradient vectors from the plurality of nodes by performing operations on the dimensions of the n dimensions in the network topology, including the following: Based on whether each node group along the said dimension includes any degenerate nodes, the node group is designated as a forwarding node group or a critical node group. For each non-degenerate node in the forwarding node group along the said dimension, the corresponding gradient vector for that non-degenerate node along the said dimension is forwarded until the corresponding gradient vector is forwarded to the corresponding receiving node in the corresponding critical node group along the said dimension. For each receiving node, the corresponding gradient vector is updated using an intermediate gradient vector, wherein the intermediate gradient vector is calculated from the corresponding gradient vector and one or more received gradient vectors. Perform a reduction for each key node group along the said dimension to generate a corresponding partial final gradient vector for the key node group, and For each key node group, the corresponding gradient vector for the representative node is updated using the corresponding partial final gradient vector.

12. The system according to claim 11, in, The dimension mentioned is the first dimension. The process of combining the respective gradient vectors from the plurality of nodes to generate the final gradient vector includes performing an operation on each dimension of the n-dimensional network topology, including the first dimension, and... The method further includes repeating the operation for the next dimension on a subnetwork that includes only the representative node, until the final gradient vector is generated.

13. The system according to claim 12, wherein, The method further includes: The model parameter values ​​of the machine learning model are updated using the final gradient vector; and The updated model parameter values ​​are broadcast to each non-degenerate node.

14. The system according to claim 11, wherein, Designating a node group as a forwarding node group or a critical node group based on whether each node group along the said dimension includes any degenerate nodes includes: The node group is determined to include one or more degraded nodes, and in response, the node group is designated as a forwarding node group.

15. The system according to claim 11, wherein, Designating a node group as a forwarding node group or a critical node group based on whether each node group along the said dimension includes any degenerate nodes includes: The node group is determined to include one or more degenerate nodes, and in response, the node group is designated as part of one or more node subnetworks, wherein the one or more node subnetworks do not include the one or more degenerate nodes.

16. The system according to claim 15, wherein, The reduction is a first reduction, and the method further includes, for each node subnetwork: For each node subnetwork along the said dimension, a second reduction is performed on the node subnetwork to generate the corresponding internal gradient vector; as well as The corresponding internal gradient vector for each node subnetwork is forwarded until the internal gradient vector is forwarded to a node in the key node group along the dimension.

17. The system according to claim 11, wherein, The network topology is a mesh topology.

18. The system according to claim 11, wherein, The network topology is a toroidal topology.

19. The system according to claim 11, wherein, Performing a reduction for each key node group along the said dimension to generate the corresponding partial final gradient vector includes performing a cyclic reduction for each key node group.

20. The system according to any one of claims 11 to 19, wherein, The method further includes: Determine that a specific node is not degenerate, and that each of the specific node's neighboring nodes degenerates along a specific dimension; and In response, the specific node is instructed to degenerate along the specific dimension.

21. One or more non-transitory computer-readable storage media, said one or more non-transitory computer-readable storage media being encoded with instructions that, when executed by said one or more computers, cause said one or more computers to perform a method for parallel processing of training data, said method comprising: A corresponding copy of a machine learning model is trained on each of a plurality of nodes organized in an n-dimensional network topology, each node having a corresponding processing unit configured to train the corresponding copy of the machine learning model in parallel on a corresponding batch of training data, where n is an integer greater than or equal to 1. Among these, one or more of the plurality of nodes have been classified as degenerate, and Each non-degenerate node stores a corresponding gradient vector obtained by training a corresponding copy of the non-degenerate node on the corresponding batch of training data; and The final gradient vector is generated by combining the respective gradient vectors from the plurality of nodes by performing operations on the dimensions of the n dimensions in the network topology, including the following: Based on whether each node group along the said dimension includes any degenerate nodes, the node group is designated as a forwarding node group or a critical node group. For each non-degenerate node in the forwarding node group along the said dimension, the corresponding gradient vector for that non-degenerate node along the said dimension is forwarded until the corresponding gradient vector is forwarded to the corresponding receiving node in the corresponding critical node group along the said dimension. For each receiving node, the corresponding gradient vector is updated using an intermediate gradient vector, wherein the intermediate gradient vector is calculated from the corresponding gradient vector and one or more received gradient vectors. Perform a reduction for each key node group along the said dimension to generate a corresponding partial final gradient vector for the key node group, and For each key node group, the corresponding gradient vector for the representative node is updated using the corresponding partial final gradient vector.

22. The computer-readable storage medium according to claim 21, in, The dimension mentioned is the first dimension. The process of combining the respective gradient vectors from the plurality of nodes to generate the final gradient vector includes performing an operation on each dimension of the n-dimensional network topology, including the first dimension, and... The method further includes repeating the operation for the next dimension on a subnetwork that includes only the representative node, until the final gradient vector is generated.

23. The computer-readable storage medium according to claim 22, wherein, The method further includes: The model parameter values ​​of the machine learning model are updated using the final gradient vector; and The updated model parameter values ​​are broadcast to each non-degenerate node.

24. The computer-readable storage medium of claim 21, wherein, Designating a node group as a forwarding node group or a critical node group based on whether each node group along the said dimension includes any degenerate nodes includes: The node group is determined to include one or more degraded nodes, and in response, the node group is designated as a forwarding node group.

25. The computer-readable storage medium according to claim 21, wherein, Designating a node group as a forwarding node group or a critical node group based on whether each node group along the said dimension includes any degenerate nodes includes: The node group is determined to include one or more degenerate nodes, and in response, the node group is designated as part of one or more node subnetworks, wherein the one or more node subnetworks do not include the one or more degenerate nodes.

26. The computer-readable storage medium according to claim 25, wherein, The reduction is a first reduction, and the method further includes, for each node subnetwork: For each node subnetwork along the said dimension, a second reduction is performed on the node subnetwork to generate the corresponding internal gradient vector; as well as The corresponding internal gradient vector for each node subnetwork is forwarded until the internal gradient vector is forwarded to a node in the key node group along the dimension.

27. The computer-readable storage medium of claim 21, wherein, The network topology is a mesh topology.

28. The computer-readable storage medium according to claim 21, wherein, The network topology is a toroidal topology.

29. The computer-readable storage medium according to claim 21, wherein, Performing a reduction for each key node group along the said dimension to generate the corresponding partial final gradient vector includes performing a cyclic reduction for each key node group.

30. The computer-readable storage medium according to any one of claims 21 to 29, wherein, The method further includes: Determine that a specific node is not degenerate, and that each of the specific node's neighboring nodes degenerates along a specific dimension; and In response, the specific node is instructed to degenerate along the specific dimension.

Images