Optimized asynchronous training of neural networks using a distributed parameter server with fast updates

Abstract

A method for training a neural network, comprising: receiving remote parameters from a set of one or more remote computing nodes at a local computing node; starting a forward pass in a local neural network in the local computing node to determine a final output based on the remote parameters; starting a backward pass in the local neural network to determine updated parameters for the local neural network; and before completing the backward pass, transmitting a subset of the updated parameters to the set of remote computing nodes.

Description

[0001] Government Rights

[0002] This invention was made with government support under the PathForward program at the Lawrence Livermore National Security Agency (prime contract number DE-AC52-07NA27344 awarded by DOE, subcontract number B620717). The government has certain rights in this invention. Background Art

[0003] In recent years, learning architectures such as deep neural networks have enabled advances in computer vision, speech recognition, natural language processing, translation, and many other fields, allowing computers in many cases to operate comparable to, or in some cases even exceed, the ability of humans to perform these tasks. The basic building block of a neural network is a neuron, and a single neural network can include any number (e.g., thousands or millions) of neurons connected in layers, where the neurons in each layer receive their inputs from the previous layer and provide their outputs to the subsequent layer. Each neuron generates an output based on performing a function (e.g., summing) on ​​its inputs, with adjustable weights applied to the inputs. The bias value added to the output can also be adjusted.

[0004] Backpropagation is a technique for performing machine learning in neural networks in which the error contribution of each neuron is calculated based on comparing the final output of the neural network to the expected output for a specific input. The weights and biases of the individual neurons that contributed to the error are then adjusted so that the neural network achieves the desired output. The error at the output is calculated, and the weight adjustments are propagated backwards from the final layer of the neural network to the initial layers. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present disclosure is illustrated by way of example and not by way of limitation in the figures of the accompanying drawings.

[0006] Figure 1 An embodiment of a cluster of computing nodes including a centralized parameter server for training a neural network is shown.

[0007] Figure 2 A computing system including a cluster of computing nodes for training a neural network is shown according to an embodiment.

[0008] Figure 3 A cluster of computing nodes for training a neural network is shown according to an embodiment.

[0009] Figure 4 Components in a compute node including a parameter server are shown according to an embodiment.

[0010] Figure 5A distributed parameter server in a cluster of compute nodes according to an embodiment is shown.

[0011] Figure 6 is a flow chart illustrating a process for training a neural network according to an embodiment. DETAILED DESCRIPTION

[0012] The following description sets forth many specific details, such as examples of specific systems, components, methods, etc., in order to provide a good understanding of the embodiments. However, it will be appreciated by those skilled in the art that at least some embodiments may be practiced without these specific details. In other cases, well-known components or methods are not described in detail, or are presented in a simple block diagram format to avoid unnecessary confusion with the embodiments. Therefore, the specific details set forth are merely exemplary. Specific implementations may differ from these exemplary details and are still considered to be within the scope of the embodiments.

[0013] Backpropagation techniques can be used when performing multiple instance learning in a neural network, where parameters in the neural network are adjusted based on the operation of the neural network on multiple instances in a training set. Each instance represents a set of inputs and expected outputs, which are compared to the actual outputs generated by the neural network. A training framework for smaller neural networks that operates according to these methods can be implemented within the limited resources of a single computing node. However, scaling the training framework to accommodate much larger neural networks may involve distributing the computations across multiple computing nodes (e.g., multiple graphics processing unit (GPU) nodes), each computing node performing a backpropagation process on a corresponding portion of the neural network (i.e., a subset of neurons) and / or using a corresponding portion of the training set.

[0014] A method for scaling a neural network training framework across multiple computing nodes uses a centralized parameter server to implement an asynchronous stochastic gradient descent (SGD) procedure. Figure 1 A block diagram of an embodiment of a cluster of computing nodes 100 is shown, wherein such a centralized parameter server 101 is used to train a neural network including layer parameters L1-LN, where LX represents the local parameters of layer X of the neural network. During the training process, the centralized parameter server 101 is responsible for updating and calculating the layer parameters (e.g., weights and biases) in these layers L1-LN, and communicating the parameters to the computing nodes 102-104.

[0015] Training a neural network according to the back-propagation process involves a forward pass and a backward pass. During the forward pass, the neural network is used to calculate the final output (i.e., the output from the final layer) for one of the training examples. During the backward pass, the error between the output calculated by the forward pass and the desired output is determined, and the error is propagated backward through each layer of the neural network to calculate the adjusted weights for the neurons in the network. Figure 1 , backward pass 112 is executed in compute node 102, while forward passes 113 and 114 are executed in compute nodes 103 and 104, respectively.

[0016] The complete backward pass generates a batch of updated parameters for all layers in the neural network, including updated weights and / or bias values ​​for some or all neurons in the neural network. At the end of the backward pass 112, the batch of updated parameters calculated from the backward pass 112 is transmitted to the centralized parameter server 101 using point-to-point communication. The centralized parameter server 101 updates the parameters in the computing nodes 103-104 by transmitting the updated parameters to other computing nodes 103-104. The computing nodes 102-104 operate in parallel and may perform different iterations of the training loop at any given time. However, in this implementation, the parameters are transmitted to and from the parameter server 101 in conjunction with the completion of the backward pass.

[0017] The centralized parameter server 101 facilitates communication of updated parameters between multiple computing nodes 102-104, thereby allowing the neural network training framework to be expanded among computing nodes 102-104 rather than being limited to a single node. However, further scalability may be limited based on the bandwidth and computational limitations of the centralized parameter server 101. The computing nodes 102-104 update their parameters before starting to compute the forward pass for computing the subsequent batch of parameters, and write the new parameters to the centralized parameter server 101 at the end of the backward pass. Therefore, the communication of updated parameters between the computing nodes 102-104 and the centralized parameter server 101 is completed at batch granularity (i.e., the backward pass is completed for all layers before the updated parameters are transmitted) and is in the critical path of the training process. The communication between the nodes 102-104 and the parameter server 101 is point-to-point, and therefore each iteration includes two critical path communications, which may have a negative impact on performance. In addition, the centralized nature of using the parameter server 101 tends to increase network contention. In addition, since the computing nodes in which the parameter server is implemented are not used to train the neural network, implementing the centralized parameter server 101 in the computing nodes may result in inefficient use of computing resources.

[0018] Furthermore, in a training framework using a centralized parameter server 101, computing nodes 102-104 may sometimes operate using stale parameter data. For example, computing node 102 may have already computed updated parameters for layer LN-L2. However, node 103 does not receive updated parameters for LN-L2 until after backward pass 112 is completed. Meanwhile, forward pass 113 continues to use the previously stale values ​​of LN-L2 to compute its output. This update delay may result in training discrepancies between multiple computing nodes.

[0019] In one embodiment of the neural network training framework, each computing node implements its own parameter server that provides asynchronous parameter updates to other computing nodes using one-sided (rather than point-to-point) collective communication semantics, through which the updated parameters are broadcast to all other computing nodes in the cluster. In one embodiment, the local parameter server of the computing node transmits the updated parameters to other nodes asynchronously in response to the calculation of the updated parameters for a layer or a subset of layers completed by backward passing in the computing node. Therefore, the communication of updated parameters for a particular layer and the calculation of updated parameters for one or more subsequent layers can be performed simultaneously. This method improves the scalability of the training framework because the update communication between nodes is not limited by the computational and communication throughput of a single centralized parameter server, and also reduces the differences caused by the use of stale data.

[0020] Figure 2 An embodiment of a computing system 200 is shown, which implements a parameter server distributed among multiple computing nodes for a neural network training framework, as described above. Typically, the computing system 200 can be embodied as any of many different types of devices, including but not limited to laptop or desktop computers, mobile phones, servers, etc. The computing system 200 includes many components 202-208 that communicate with each other through a bus 201. In the computing system 200, each of the components 202-208 can communicate with any other component 202-208 directly through the bus 201 or via one or more of the other components 202-208. The components 201-208 in the computing system 200 are contained in a single physical housing, such as a laptop or desktop computer rack or a mobile phone housing. In an alternative embodiment, some components of the computing system 200 are embodied as peripheral devices, so that the entire computing system 200 does not reside in a single physical housing.

[0021] The computing system 200 also includes a user interface device for receiving information from a user or providing information to a user. Specifically, the computing system 200 includes an input device 202, such as a keyboard, a mouse, a touch screen, or other device for receiving information from a user. The computing system 200 displays information to a user via a display 205, such as a monitor, a light emitting diode (LED) display, a liquid crystal display, or other output device.

[0022] The computing system 200 additionally includes a network adapter 207 for transmitting and receiving data over a wired or wireless network. The computing system 200 also includes one or more peripheral devices 208. The peripheral devices 208 may include mass storage devices, position detection devices, sensors, input devices, or other types of devices used by the computing system 200.

[0023] The computing system 200 includes a processor 204 configured to receive and execute instructions 206a stored in a memory subsystem 206. The memory subsystem 206 includes memory devices such as random access memory (RAM) modules, read-only memory (ROM) modules, hard disks, and other non-transitory computer-readable media used by the computing system 200. In one embodiment, the memory subsystem 206 includes logic to implement one or more memory retention controllers, each memory retention controller corresponding to a memory region in the memory subsystem 206.

[0024] Some embodiments of computing system 200 may include Figure 2 The embodiments shown have fewer or more components. For example, some embodiments are implemented without any display 205 or input device 202. Other embodiments have more than one of a particular component; for example, an embodiment of computing system 200 may have multiple processors 204, network adapters 207, etc.

[0025] The computing system 200 also includes a computing node cluster 209, which includes a collection of computing nodes in which a neural network training framework is implemented. In one embodiment, the computations performed in the computing node cluster 209 are directed by a processor 204 operating according to a program defined by instructions 206a.

[0026] Figure 3 An embodiment of a computing node cluster 209 is shown. The computing node cluster 209 includes a plurality of computing nodes 301-304 connected to each other by a communication network 310. The communication network 310 allows each of the computing nodes 301-304 to communicate with any of the other computing nodes 301-304. In one embodiment, the communication network 310 is an internal bus within a single computer system rack or housing, and each of the computing nodes 301-304 is connected to the internal bus. Alternatively, in an embodiment where the computing nodes 301-304 are not located in the same rack or housing, the communication network 310-304 is implemented as a larger network such as a local area network (LAN) or a wide area network (WAN). The communication network can be implemented using any combination of various communication protocols and topologies that allow communication between the computing nodes 301-304 as described herein.

[0027] In one embodiment, computing node cluster 209 may include any number of computing nodes 301-304. Each of computing nodes 301-304 in cluster 209 maintains its own copy of the neural network and determines update parameters for different subsets of neurons in the neural network. In one embodiment, a large set of training instances is also partitioned among computing nodes 301-304, such that each of computing nodes 301-304 uses a different subset of training instances to train its local copy of the neural network or a subset of neurons in its local copy.

[0028] Figure 4 The embodiment of computing node 301 in computing node cluster 209 is shown. In one embodiment, computing node 301 includes processing unit 402, such as CPU or GPU installed on carrier board or mainboard, which is installed in single computer chassis or rack with similar computing nodes (e.g., nodes 302-304). In alternative embodiments, computing node 301 is contained in its own chassis or rack and is connected to similar computing nodes (e.g., nodes 302-304) via LAN or WAN. In one embodiment, other components 401 and 403-405 in processing unit 402 and computing node 301 communicate with each other via bus 406.

[0029] The computing node 301 also includes a memory 403 for storing data. The memory 403 may include volatile and / or non-volatile memory implemented using any of a variety of memory technologies, such as dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, etc. In one embodiment, data (e.g., variables and data structures) used to define the neural network 404 is stored in the memory 403. Such data in the memory 403 defines the neurons of the neural network 404, the connections between the neurons, the weights and biases of the neurons, etc.

[0030] The computing node 301 also includes a communication network interface 401 that allows the computing node 301 to transmit and receive messages via the communication network 310. In particular, the communication network interface 401 receives remote parameters from one or more of the remote computing nodes 302-304 to update the local neural network 404. In one embodiment, remote direct memory access (RDMA) or a protocol with similar semantics is used to enable the remote nodes 302-304 to write the updated parameters directly to the local neural network model 404 in the memory 403 of the computing node 301. After receiving the updated parameters from the remote computing nodes, the communication network interface 401 writes the received parameters directly to the neural network 404 in the memory 403 without involving the operating system of the computing node 301 or its processor 402. The transmission of the updated parameters from the computing node 301 to other computing nodes 302-304 is also completed without involving the operating system and processor 402 via remote direct memory access or similar mechanisms.

[0031] During the neural network training process, parameter server 405 determines whether updated parameters of neural network 404 should be transmitted to remote computing nodes (e.g., 302-304), and transmits the updated parameters as appropriate via communication network interface 401. Parameter server 405 is replicated in each of computing nodes 301-304 and functions similarly in each instance. In one embodiment, parameter server 405 is implemented as a hardware circuit module. In alternative embodiments, parameter server 405 is implemented in firmware, software (e.g., defined by instructions in memory 403 executed by processing unit 402), or a combination of hardware, software, and / or firmware. In one embodiment, parameter server 405 is implemented as a kernel running on each of computing nodes 301-304, which is tuned to minimize resource utilization.

[0032] Figure 5is a block diagram showing the unilateral collective asynchronous transmission of updated parameters of a neural network from one computing node in a cluster to other computing nodes according to an embodiment. Computing nodes 301-304 train the neural network using backpropagation techniques, including forward and backward pass calculations. In computing node 301, processing unit 402 performs a forward pass in neural network 404 to calculate the final output of the neural network based on the input set provided by the training instance and based on previously established parameters (e.g., weights and biases) for neurons in neural network 404. The previously established parameters can be an initial parameter set, or remote parameters received from other computing nodes. Processing unit 402 uses the initial or remote parameters in the first layer of neurons in the neural network to calculate the intermediate output of the first layer based on the input set provided by the training instance. For each subsequent neuron layer in the local neural network, processing unit 402 calculates the intermediate output of the layer based on the intermediate output of at least one previous layer in the local neural network. The output of each layer includes one or more output signals. The output of the final layer is the final output of the neural network.

[0033] In one embodiment, computing node 301 asynchronously updates parameters in neural network 404 in response to receiving updated remote parameters from another computing node in cluster 209. Thus, the parameters are updated during execution of the forward pass. In other words, processing unit 402 begins execution of the forward pass before receiving the remote parameters, and completes execution of the forward pass to compute outputs based on the newly updated remote parameters after updating the neural network with the remote parameters.

[0034] At the communication network interface 401 in the computing node 301, the remote parameters are received from the other computing nodes 302-304 in the cluster 209. The communication network interface 401 asynchronously updates the neural network with the remote parameters in response to receiving the remote parameters. That is, the timing of the parameter update is independent of whether the receiving node has completed the forward pass or whether one or more transmitting nodes have completed their respective backward passes. In one embodiment, receiving the remote parameters is sufficient to cause the update of the neural network to occur.

[0035] In one embodiment, each of the other computing nodes 302-304 in the cluster updates parameters in their respective neural network instances in a similar manner. Thus, in each computing node 302-304, the communication network interface of the computing node updates the neural network with the received remote parameters during execution of the forward pass (i.e., after the forward pass begins and before the forward pass completes) so as to compute the output of the forward pass using the latest parameters.

[0036] In one embodiment, the communication network interface 401 of the computing node 301 supports a direct memory write mechanism that allows a remote device to write directly to the local memory 403 via the communication network interface 401 without the involvement of the processing unit 402 and the operating system of the computing node 301. Therefore, the communication network interface 401 responds to receiving updated parameters from the remote computing node by asynchronously updating its local copy of the neural network 404 with the remote parameters. This requires performing a direct memory write to store the remote parameters at the location in the local memory 403 where the neural network 404 parameters are stored.

[0037] After computing the forward pass in computing node 301, processing unit 402 performs a backward pass in local neural network 404 to determine updated parameters 501 for the neural network. Processing unit 402 performs the backward pass by computing a set of updated parameters for each layer in local neural network 404 based on the error between the final output determined from the forward pass and the expected output of the training instance used to generate the final output via the forward pass. During the execution of the backward pass, computing node 301 directly communicates the updated parameters 501 to other computing nodes 302-304 in a fine-grained manner, rather than transmitting the updated parameters via any intermediate device (e.g., a centralized parameter server).

[0038] In one embodiment, update parameters 501 are transmitted by parameter server 405 to parameter servers 415-425 in other computing nodes 302-304 as the parameters of each layer are calculated through the backward pass before the entire backward pass is completed. For example, the transmission of update parameters 501 can be performed during the execution of the backward pass in response to the completion of a layer or a subset of layers in the backward pass. Since update parameters 501 are transmitted before the entire backward pass is completed, parameters 501 represent a subset of parameters calculated through the entire backward pass (e.g., parameters of neurons in a layer or neurons in a subset of layers).

[0039] Processing unit 402 generates a subset of update parameters 501 by performing a backward pass of segment 502, which includes calculations on a subset of layers in neural network 404, including layers LN'-L2'. In one embodiment, segment 502 includes multiple layers of neural network 404, such as Figure 5 In an alternative embodiment, segment 502 includes no more than a single layer. Thus, update parameters 501 may include parameters for neurons in multiple layers or a single layer.

[0040] In one embodiment, parameter server 405 transmits a subset of update parameters in response to completion of every nth layer when performing a backward pass. For example, segment 502 of the backward pass includes n layers, and a subset of update parameters 501 is transmitted after segment 502 is completed and includes parameters for neurons in n layers. Parameter server 405 may also adjust n during the execution of the backward pass in response to network conditions, computing resource utilization, or other factors. In one embodiment, parameter server 405 determines whether update parameters 501 are different from corresponding previous parameters of the layer (i.e., calculated from an earlier iteration of the backward pass), and if the parameters have not changed, no update parameters 501 are transmitted.

[0041] The transmission of updated parameters is performed asynchronously by parameter server 405 in response to processing unit 402 determining updated parameters for neurons in n layers in the neural network during the execution of the backward pass. In other words, the timing of the transmission is independent of whether any particular backward pass or forward pass has been started or completed in computing node 301. The parameter update mechanism also allows processing unit 402 to continue executing the next segment 505 of the backward pass while transmitting updated parameters 501 calculated from the previous segment 502.

[0042] In addition, the computing nodes 302-304 that receive the update parameters 501 perform a flash update on their respective copies of the neural network as soon as practicable after receiving the update parameters 501. In one embodiment, this is achieved by directly memory writing the update parameters 501 into the memory of the nodes 302-304. The forward pass segments 503 and 504 executed in the computing nodes 302 and 304, respectively, are able to calculate outputs based on the latest parameters (e.g., neuron weights and biases) calculated by the backward pass segment 502 without waiting for the entire backward pass to complete in the node 301. For example, the forward pass segments 503 and 504 use the updated weights of the layer L2 determined by the backward pass segment 502 even before the backward pass is completed in the node 301.

[0043] Asynchronous parameter updates allow different computing nodes to process different training instances simultaneously. For example, when computing nodes 301-304 are used to train a neural network using the same set of training instances in each node, each of nodes 301-304 can perform a forward pass or a backward pass on the same or different training instances at any given time. Figure 5 As shown, computing node 301 performs a backward pass (including segment 502), while computing nodes 302 and 304 perform a forward pass (including segments 503 and 504, respectively). The forward pass in computing node 301 may also be performed concurrently with a backward pass for one or more of computing nodes 302-304.

[0044] Figure 6 is a flow chart illustrating a process 600 for training a neural network according to an embodiment. The training process 600 is similarly performed in each of the computing nodes 301-304 in the computing node cluster 209. Figure 6 In the example, the computing node executing the process (e.g., computing node 301) is referred to as a local computing node, and the other nodes in cluster 209 (e.g., computing nodes 302-304) are referred to as remote computing nodes. Each of the local computing node and the remote computing node can execute specific portions of process 600 simultaneously or at different times.

[0045] Process 600 begins at block 601. At block 601, processing logic 402 begins performing a forward pass for a local neural network 404 of a compute node 301. The forward pass accepts inputs provided by a training instance and uses neural network 404 to generate outputs based on the provided inputs.

[0046] At block 603, computing node 301 determines whether remote parameters have been received from one or more of remote computing nodes 302-304. Remote parameters are update parameters generated by performing a backward pass in a remote computing node and transmitted to local computing node 301 via communication network 310. Remote parameters are received by local computing node 301 at communication network interface 401. At block 603, if remote parameters have not been received, process 600 continues at block 607, where processing unit 402 continues to perform a forward pass using existing parameters of the neural network. At block 603, if remote parameters have been received, process 600 continues at block 605.

[0047] At block 605, in response to receiving the remote parameters, the computing node 301 updates its copy of the neural network with the remote parameters. In one embodiment, the communication network interface 401 includes logic circuitry for performing a direct memory write of the remote parameters to the local memory 403 storing the neural network 404 parameters. Thus, the communication network interface 401 updates the local neural network 404 by writing the received remote parameters directly to the memory locations in the memory 403 storing the corresponding neural network parameters. In one embodiment, in response to receiving the remote parameters, the neural network 404 is updated asynchronously with the received remote parameters so that the ongoing forward pass uses the updated parameters. Thus, the computing node 301 receives the remote parameters from the other computing nodes 302-304 and updates the parameters in the local neural network 404 accordingly at any time during the execution of the forward pass.

[0048] After updating the neural network 404 with the remote parameters, the processing unit 402 continues to perform a forward pass to calculate the output of the training instance based on the remote parameters at block 607. For example, the remote parameters may be used as weights and / or biases in neurons of the neural network, where the weights and / or biases affect the output generated from the forward pass.

[0049] At block 609, if the forward pass is not complete (i.e., computations have not been performed for all layers in the neural network), process 600 returns to block 603. Thus, processing unit 402 performs the forward pass (as provided at block 607) while updating the parameters in the neural network (as provided at blocks 603-605) according to any received remote parameters until the forward pass is completed at block 609. When the forward pass is completed, process 600 continues at block 611.

[0050] At block 611, processing unit 402 begins executing a backward pass in computing node 301. During the backward pass, processing unit 402 determines the difference between the expected output of the training instance and the final output generated by the forward pass, and then propagates the error through the layers of the neural network in a backward order to determine the contribution of each neuron to the error. The parameters of each neuron are adjusted based on their respective contributions to reduce the error. Thus, the backward pass calculation determines an updated set of parameters for the neural network.

[0051] At block 613, the backward pass of segment 502 is performed for a subset of the neuron layers in neural network 404; thus, a corresponding subset of update parameters is determined. In one embodiment, the layer subset includes only one layer; alternatively, the layer subset includes more than one layer and less than all layers in neural network 404 (e.g., Figure 5 shown).

[0052] At block 615, parameter server 405 transmits the subset of updated parameters determined from block 613 to remote computing nodes 302-304. In one embodiment, parameter server 405 avoids transmitting parameters that have not been changed by the backward pass. Thus, parameter server 405 transmits updated parameters 501 in response to determining that the subset of updated parameters for a subset of one or more layers is different from the set of corresponding previous parameters for the subset. In one embodiment, parameter server 405 transmits only parameters that have changed relative to the corresponding previous parameters, where the corresponding previous parameters are the most recently used parameter values ​​for the same neuron during the most recent forward pass prior to the ongoing backward pass.

[0053] The transmission of the subset of update parameters 501 is performed asynchronously during execution of the backward pass in response to determining update parameters 501 for a subset of layers in the segment 502 of the backward pass. In one embodiment, the segment 502 includes a plurality of layers LN'-L2', such that the update parameters 501 are transmitted asynchronously in response to completion of the last layer in the segment 502. In an alternative embodiment, the segment 502 of the backward pass includes no more than a single layer, such that the subset of update parameters 501 is transmitted asynchronously during execution of the backward pass in response to determining update parameters 501 for a single layer.

[0054] At block 617, if the backward pass is not complete (i.e., the backward pass has not traversed all layers in the neural network), process 600 returns to block 613. At block 613, processing unit 402 continues to perform the backward pass for the layers that have not yet been processed. In one embodiment, transmitting the subset of update parameters 501 (as provided at block 615) is performed simultaneously with continuing to perform the backward pass for one or more subsequent layers in the local neural network (as provided at block 613).

[0055] Thus, blocks 613-617 are repeated until the backward pass is completed. In one embodiment, the backward pass includes multiple segments, each segment traversing a different subset of the layers in the neural network. Thus, the subset of update parameters 501 is one of many subsets of parameters transmitted in response to the completion of each segment when performing the backward pass. In one embodiment, each segment traverses n layers, and each subset of update parameters therefore includes parameters for n layers. In some embodiments, the value of n is dynamically adjusted in response to network conditions, computing resource utilization, or other factors during the execution of the backward pass. At block 617, if the backward pass is complete, process 600 returns to block 601 to begin the next subsequent forward pass.

[0056] In one embodiment, remote parameter updates received from other nodes 302-304 during the execution of the backward pass are queued until the forward pass begins. Thus, the neural network 404 is updated with the queued remote parameters before the forward pass begins at block 601. Alternatively, the remote parameters received during the execution of the backward pass may be used to update the neural network during the execution of the backward pass (i.e., before the backward pass is completed). In one embodiment, the neural network 404 is updated with any remote parameters received during the execution of the backward pass (at blocks 611-617) through a similar process as provided at blocks 603 and 605, as these parameters are received via direct memory writes to the local memory 403.

[0057] Each of the other computing nodes 302-304 includes components similar to computing node 301; therefore, each of these nodes also performs a process similar to process 600. In one embodiment, each of computing nodes 301-304 performs a similar back-propagation process, including forward and backward passes, to calculate parameters for different parts of the same neural network. In particular, each of computing nodes 301-304 stores its own copy of the neural network and calculates parameters for different subsets of neurons in the neural network. In one embodiment, each neuron in the neural network is included in no more than one of these subsets of neurons. For example, node 301 can calculate parameters for a first subset of neurons, while node 302 calculates parameters for a second subset of neurons, wherein the first subset of neurons and the second subset of neurons are mutually exclusive sets. In an alternative embodiment, each neuron in the neural network is included in more than one subset of neurons. Each computing node 301-304 updates the parameters in its own copy of the neural network based on its own calculations and based on the updated parameters received from other computing nodes.

[0058] Considering computing node 301 as a local computing node, remote computing nodes 302-304 also each execute process 600 to calculate parameters of different subsets of neurons. The time to execute the backward pass and forward pass in any one of nodes 301-304 is independent of the execution time of the backward and forward passes in other nodes.

[0059] Thus, the backward pass in the local computing node 301 may be performed simultaneously with the forward pass in one or more of the remote computing nodes 302-304, so that the updated parameters transmitted by the local computing node 301 during the local backward pass are used to update the remote copies of their neural networks at the remote computing nodes 302-304 during their respective forward passes. Each of the remote computing nodes 302-304 executes its own version of block 603 and, in response to receiving a subset of the updated parameters 501 from the local computing node 301, updates its copy of the neural network with the received parameters 501, as similarly provided at block 605. In one embodiment, the parameters 501 received at each remote computing node are for a subset of neurons that are mutually exclusive with the set of neurons for which the parameters are calculated in the remote computing node. Each remote computing node 302-304 completes its own remote forward pass based on the updated parameters 501 that are updated in their respective copies of the neural network stored in the remote computing node.

[0060] The backward pass (similar to blocks 611-617) in one or more of the remote computing nodes 302-304 may be performed concurrently with the execution of the forward pass in the local neural network 404 of the local computing node 301. In this case, the update parameters determined from the remote backward pass are transmitted to the local computing node 301 and used during the execution of the local forward pass to update the local neural network 404. Each of the remote computing nodes 302-304 performs the remote backward pass in its own copy of the neural network to determine the update parameters for its assigned subset of neurons.

[0061] Through the operation of the training process 600 in each of the computing nodes 301-304, each computing node performs a backward pass (wherein the parameters are asynchronously transmitted at a fine granularity when the update parameters are determined), and performs a forward pass (wherein the parameters are asynchronously updated when they are received in the neural network during the forward pass). Using this method, the delay between determining the update parameters in one computing node and updating the parameters in another computing node is minimized. The fine granularity of the updates and the resulting smaller size of the update messages reduce the possibility of network congestion because the network load can be more evenly distributed over time. In addition, due to the asynchronous nature of the messages and updates, successfully sending and receiving update messages is not on the critical path of the training process. Overall, this will result in the training process converging faster while more efficiently utilizing available network and computing resources.

[0062] The method 600 for training a neural network 404 includes: receiving remote parameters from a set of one or more remote computing nodes 302-304 at a local computing node 301; starting a forward pass in a local neural network 404 in the local computing node 301 to determine an output based on the remote parameters; starting a backward pass in the local neural network 404 to determine updated parameters 501 for the local neural network 404; and before completing the backward pass, transmitting a subset of the updated parameters 501 to the set of remote computing nodes 302-304.

[0063] In method 600, the performance of the backward pass includes computing a subset of updated parameters 501 for a first layer in the local neural network 404, and transmitting the subset of updated parameters 501 in response to determining that the subset of updated parameters 501 for the layer is different from a corresponding set of previous parameters for the layer.

[0064] In method 600, the execution of the backward pass includes: calculating a subset of update parameters 501 for a first layer in the local neural network 404, wherein transmitting the subset of update parameters 501 is performed simultaneously with performing the backward pass for one or more subsequent layers in the local neural network 404.

[0065] In method 600, the subset of update parameters 501 is one of a plurality of subsets of update parameters 501 transmitted in response to completion of every nth layer while performing a backward pass, wherein method 600 further comprises dynamically adjusting a value of n during performing the backward pass.

[0066] In method 600, execution of the forward pass begins before receiving the remote parameters, and method 600 also includes updating the local neural network 404 with the remote parameters, and completing execution of the forward pass to compute outputs based on the remote parameters.

[0067] The method 600 also includes asynchronously updating the local neural network 404 with the remote parameters in response to receiving the remote parameters, wherein transmitting a subset of the update parameters 501 is performed asynchronously in response to determining update parameters 501 for one of the plurality of layers in the neural network 404 during performing a backward pass.

[0068] The method 600 further includes: in response to receiving the remote parameters, updating the local neural network 404 with the remote parameters by executing a direct memory write of the remote parameters to the local memory 403 in the local computing node 301 .

[0069] In method 600, update parameters 501 for a first subset of neurons in local neural network 404 are determined, and method 600 further includes: updating a second subset of neurons in local neural network 404 with remote parameters, wherein the first subset of neurons and the second subset of neurons are mutually exclusive.

[0070] Method 600 also includes: for each remote computing node in the set of remote computing nodes 302-304: receiving, at the remote computing node, a subset of update parameters 501 from the local computing node 301; performing a remote forward pass in a copy of the local neural network 404 stored in the remote computing node based on the update parameters 501; determining the remote parameters during a remote backward pass performed in the copy of the local neural network 404; and transmitting the remote parameters from the remote computing node to the local computing node 301 before completing the remote backward pass.

[0071] The method 600 also includes performing a forward pass in the local neural network 404 concurrently with performing a remote backward pass in at least one of the set of remote computing nodes 302-304.

[0072] The computing node 301 includes: a communication network interface 401, which is configured to receive remote parameters from a set of one or more remote computing nodes 302-304; a processing unit 402, which is configured to perform a forward pass in the neural network 404 to calculate outputs based on the remote parameters, and to perform a backward pass in the neural network 404 to determine updated parameters 501 for the neural network 404; and a parameter server 405, which is coupled to the processing unit 402 and is configured to transmit a subset of the updated parameters 501 to the set of remote computing nodes 302-304 before completing the backward pass.

[0073] In the computing node 301, the processing unit 402 is also configured to perform a backward pass by calculating a subset of updated parameters 501 for a layer in the neural network 404, and the parameter server 405 is also configured to transmit the subset of updated parameters 501 in response to determining that the subset of updated parameters 501 is different from the set of corresponding previous parameters for the layer.

[0074] In the computing node 301, the parameter server 405 is configured to transmit one or more of multiple subsets of update parameters 501 in response to completion of each nth layer when performing a backward pass, and the processing unit 402 is also configured to dynamically adjust the value of n during the execution of the backward pass.

[0075] The computing node 301 also includes a memory 403 coupled to the communication network interface 401, wherein the communication network interface 401 is further configured to update the neural network 404 by storing the remote parameters in the memory 403 via direct memory write, and wherein the processing unit 402 is further configured to start executing the forward pass before receiving the remote parameters, and after updating the neural network 404 with the remote parameters, complete the execution of the forward pass to calculate the output based on the remote parameters.

[0076] In computing node 301, communication network interface 401 is further configured to asynchronously update neural network 404 with remote parameters in response to receiving the remote parameters, and parameter server 405 is further configured to asynchronously transmit a subset of update parameters 501 in response to determining update parameters 501 for one of the plurality of layers in neural network 404 during execution of a backward pass.

[0077] The computing system 200 includes a communication network 310 and a plurality of computing nodes 301-304 coupled to the communication network 310, wherein each of the plurality of computing nodes 301-304 includes: a communication network interface 401 configured to receive remote parameters from other computing nodes in the plurality of computing nodes 301-304; a processing unit 402 configured to perform a forward pass in the neural network 404 to calculate outputs based on the remote parameters, and to perform a backward pass in the neural network 404 to determine updated parameters 501 of the neural network 404; and a parameter server 405 coupled to the processing unit 402 and configured to transmit a subset of the updated parameters 501 to other computing nodes before completing the backward pass.

[0078] In computing system 200, a forward pass to a first computing node among the plurality of computing nodes 301-304 is performed concurrently with a backward pass to a second computing node among the plurality of computing nodes 301-304.

[0079] In computing system 200 , each computing node in plurality of computing nodes 301 - 304 is configured to determine update parameters 501 for a different subset of neurons in neural network 404 .

[0080] In computing system 200, for each computing node in the plurality of computing nodes 301-304, the communication network interface 401 of the computing node is further configured to update the neural network 404 with remote parameters after starting the forward pass and before completing the forward pass.

[0081] In the computing system 200, for each of the plurality of computing nodes 301-304, the communication network interface 401 of the computing node is further configured to, in response to receiving the remote parameters, asynchronously update the neural network 404 with the remote parameters by performing a direct memory write of the remote parameters to a local memory 403 in the local computing node 301, and the parameter server 405 of the computing node is further configured to, during execution of a backward pass, asynchronously transmit a subset of update parameters 501 in response to the processing unit 402 determining an update parameter 501 for one of the plurality of layers in the neural network 404.

[0082] As used herein, the term "coupled to" may mean coupled directly or indirectly through one or more intermediate components. Any signal provided by the various buses described herein may be time-division multiplexed with other signals and may be provided by one or more common buses. In addition, the interconnections between circuit components or blocks may be shown as buses or single signal lines. Each of the buses may alternatively be one or more single signal lines, and each of the single signal lines may alternatively be a bus.

[0083] Certain embodiments may be implemented as a computer program product, which may include instructions stored on a non-transitory computer-readable medium. These instructions may be used to program a general-purpose or special-purpose processor to perform the described operations. Computer-readable media include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form (e.g., software, processing application). Non-transitory computer-readable storage media may include, but are not limited to, magnetic storage media (e.g., floppy disks); optical storage media (e.g., CD-ROMs); magneto-optical storage media; read-only memory (ROM); random access memory (RAM); erasable programmable memory (e.g., EPROM and EEPROM); flash memory or another type of medium suitable for storing electronic instructions.

[0084] In addition, some embodiments can be practiced in a distributed computing environment, where the computer-readable medium is stored on and / or executed by more than one computer system. In addition, information transmitted between computer systems can be pulled or pushed between the transmission media connecting the computer systems.

[0085] In general, the data structure representing the computing system 200 and / or the portion thereof carried on the computer-readable storage medium may be a database or other data structure that can be read by a program and used directly or indirectly to manufacture hardware including the computing system 200. For example, the data structure may be a behavioral level description of the hardware functionality in a high-level design language (HDL), such as Verilog or VHDL, or a register transfer level (RTL) description. The description may be read by a synthesis tool, which may synthesize the description to generate a netlist including a series of gates from a synthesis library. The netlist includes a set of gates that also represent the functionality of the hardware including the computing system 200. The netlist may then be placed and routed to generate a data set describing a geometric shape to be applied to a mask. The mask may then be used in various semiconductor manufacturing steps to produce one or more semiconductor circuits corresponding to the computing system 200. Alternatively, the database on the computer-readable storage medium may be a netlist (with or without a synthesis library) or a data set (as desired), or a Graphic Data System (GDS) II data.

[0086] Although the operation of this paper method is shown and described in a particular order, the order of operation of each method can be changed so that some operations can be performed in reverse order or so that some operations can be performed at least in part simultaneously with other operations. In another embodiment, the instructions or sub-operations of different operations can be intermittent and / or alternating modes.

[0087] In the above description, the embodiments have been described with reference to specific exemplary embodiments thereof. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader scope of the embodiments as set forth in the appended claims. Therefore, the description and drawings are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for training a neural network, comprising: Begin executing a forward pass in the local neural network in the local compute node; After initiating execution of the forward pass, receiving remote parameters from the set of one or more remote computing nodes; updating the local neural network with the remote parameters, and completing said execution of said forward pass to compute a final output based on said remote parameters; Initiating a backward pass in the local neural network to determine updated parameters of the local neural network; and Prior to completing the backward pass, a subset of the update parameters is transmitted to the set of one or more remote computing nodes.

2. The method of claim 1, wherein: said performing said backward pass comprises computing said subset of said updated parameters for a first layer in said local neural network, and Transmitting the subset of updated parameters is performed in response to determining that the subset of updated parameters for the first layer is different than a set of corresponding previous parameters for the first layer.

3. The method of claim 1 , wherein said performing the backward pass comprises computing the subset of the updated parameters for a first layer in the local neural network, and wherein said transmitting the subset of the updated parameters is performed concurrently with performing the backward pass for one or more subsequent layers in the local neural network.

4. The method of claim 1, wherein the subset of the update parameters is one of a plurality of subsets of the update parameters transmitted in response to completion of each nth layer in performing the backward pass, and wherein the method further comprises dynamically adjusting during the performing the backward pass n The value of .

5. The method of claim 1, further comprising: In response to receiving the remote parameters, asynchronously updating the local neural network with the remote parameters, wherein transmitting the subset of the updated parameters is performed asynchronously during performing the backward pass in response to determining the updated parameters for one of a plurality of layers in the local neural network.

6. The method of claim 1, further comprising: In response to receiving the remote parameters, the local neural network is updated with the remote parameters by performing a direct memory write of the remote parameters to a local memory in the local computing node.

7. The method of claim 1, wherein: The update parameters are determined for a first subset of neurons in the local neural network; and The method also includes updating a second subset of neurons in the local neural network with the remote parameters, wherein the first subset of neurons and the second subset of neurons are mutually exclusive.

8. The method of claim 1, further comprising, for each remote computing node in the set of one or more remote computing nodes: receiving, at the remote computing node, the subset of the update parameters from the local computing node; performing a remote forward pass in a copy of the local neural network stored in the remote computing node based on the updated parameters; determining the long-range parameters during a long-range backward pass performed in the copy of the local neural network; and Prior to completing the remote backward pass, the remote parameters are transmitted from the remote computing node to the local computing node.

9. The method of claim 8, further comprising: The forward pass is performed in the local neural network simultaneously with the remote backward pass being performed in at least one remote computing node in the set of one or more remote computing nodes.

10. The method of claim 1, further comprising: For each of a plurality of layers in the local neural network, the forward pass is performed by computing an intermediate output of the layer based on at least one of the following: The set of inputs provided by the training examples, and The intermediate output of the previous layer in the local neural network.

11. The method of claim 10, further comprising: The backward pass is performed by computing, for each of the plurality of layers in the local neural network, the subset of updated parameters for the layer based on an error between the final output determined from the forward pass and an expected output for the training instance.

12. A computing device for training a neural network, comprising: a communication network interface configured to receive remote parameters from a set of one or more remote computing nodes, and to update the neural network by storing the remote parameters in a memory; A processing unit, wherein the processing unit is configured to: prior to said receiving remote parameters, starting execution of a forward pass in said neural network, After updating the neural network using the remote parameters, completing execution of the forward pass to compute a final output based on the remote parameters, performing a backward pass through the neural network to determine updated parameters for the neural network; as well as A parameter server is coupled to the processing unit and configured to transmit a subset of the updated parameters to the one or more sets of remote computing nodes before completing the backward pass.

13. The computing device of claim 12, wherein: The processing unit is further configured to perform the backward pass by computing the subset of the update parameters for the layers in the neural network, and The parameter server is further configured to transmit the subset of updated parameters in response to determining that the subset of updated parameters is different from a corresponding set of previous parameters for the layer.

14. The computing device of claim 12, wherein: The parameter server is configured to transmit one or more of a plurality of subsets of the updated parameters in response to completion of each nth layer in performing the backward pass, and The processing unit is further configured to dynamically adjust during the execution of the backward pass n The value of .

15. The computing device of claim 12, further comprising: Wherein the communication network interface is further configured to store the remote parameters in the memory via direct memory write.

16. The computing device of claim 12, wherein: The communication network interface is further configured to asynchronously update the neural network with the remote parameters in response to receiving the remote parameters, and The parameter server is further configured to asynchronously transmit a subset of the update parameters in response to determining the update parameters for one of a plurality of layers in the neural network during performing the backward pass.

17. The computing device of claim 12, wherein the processing unit is further configured to: For each of a plurality of layers in the local neural network, the forward pass is performed by computing an intermediate output of the layer based on at least one of the following: The set of inputs provided by the training examples, and an intermediate output of a previous layer in the local neural network; and The backward pass is performed by computing, for each layer of the plurality of layers in the local neural network, the subset of updated parameters for the layer based on an error between the final output determined from the forward pass and an expected output for the training instance.

18. A computing system for training a neural network, comprising: Communications networks; and A plurality of computing nodes coupled to the communication network, wherein each computing node in the plurality of computing nodes comprises: a communication network interface configured to receive remote parameters from other computing nodes of the plurality of computing nodes and to update the neural network by storing the remote parameters in a memory; A processing unit, wherein the processing unit is configured to: prior to said receiving remote parameters, starting execution of a forward pass in said neural network, After updating the neural network using the remote parameters, completing execution of the forward pass to compute a final output based on the remote parameters, and performing a backward pass through the neural network to determine updated parameters for the neural network; and A parameter server is coupled to the processing unit and configured to transmit the subset of updated parameters to the other computing nodes before completing the backward pass.

19. The computing system of claim 18, wherein the forward pass to a first computing node of the plurality of computing nodes is performed concurrently with the backward pass to a second computing node of the plurality of computing nodes.

20. The computing system of claim 18, wherein each computing node of the plurality of computing nodes is configured to determine the update parameters for a different subset of neurons in the neural network.

21. The computing system of claim 18, wherein for each computing node in the plurality of computing nodes, the communication network interface of the computing node is further configured to update the neural network with the remote parameters after starting the forward pass and before completing the forward pass.

22. The computing system of claim 18, wherein for each computing node in the plurality of computing nodes: The communication network interface of the computing node is further configured to, in response to receiving the remote parameters, asynchronously update the neural network with the remote parameters by performing a direct memory write of the remote parameters to a local memory in a local computing node; and The parameter server of the computational node is further configured to asynchronously transmit a subset of the updated parameters in response to the processing unit determining the updated parameters for one of a plurality of layers in the neural network during execution of the backward pass.

Images