A neural network distributed training method, device, equipment and storage medium

Abstract

The present application provides a neural network distributed training method, device, equipment and storage medium, and relates to the field of machine learning technology. The method improves the training architecture of the distributed neural network and improves the computing resource utilization of each node in the distributed training, including: for each node in the cluster, a synchronization process and a training process running in parallel with the synchronization process are established; the node determines the Nth training cycle of the training process according to the status information of the synchronization process; in the Nth training cycle, the node trains the target neural network model to obtain the local gradient; the node accumulates the local gradient to obtain the cumulative gradient; when the node performs the training of the N+1th training cycle in the training process, the node accumulates the gradient synchronously with other nodes in the synchronization process, and calculates the first average gradient; the node uses the first average gradient in the synchronization process to update the parameters of the target neural network model at the end of the Nth training cycle.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a neural network distributed training method, apparatus, device and storage medium. Background Art

[0002] Training neural networks based on large datasets has achieved impressive performance in many fields, for example, using large datasets to train neural networks improves the accuracy of applications in areas such as image recognition, natural language processing, fraud detection, and recommendation systems. However, training neural networks based on large datasets requires high performance from computing devices, which is difficult to accomplish with a single computing device.

[0003] Distributed neural network training makes it possible to train complex neural networks and train neural networks using large-scale training data sets. However, due to performance differences among nodes involved in distributed training and different time required to process data, some node resources are wasted. At the same time, the inevitable and time-consuming information exchange process between nodes also leads to low node utilization. Summary of the invention

[0004] The embodiments of the present application provide a neural network distributed training method, apparatus, device and storage medium to improve the training architecture of the distributed neural network and increase the computing resource utilization of each node in the distributed training.

[0005] A first aspect of an embodiment of the present application provides a neural network distributed training method, the method comprising:

[0006] For each node in the cluster, establish a synchronization process and a training process running in parallel with the synchronization process;

[0007] The node determines the Nth training cycle of the training process according to the state information of the synchronization process, where N is an integer greater than 0;

[0008] In the Nth training cycle, the node trains the target neural network model in the training process to obtain a local gradient;

[0009] The node accumulates local gradients obtained by training the target neural network model multiple times to obtain a cumulative gradient;

[0010] When the node performs training for the N+1th training cycle in the training process, the node synchronizes the accumulated gradient with other nodes that jointly train the target neural network model in the synchronization process, and calculates a first average gradient based on the accumulated gradients of the other nodes;

[0011] After the training process completes the training of the N+1th training cycle, the node uses the first average gradient in the synchronization process to update the parameters of the target neural network model at the end of the Nth training cycle.

[0012] Optionally, the method further comprises:

[0013] A counter is set in the training process; wherein the counter is used to record the number of times the target neural network model is trained in one cycle;

[0014] The node determines, according to the state information of the synchronization process, an Nth training cycle of the training process, including:

[0015] Initializing the target neural network model and determining the first training cycle of the training process;

[0016] Take N=n+1, where n is an integer greater than 0, and execute the following steps: when the status information of the synchronization process is in an idle state, the node assigns the synchronization state to the status information of the synchronization process through the training process so that the synchronization process enters a synchronization state; at the same time, the node sets the cumulative number of training times recorded by the counter to zero, and determines that the training process ends the N-1th training cycle and starts the Nth training cycle.

[0017] Optionally, in the Nth training cycle, the node trains the target neural network model in the training process to obtain a local gradient, including:

[0018] Sequentially taking m from 1 to M, where M is an integer greater than 1, performing an m-th training on the target neural network model in the N-th training cycle, and obtaining a local gradient corresponding to the m-th training;

[0019] When the cumulative number of training times recorded by the counter is zero, the node clones the parameters of the target neural network model locally using the training process to obtain cloned model parameters;

[0020] When the cumulative number of training times recorded by the counter is not zero, the clone model parameters are updated using the local gradient corresponding to the mth training, and the cumulative number of training times recorded by the counter is increased by one until the state information of the synchronization process is assigned an idle state.

[0021] Optionally, the node accumulates local gradients obtained by training the target neural network model multiple times to obtain a cumulative gradient, including:

[0022] Sequentially taking m from 1 to M, where M is an integer greater than 1, performing an m-th training on the target neural network model in the N-th training cycle, and obtaining a local gradient corresponding to the m-th training;

[0023] When m is equal to 1, the local gradient corresponding to the m-th training is stored locally to obtain the cumulative gradient corresponding to the m-th training;

[0024] When m is not equal to 1, the local gradient corresponding to the m-th training is added to the cumulative gradient corresponding to the m-1-th training to obtain the cumulative gradient corresponding to the m-th training.

[0025] Optionally, when the state information of the synchronization process is in an idle state, the node assigns a synchronization state to the state information of the synchronization process through the training process so that the synchronization process enters a synchronization state, and the method further includes:

[0026] In the synchronization process, the node obtains the parameters of the target neural network model at the end of the N-1th training cycle according to the accumulated number of training times recorded by the counter, and obtains the model parameters to be updated;

[0027] The node synchronizes the accumulated gradient with other nodes that jointly train the target neural network model in the synchronization process, and calculates and obtains a first average gradient based on the accumulated gradient of other nodes, including:

[0028] When the synchronization process enters the synchronization state for the first time, directly synchronizing the accumulated gradient with other nodes of the target neural network model to calculate a first average gradient;

[0029] When the synchronization process does not enter the synchronization state for the first time, a second average gradient is obtained; wherein the second average gradient is obtained by the accumulated gradient obtained by the node synchronization corresponding to the N-1th training cycle;

[0030] Updating the model parameters to be updated using the second average gradient;

[0031] The accumulated gradient is synchronized with other nodes of the target neural network model to obtain a first average gradient.

[0032] Optionally, when the node performs training for the N+1th training cycle in the training process, after synchronizing the accumulated gradient with other nodes that jointly train the target neural network model in the synchronization process, the method further includes:

[0033] The synchronization state is assigned to the state information of the synchronization process through the synchronization process, so that the training process completes the N+1th training cycle.

[0034] A second aspect of an embodiment of the present application provides a neural network distributed training device, the device comprising:

[0035] A process establishment module, used to establish a synchronization process and a training process running in parallel with the synchronization process for each node in the cluster;

[0036] A training cycle determination module, used to enable the node to determine the Nth training cycle of the training process according to the state information of the synchronization process, where N is an integer greater than 0;

[0037] A training module, configured to, during the Nth training cycle, train the target neural network model in the training process by the node to obtain a local gradient;

[0038] An accumulation module, used to enable the node to accumulate local gradients obtained by training the target neural network model multiple times to obtain a cumulative gradient;

[0039] A synchronization module, configured to synchronize the accumulated gradient with other nodes that jointly train the target neural network model in the synchronization process when the node performs the training for the N+1th training cycle in the training process, and calculate a first average gradient based on the accumulated gradients of the other nodes;

[0040] An updating module is used for, after the training process completes the training of the N+1th training cycle, the node uses the first average gradient in the synchronization process to update the parameters of the target neural network model at the end of the Nth training cycle.

[0041] Optionally, the device further comprises:

[0042] A setting module, used to set a counter in the training process; wherein the counter is used to record the number of times the target neural network model is trained in one cycle;

[0043] The training cycle determination module comprises:

[0044] An initialization submodule, used to initialize the target neural network model and determine the first training cycle of the training process;

[0045] The training cycle determination submodule is used to take N=n+1, where n is an integer greater than 0, and execute the following steps: when the status information of the synchronization process is in an idle state, the node assigns the synchronization state to the status information of the synchronization process through the training process so that the synchronization process enters a synchronization state; at the same time, the node sets the cumulative number of training times recorded by the counter to zero, and determines that the training process ends the N-1th training cycle and starts the Nth training cycle.

[0046] Optionally, the training module includes:

[0047] The first training submodule is used to sequentially take m from 1 to M, where M is an integer greater than 1, perform an m-th training on the target neural network model in the N-th training cycle, and obtain a local gradient corresponding to the m-th training;

[0048] A cloning submodule, configured to, when the cumulative number of training times recorded by the counter is zero, cause the node to clone the parameters of the target neural network model locally using the training process to obtain cloned model parameters;

[0049] The first updating submodule is used to update the clone model parameters using the local gradient corresponding to the mth training when the cumulative number of training times recorded by the counter is not zero, and add one to the cumulative number of training times recorded by the counter until the state information of the synchronization process is assigned an idle state.

[0050] Optionally, the accumulation module includes:

[0051] The second training submodule is used to sequentially take m from 1 to M, where M is an integer greater than 1, perform an m-th training on the target neural network model in the N-th training cycle, and obtain a local gradient corresponding to the m-th training;

[0052] A first cumulative gradient, used for storing the local gradient corresponding to the m-th training locally when m is equal to 1, to obtain the cumulative gradient corresponding to the m-th training;

[0053] The second accumulated gradient is used for accumulating the local gradient corresponding to the m-th training to the accumulated gradient corresponding to the m-1-th training to obtain the accumulated gradient corresponding to the m-th training when m is not equal to 1.

[0054] Optionally, the device further comprises:

[0055] A parameter acquisition module, used for, in the synchronization process, the node to acquire the parameters of the target neural network model at the end of the N-1th training cycle according to the cumulative number of training times recorded by the counter, and obtain the model parameters to be updated;

[0056] The synchronization module comprises:

[0057] A first synchronization submodule, configured to directly synchronize the accumulated gradient with other nodes of the target neural network model and calculate a first average gradient when the synchronization process enters the synchronization state for the first time;

[0058] A gradient acquisition submodule, used for acquiring a second average gradient when the synchronization process does not enter the synchronization state for the first time; wherein the second average gradient is obtained by the accumulated gradient obtained by the node synchronization corresponding to the N-1th training cycle;

[0059] A second updating submodule, used for updating the model parameters to be updated using the second average gradient;

[0060] The first synchronization submodule is used to synchronize the accumulated gradient with other nodes of the target neural network model to obtain a first average gradient.

[0061] Optionally, the device further comprises:

[0062] An assignment module is used to assign the synchronization state to the state information of the synchronization process through the synchronization process, so that the training process completes the N+1th training cycle.

[0063] A third aspect of an embodiment of the present application provides a readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps in the method described in the first aspect of the present application are implemented.

[0064] A fourth aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method described in the first aspect of the present application are implemented.

[0065] The embodiment of the present application sets a synchronization process and a training process running in parallel at the node, so that the training process, based on the state information of the synchronization process, continuously repeats the local training task of the current cycle when the synchronization process synchronizes the gradient task, and updates the accumulated gradient with the gradient obtained by local training. When the synchronization process ends the task of synchronizing the gradient, the accumulated gradient is synchronized, and the parameters of the neural network model before the start of the current cycle are updated with the average gradient obtained by the last synchronization. In the above training process, the training process is always calculating while the node is synchronizing the gradient, making full use of the computing resources of the node. At the same time, because the node is always updating the accumulated gradient in the training process, after the end of the Nth training cycle, the accumulated gradient will be used to update the parameters of the target neural network model before the start of the Nth training cycle. The target neural network will not be affected by the gradient obtained by a single training in the Nth training cycle, thereby ensuring the timeliness of the target neural network parameters and further ensuring the convergence of the target neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0067] Figure 1 It is the timeline diagram of the synchronous stochastic gradient descent algorithm;

[0068] Figure 2 is a flowchart of the steps of the neural network distributed training method according to an embodiment of the present application;

[0069] Figure 3 is a timeline diagram of a neural network distributed training method in an example of an embodiment of the present application;

[0070] Figure 4 is a flow chart of the distributed training of a neural network according to an embodiment of the present application;

[0071] Figure 5 It is a schematic diagram of the structure of a neural network distributed training device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0073] Distributed training of neural networks refers to the joint training of a model constructed by a neural network by multiple nodes. Nodes can be computer devices, and multiple nodes form a cluster of training models. In order to ensure the convergence of the model during distributed training, there is an inevitable and time-consuming information exchange process between nodes. During the information exchange process between nodes, the nodes cannot train the model, which leads to a decrease in the utilization rate of the cluster.

[0074] Taking the classic synchronous stochastic gradient descent (SSGD) algorithm as an example, the training process of the SSGD algorithm is mainly divided into the following four steps: (1) SSGD evenly distributes the entire training data set, and each node in the cluster obtains a sample of the training data set. (2) Each node uses the received training data set sample to train the neural network model locally and calculate the model gradient. (3) The gradient information is synchronized between the nodes in the cluster, and each node will obtain a global average gradient that averages the gradients of all nodes. (4) Each node in the cluster uses the global average gradient to update the neural network model.

[0075] The SSGD algorithm maintains the convergence of the neural network training through step (3), which is used as a separate data synchronization stage in which the gradients of each node are synchronized. Figure 1It is the timeline diagram of the synchronous stochastic gradient descent algorithm, such as Figure 1 As shown in the figure, after receiving the same size of training data set samples, nodes 1, 2 and 3 start the local training phase. Due to the different performance of nodes 1, 2 and 3, the time for nodes 1, 2 and 3 to complete the local training phase is also different. However, nodes 1, 2 and 3 need to start synchronizing gradients at the same time. Inevitably, some nodes will have idle time between the local training phase and the data synchronization phase, resulting in a waste of node computing resources. In addition, compared with the idle time, the data synchronization phase takes longer. Nodes 1, 2 and 3 in the data synchronization phase are completely in a waiting state, and their processing cores (CPU, GPU) are almost unloaded in the data synchronization phase. Therefore, the data synchronization phase will cause greater resource waste compared to the idle time.

[0076] On the other hand, the four steps in the distributed training process of the neural network model trained using the SSGD algorithm are highly coupled, that is, each node can only update the neural network model using the synchronized gradients after the synchronization of gradient information in step (3) is completed, resulting in the inability of the synchronization data stage to coincide with the local training stage.

[0077] In order to alleviate the problem of too long data synchronization phase of each node in the training of distributed neural network models, the solutions proposed in the related field include: (1) increasing the batch size of training data during the training process. The increase of batch size can significantly reduce the number of SSGD cycles, reduce the number of synchronizations, and thus reduce the synchronization time. However, it will increase the training time of the node (increase the amount of training per time), increase the configuration requirements for a single node, and reduce the generalization ability of the model. (2) Periodic synchronization of nodes. The node performs local training several times and then synchronizes once, thereby reducing the synchronization time by reducing the synchronization frequency. However, it will reduce the convergence rate of the model, and the number of local trainings required between two synchronizations is difficult to determine. (3) Data compression method. The node compresses the data before synchronization and decompresses the data after synchronization. By reducing the single synchronization time, the synchronization time of the entire training process is reduced. Although the data compression method can greatly reduce the synchronization time, it will introduce additional steps in the training process, including sampling, compression, decompression, etc., which to some extent offsets the synchronization time saved.

[0078] In order to improve the efficiency of distributed training of neural network models, the applicant improved the training architecture of distributed neural networks and proposed a parallel training method of local training and synchronized gradients. Two processes were initialized at each node participating in the training. One process performs local training continuously, and the other process synchronizes gradients continuously, ensuring that the hardware (CPU, GPU) utilization of a single node is improved, thereby improving the utilization of the entire distributed cluster.

[0079] Figure 2 is a flowchart of the steps of the neural network distributed training method according to the embodiment of the present application, Figure 3 is a timeline diagram of a neural network distributed training method in an example of an embodiment of the present application, such as Figure 2 and Figure 3 As shown, training a neural network includes the following steps:

[0080] Step S21: for each node in the cluster, establishing a synchronization process and a training process running in parallel with the synchronization process;

[0081] A cluster refers to a distributed cluster consisting of multiple nodes participating in the distributed training of a neural network model.

[0082] The synchronization process and the training process share a cluster number, which is determined by the nodes where the synchronization process and the training process are located. For example, there are Q computer devices in the cluster, each with a different cluster number, and the cluster number of computer device A is i. Then the synchronization process and the training process established by computer device A share the cluster number i.

[0083] The synchronization process and the training process exchange data through memory, that is, the synchronization process can read part of the data of the training process, and the training process can also read part of the data of the synchronization process.

[0084] Step S22: the node determines the Nth training cycle of the training process according to the state information of the synchronization process, where N is an integer greater than 0;

[0085] Each node in the cluster determines the number of training cycles currently being performed locally based on the status information of the local synchronization process.

[0086] The status information of the synchronization process includes the synchronization state and the idle state. Assume that the training process of node A has been executed to the 5th training cycle. When the status information of the synchronization process of node A is the synchronization state, node A continues the local training of the 5th training cycle. When the status information of the synchronization process of node A is the idle state, node A starts to execute the local training of the 6th training cycle.

[0087] Step S23: In the Nth training cycle, the node trains the target neural network model in the training process to obtain a local gradient;

[0088] The target neural network model refers to a pre-built neural network to be trained. The target neural network can be built using a convolutional neural network, a recurrent neural network, a feedforward neural network, etc.

[0089] Step S24: the node accumulates local gradients obtained by training the target neural network model multiple times to obtain a cumulative gradient;

[0090] In the Nth training cycle, the node will train the target neural network model multiple times, obtain a local gradient each time, and accumulate the local gradients obtained from each training to obtain the cumulative gradient.

[0091] For example, assuming that in the Nth training cycle, the node performs local training three times in the training process and obtains local gradients △A1, △A2 and △A3 respectively, then the cumulative gradient is: △A1+△A2+△A3.

[0092] Step S25: when the node performs training for the N+1th training cycle in the training process, the node synchronizes the accumulated gradient with other nodes that jointly train the target neural network model in the synchronization process, and calculates a first average gradient based on the accumulated gradients of the other nodes;

[0093] Before the synchronization process synchronizes the accumulated gradients with other nodes of the jointly trained target neural network model each time, the synchronization process will first determine that other nodes of the jointly trained target neural network model have also entered the synchronization process before starting the synchronization together.

[0094] The first average gradient refers to the average gradient calculated by the accumulated gradient obtained by synchronizing the Nth training cycle of each node in the cluster. The process of the training process executing the N+1th training cycle and the process of the synchronization process executing the accumulated gradient obtained by synchronizing the Nth training cycle are performed simultaneously.

[0095] Continuing with the above example, when the training process of the node reads the status information of the synchronization process as idle, it ends the Nth training cycle and starts the N+1th training cycle, and the accumulated gradient obtained by synchronization in the synchronization process is: △A1+△A2+△A3.

[0096] like Figure 3As shown, assuming that in addition to node A, the nodes in the cluster for training the target neural network model also include node B and node C, based on the same training method, node B obtains the cumulative gradient: △B1+△B2+△B3, node C obtains the cumulative gradient: △C1+△C2+△C3, nodes A, node B and node C synchronously accumulate gradients: △A1+△A2+△A3, cumulative gradients: △B1+△B2+△B3 and cumulative gradients: △C1+△C2+△C3, and obtain the first average gradient: △A1+△A2+△A3+△B1+△B2+△B3+△C1+△C2+△C3 / 3.

[0097] Step S26: After the training process completes the training of the N+1th training cycle, the node uses the first average gradient in the synchronization process to update the parameters of the target neural network model at the end of the Nth training cycle.

[0098] When the node is performing the N+1th training cycle in the training process, after synchronizing the accumulated gradients with other nodes of the common training target neural network model in the synchronization process, the step of assigning the synchronization state to the state information of the synchronization process through the synchronization process is executed, so that the training process completes the N+1th training cycle.

[0099] Taking node A as an example, after the training process of node A completes the first training cycle, its synchronization process starts to synchronize the accumulated gradient obtained in the first training cycle to obtain the average gradient. Since synchronizing the gradient takes time and the neural network is also in continuous training during this time, the average gradient is stored first. At the same time, the training process of node A trains the target neural network model for the first training cycle. After the training process of node A completes the first training cycle, the stored average gradient is used to update the parameters of the target neural network model before the start of the second training cycle. Since the target neural network model was not updated in the first training cycle, the parameters of the target neural network model before the start of the second training cycle are also the original parameters of the target neural network model.

[0100] The embodiment of the present application sets a synchronization process and a training process running in parallel at the node, so that the training process, based on the state information of the synchronization process, continuously repeats the local training task of the current cycle when the synchronization process synchronizes the gradient task, and updates the accumulated gradient with the gradient obtained by local training. When the synchronization process ends the task of synchronizing the gradient, the accumulated gradient is synchronized, and the parameters of the neural network model before the start of the current cycle are updated with the average gradient obtained by the last synchronization. In the above training process, the training process is always calculating while the node is synchronizing the gradient, making full use of the computing resources of the node. At the same time, because the node is always updating the accumulated gradient in the training process, after the end of the Nth training cycle, the accumulated gradient will be used to update the parameters of the target neural network model before the start of the Nth training cycle. The target neural network will not be affected by the gradient obtained by a single training in the Nth training cycle, thereby ensuring the timeliness of the target neural network parameters and further ensuring the convergence of the target neural network.

[0101] In another embodiment of the present application, a counter may be set in the training process to ensure the orderly training of the target neural network and the accuracy of the accumulated gradient in the training process during the Nth training cycle.

[0102] A counter is set in the training process; wherein the counter is used to record the number of times the target neural network model is trained in one cycle; and the synchronization process can communicate data through the memory and read the data of the counter in the training process.

[0103] The synchronization process and the training process are both local tasks for training execution. Based on this basic situation, the node determines the Nth training cycle of the training process according to the status information of the synchronization process, including the following steps:

[0104] Initializing the target neural network model and determining the first training cycle of the training process;

[0105] Take N=n+1, where n is an integer greater than 0, and execute the following steps: when the status information of the synchronization process is in an idle state, the node assigns the synchronization state to the status information of the synchronization process through the training process so that the synchronization process enters a synchronization state; at the same time, the node sets the cumulative number of training times recorded by the counter to zero, and determines that the training process ends the N-1th training cycle and starts the Nth training cycle.

[0106] Before starting to train the target neural network model, the node will initialize the training process and the synchronization process. After initializing the training process, the node initializes the parameters of the target neural network model, determines to start training the target neural network model in the first training cycle, and initializes the counter so that the counter starts counting from zero. The training process reads the status information of the synchronization process. When the training process starts training, the synchronization process is in an idle state. The training process assigns the synchronization state to the synchronization process so that the synchronization process starts to perform the synchronization task. At the same time, the training process starts the training in the second training cycle, and initializes the counter again so that the counter returns to zero. After the synchronization process ends the synchronization, the idle state is assigned to its own state information so that the training process can assign the synchronization state to the state information again to enter the second training cycle. The training process and the synchronization process continuously assign the state information so that the training process executes the first training cycle, the second training cycle, the second training cycle... until the training ends.

[0107] The embodiment of the present application utilizes the feature that the synchronization process and the training process can communicate through shared memory, and adopts a method of assigning a value to the status information of the synchronization process, so that when the synchronization process synchronizes the gradient, the training process executes the current Nth training cycle, and when the synchronization process ends the gradient synchronization, the next training starts, that is, the N+1th training cycle starts, and the cycle continues to determine the training cycle executed by the training process.

[0108] Another embodiment of the present application details a method for training a target neural network model to obtain local gradients and calculate cumulative gradients.

[0109] Take m from 1 to M in sequence, where M is an integer greater than 1, and perform the mth training on the target neural network model in the Nth training cycle to obtain the local gradient corresponding to the mth training; when the cumulative number of training times recorded by the counter is zero, the node clones the parameters of the target neural network model to the local using the training process to obtain the cloned model parameters; when the cumulative number of training times recorded by the counter is not zero, update the cloned model parameters using the local gradient corresponding to the mth training, and add one to the cumulative number of training times recorded by the counter until the status information of the synchronization process is assigned to the idle state.

[0110] In a training cycle, the parameters of the target neural network model are cloned only once, that is, after the training process assigns the synchronization state to the state information of the synchronization process and the counter of the training process returns to zero, the parameters of the target neural network model at this time are obtained and cloned locally.

[0111] In each training cycle, the number of training times is recorded by a counter so that the synchronization process can obtain the accurate state of the target neural network model for updating.

[0112] The four steps in the distributed training process of the neural network model trained by the SSGD algorithm are highly coupled. The existing technology cannot perform the training of the neural network model alone, nor can it perform the gradient synchronization between multiple nodes alone. The reasons are as follows: Assume that node 1 uses the model parameter W a Calculate the local gradient At this point, the cluster starts synchronizing It will be used after the synchronization is complete W a Update to W a1 However, since the cluster is still training during synchronization, when the above synchronization is completed, W a It may have been updated α times and become W a+α At this time, if the application gets the Update the current parameters of the neural network model, which is equivalent to model W a+α Used a Calculated Update to W (a+α)1 , obviously W (a+α) Not equal to W a1 , which seriously interferes with the convergence of the neural network model.

[0113] In response to the above problems, the embodiments of the present application clone the parameters of the target neural network model locally at the beginning of the Nth training cycle, so that the local gradients obtained from each training of the target neural network model in the Nth training cycle are used to update the parameters cloned locally, without affecting the parameters of the target neural network model itself. After the Nth training cycle is completed, the gradients of each node are synchronized to obtain the first average gradient, and then after the N+1th training cycle is completed, the first average gradient is used to update the parameters of the target neural network model.

[0114] After the training process finishes the Nth training cycle, the step of the synchronization process executing the gradient synchronization task includes: the node determines whether the current synchronization is executing the first synchronization gradient task, and if it is the first synchronization gradient task, directly synchronizes the accumulated gradient with other nodes. When the synchronization process enters the synchronization state for the first time, directly synchronizes the accumulated gradient with other nodes of the target neural network model to calculate the first average gradient;

[0115] If it is not the first synchronization gradient task, then in the synchronization process, the node obtains the parameters of the target neural network model at the end of the N-1th training cycle according to the cumulative number of training times recorded by the counter, and obtains the model parameters to be updated;

[0116] When the synchronization process is not the first time to enter the synchronization state, obtain a second average gradient; wherein the second average gradient is obtained by the cumulative gradient obtained by the node synchronization corresponding to the N-1th training cycle; update the model parameters to be updated using the second average gradient; synchronize the cumulative gradient with other nodes of the target neural network model to obtain the first average gradient

[0117] Assuming that in the Nth training cycle, the number of training times recorded by the counter is counter, then after the end of the Nth training cycle, the target neural network model has been trained e times, and the node obtains the parameters of the target neural network model at the e-counter training, that is, the parameters of the target neural network model before the start of the Nth training cycle, as the model parameters to be updated.

[0118] Assume that at the beginning of the Nth training cycle, when cloning the parameters of the target neural network model, the target neural network model is W N , because in the Nth training cycle, the clone model parameters are updated by local gradients, ensuring that the parameters of the target neural network model will not change, so at the end of the Nth training cycle, the target neural network model is still W N At this time, the synchronization process uses the second average gradient obtained in the previous synchronization to update the model parameters W of the target neural network model after the last training cycle. N-1 , get the model parameters W of the target neural network model before entering the N+1th training cycle N+1 After the N+1th training cycle, the synchronization process uses the training W N The first average gradient obtained updates the parameters W of the target neural network obtained at the end of the Nth training cycle N , to train W multiple times N The averaged cumulative gradient update W N , which solves the gradient expiration problem caused by nodes performing model training and gradient synchronization independently.

[0119] Another embodiment of the present application completes the calculation of the cumulative gradient in the Nth training cycle according to the number of training times recorded by the counter.

[0120] Sequentially taking m from 1 to M, where M is an integer greater than 1, the target neural network model is trained for the mth time in the Nth training cycle to obtain a local gradient corresponding to the mth training; when m is equal to 1, the local gradient corresponding to the mth training is stored locally to obtain a cumulative gradient corresponding to the mth training;

[0121] When m is not equal to 1, the local gradient corresponding to the m-th training is added to the cumulative gradient corresponding to the m-1-th training to obtain the cumulative gradient corresponding to the m-th training.

[0122] m can be the number of training times recorded by the counter - 1. When the number of training times is zero, the target neural network model is trained for the first time in the Nth training cycle, and the parameters of the current target neural network model are cloned to obtain the cloned model parameters that are updated to replace the target neural network model in the Nth training cycle. After the local gradient is obtained through training, it is stored locally.

[0123] When the number of training times is not zero, in the second training of the target neural network model in the Nth training cycle, after the local gradient is obtained through training, the clone model parameters obtained through the first training are updated using the local gradient, and the local gradient is accumulated to the local gradient obtained through the first training stored locally to obtain the cumulative gradient 1. In the third training, after the local gradient is obtained through training, the clone model parameters obtained through the first training are updated using the local gradient, and the local gradient is accumulated to the locally stored cumulative gradient 1 to obtain the cumulative gradient 2, and this cycle is repeated until the end of the Nth training cycle.

[0124] The embodiment of the present application ensures that the synchronization process can obtain the parameters of the target neural network model before the Nth training cycle starts based on the number of training times recorded by the counter after the Nth training cycle, and ensures that the accumulated gradients are accumulated in an orderly manner, providing a basis for obtaining an accurate first average gradient.

[0125] Figure 4 This is a flowchart of the distributed training of the neural network in the embodiment of the present application, refer to Figure 4 The process of executing the distributed training of the neural network in the embodiment of the present application is as follows:

[0126] First of all, the data that the synchronization process and the training process can communicate through shared memory include: the status information of the synchronization process, the cloned model parameters, the local gradient obtained by the i-th node in the m-th training in the N-th training cycle, and the accumulated gradient in the N-th training cycle. The training is performed after the Nth training cycle ends. The total number of local training times is counter.

[0127] With p t represents the training process, p s represents the synchronization process, W0 represents the initial parameters of the target neural network model, status represents the status information of the synchronization process, idling represents the idle state, and syncrhronizing represents the synchronization state. It represents the local gradient obtained by the node in the mth training in the Nth training cycle, replica represents the cloned model parameters, and e represents the number of local trainings performed by the target neural network model.

[0128] like Figure 4As shown, when status is in the syncrhronizing state, it indicates that p s The synchronization task is being executed; at this time, p t It will check whether the counter is 0. If the counter is 0, then p t The global parameters of the current neural network model will be cloned to the replica. Then, p t The gradient calculated using the current training target neural network model Update the cloned replica and accumulate arrive In the last p t Increase the counter by 1.

[0129] When the status is in the idling state, it indicates that p s Idle; at this time p t Command p s Start synchronization and overwrite status to syncrhronizing. s After synchronization starts, if there is a gradient from the previous synchronization, the gradient obtained from the previous synchronization is used to update the model W. e-counter , then p s Start syncing at the last sync time t Local training cumulative Finally, reset the counter to 0 and overwrite the status bit idling.

[0130] The embodiment of the present application proposes that in the Nth training cycle, the node can use the following formula to update the target neural network model in the synchronization process:

[0131]

[0132] Where μ represents the learning rate of the target neural network model, n represents the number of nodes in the cluster, e1 and e2 represent that in the Nth training cycle, the node has been trained locally from e1 to e2 times;

[0133] represents the parameters of the target neural network model when the i-th node is trained for the e1th time; Represents the local gradient obtained by training the target neural network model for the eth time at the i-th node in the N-th training cycle.

[0134] Based on the same inventive concept, an embodiment of the present application provides a neural network distributed training device. Figure 5 Schematic diagram of the structure of the neural network distributed training device proposed in the embodiment of the present application. Figure 5 As shown, the device comprises:

[0135] A process establishment module 51 is used to establish a synchronization process and a training process running in parallel with the synchronization process for each node in the cluster;

[0136] A training cycle determination module 52, configured to enable the node to determine the Nth training cycle of the training process according to the state information of the synchronization process, where N is an integer greater than 0;

[0137] A training module 53, configured to, during the Nth training cycle, train the target neural network model in the training process by the node to obtain a local gradient;

[0138] An accumulation module 54 is used to enable the node to accumulate local gradients obtained by training the target neural network model multiple times to obtain a cumulative gradient;

[0139] A synchronization module 55 is used to synchronize the accumulated gradient with other nodes that jointly train the target neural network model in the synchronization process when the node performs the training for the N+1th training cycle in the training process, and calculate a first average gradient based on the accumulated gradients of the other nodes;

[0140] An updating module 56 is used for, after the training process completes the training of the N+1th training cycle, the node uses the first average gradient in the synchronization process to update the parameters of the target neural network model at the end of the Nth training cycle.

[0141] Optionally, the device further comprises:

[0142] A setting module, used to set a counter in the training process; wherein the counter is used to record the number of times the target neural network model is trained in one cycle;

[0143] The training cycle determination module comprises:

[0144] An initialization submodule, used to initialize the target neural network model and determine the first training cycle of the training process;

[0145] The training cycle determination submodule is used to take N=n+1, where n is an integer greater than 0, and execute the following steps: when the status information of the synchronization process is in an idle state, the node assigns the synchronization state to the status information of the synchronization process through the training process so that the synchronization process enters a synchronization state; at the same time, the node sets the cumulative number of training times recorded by the counter to zero, and determines that the training process ends the N-1th training cycle and starts the Nth training cycle.

[0146] Optionally, the training module includes:

[0147] The first training submodule is used to sequentially take m from 1 to M, where M is an integer greater than 1, perform an m-th training on the target neural network model in the N-th training cycle, and obtain a local gradient corresponding to the m-th training;

[0148] A cloning submodule, configured to, when the cumulative number of training times recorded by the counter is zero, cause the node to clone the parameters of the target neural network model locally using the training process to obtain cloned model parameters;

[0149] The first updating submodule is used to update the clone model parameters using the local gradient corresponding to the mth training when the cumulative number of training times recorded by the counter is not zero, and add one to the cumulative number of training times recorded by the counter until the state information of the synchronization process is assigned an idle state.

[0150] Optionally, the second training submodule is used to sequentially take m from 1 to M, where M is an integer greater than 1, perform an m-th training on the target neural network model in the N-th training cycle, and obtain a local gradient corresponding to the m-th training;

[0151] A first cumulative gradient, used for storing the local gradient corresponding to the m-th training locally when m is equal to 1, to obtain the cumulative gradient corresponding to the m-th training;

[0152] The second accumulated gradient is used for accumulating the local gradient corresponding to the m-th training to the accumulated gradient corresponding to the m-1-th training to obtain the accumulated gradient corresponding to the m-th training when m is not equal to 1.

[0153] Optionally, the device further comprises:

[0154] A parameter acquisition module, used for, in the synchronization process, the node to acquire the parameters of the target neural network model at the end of the N-1th training cycle according to the cumulative number of training times recorded by the counter, and obtain the model parameters to be updated;

[0155] Optionally, the synchronization module includes:

[0156] A first synchronization submodule, configured to directly synchronize the accumulated gradient with other nodes of the target neural network model and calculate a first average gradient when the synchronization process enters the synchronization state for the first time;

[0157] A gradient acquisition submodule, used for acquiring a second average gradient when the synchronization process does not enter the synchronization state for the first time; wherein the second average gradient is obtained by the accumulated gradient obtained by the node synchronization corresponding to the N-1th training cycle;

[0158] A second updating submodule, used for updating the model parameters to be updated using the second average gradient;

[0159] The first synchronization submodule is used to synchronize the accumulated gradient with other nodes of the target neural network model to obtain a first average gradient.

[0160] Optionally, the device further comprises:

[0161] An assignment module is used to assign the synchronization state to the state information of the synchronization process through the synchronization process, so that the training process completes the N+1th training cycle.

[0162] Based on the same inventive concept, another embodiment of the present application provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the neural network distributed training method as described in any of the above embodiments of the present application.

[0163] Based on the same inventive concept, another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the steps in the neural network distributed training method described in any of the above embodiments of the present application.

[0164] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0165] Each embodiment in this specification is described in a progressive or illustrative manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referenced to each other.

[0166] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, devices, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0170] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0171] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0172] The above is a detailed introduction to a neural network distributed training method, device, equipment and storage medium provided by the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application; at the same time, for a person skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A neural network distributed training method, characterized in that: The method comprises: For each node in the cluster, establish a synchronization process and a training process running in parallel with the synchronization process; The node determines the Nth training cycle of the training process according to the state information of the synchronization process, where N is an integer greater than 0; In the Nth training cycle, the node trains the target neural network model in the training process to obtain a local gradient; The node accumulates local gradients obtained by training the target neural network model multiple times to obtain a cumulative gradient; When the node performs training for the N+1th training cycle in the training process, the node synchronizes the accumulated gradient with other nodes that jointly train the target neural network model in the synchronization process, and calculates a first average gradient based on the accumulated gradients of the other nodes; After the training process completes the training of the N+1th training cycle, the node updates the parameters of the target neural network model at the end of the Nth training cycle using the first average gradient in the synchronization process; The method further comprises: A counter is set in the training process; wherein the counter is used to record the number of times the target neural network model is trained in one cycle; The node determines, according to the state information of the synchronization process, an Nth training cycle of the training process, including: Initializing the target neural network model and determining the first training cycle of the training process; Take N=n+1, where n is an integer greater than 0, and execute the following steps: when the state information of the synchronization process is in an idle state, the node assigns the synchronization state to the state information of the synchronization process through the training process, so that the synchronization process enters a synchronization state; at the same time, the node sets the accumulated number of training times recorded by the counter to zero, and determines that the training process ends the N-1th training cycle and starts the Nth training cycle; When the state information of the synchronization process is in an idle state, the node assigns a synchronization state to the state information of the synchronization process through the training process so that the synchronization process enters a synchronization state, and the method further includes: In the synchronization process, the node obtains the parameters of the target neural network model at the end of the N-1th training cycle according to the accumulated number of training times recorded by the counter, and obtains the model parameters to be updated; The node synchronizes the accumulated gradient with other nodes that jointly train the target neural network model in the synchronization process, and calculates and obtains a first average gradient based on the accumulated gradient of other nodes, including: When the synchronization process enters the synchronization state for the first time, directly synchronizing the accumulated gradient with other nodes of the target neural network model to calculate a first average gradient; When the synchronization process does not enter the synchronization state for the first time, a second average gradient is obtained; wherein the second average gradient is obtained by the accumulated gradient obtained by the node synchronization corresponding to the N-1th training cycle; Updating the model parameters to be updated using the second average gradient; The accumulated gradient is synchronized with other nodes of the target neural network model to obtain a first average gradient.

2. The method according to claim 1, characterized in that In the Nth training cycle, the node trains the target neural network model in the training process to obtain a local gradient, including: Sequentially taking m from 1 to M, where M is an integer greater than 1, performing an m-th training on the target neural network model in the N-th training cycle, and obtaining a local gradient corresponding to the m-th training; When the cumulative number of training times recorded by the counter is zero, the node clones the parameters of the target neural network model locally using the training process to obtain cloned model parameters; When the cumulative number of training times recorded by the counter is not zero, the clone model parameters are updated using the local gradient corresponding to the mth training, and the cumulative number of training times recorded by the counter is increased by one until the state information of the synchronization process is assigned an idle state.

3. The method according to claim 1, characterized in that The node accumulates local gradients obtained by training the target neural network model multiple times to obtain a cumulative gradient, including: Sequentially taking m from 1 to M, where M is an integer greater than 1, performing an m-th training on the target neural network model in the N-th training cycle, and obtaining a local gradient corresponding to the m-th training; When m is equal to 1, the local gradient corresponding to the m-th training is stored locally to obtain the cumulative gradient corresponding to the m-th training; When m is not equal to 1, the local gradient corresponding to the m-th training is added to the cumulative gradient corresponding to the m-1-th training to obtain the cumulative gradient corresponding to the m-th training.

4. The method according to claim 1, characterized in that: When the node performs training for the N+1th training cycle in the training process, after synchronizing the accumulated gradient with other nodes that jointly train the target neural network model in the synchronization process, the method further includes: The synchronization process assigns the synchronization state to the state information of the synchronization process, so that the training process completes the N+1th training cycle.

5. A neural network distributed training device, characterized in that: The device comprises: A process establishment module, used to establish a synchronization process and a training process running in parallel with the synchronization process for each node in the cluster; A training cycle determination module, used to enable the node to determine the Nth training cycle of the training process according to the state information of the synchronization process, where N is an integer greater than 0; A training module, configured to, during the Nth training cycle, train the target neural network model in the training process by the node to obtain a local gradient; An accumulation module, used to enable the node to accumulate local gradients obtained by training the target neural network model multiple times to obtain a cumulative gradient; A synchronization module, configured to synchronize the accumulated gradient with other nodes that jointly train the target neural network model in the synchronization process when the node performs the training for the N+1th training cycle in the training process, and calculate a first average gradient based on the accumulated gradients of the other nodes; An updating module, configured to update, by the node in the synchronization process, the parameters of the target neural network model at the end of the Nth training cycle using the first average gradient after the training process completes the training of the N+1th training cycle; The device also includes: A setting module, used to set a counter in the training process; wherein the counter is used to record the number of times the target neural network model is trained in one cycle; The training cycle determination module comprises: An initialization submodule, used to initialize the target neural network model and determine the first training cycle of the training process; The training cycle determination submodule is used to take N=n+1, where n is an integer greater than 0, and execute the following steps: when the state information of the synchronization process is in an idle state, the node assigns the synchronization state to the state information of the synchronization process through the training process, so that the synchronization process enters a synchronization state; at the same time, the node sets the accumulated number of training times recorded by the counter to zero, and determines that the training process ends the N-1th training cycle and starts the Nth training cycle; A parameter acquisition module, used for, in the synchronization process, the node to acquire the parameters of the target neural network model at the end of the N-1th training cycle according to the cumulative number of training times recorded by the counter, and obtain the model parameters to be updated; The synchronization module comprises: A first synchronization submodule, configured to directly synchronize the accumulated gradient with other nodes of the target neural network model and calculate a first average gradient when the synchronization process enters the synchronization state for the first time; A gradient acquisition submodule, used for acquiring a second average gradient when the synchronization process does not enter the synchronization state for the first time; wherein the second average gradient is obtained by the accumulated gradient obtained by the node synchronization corresponding to the N-1th training cycle; A second updating submodule, used for updating the model parameters to be updated using the second average gradient; The first synchronization submodule is used to synchronize the accumulated gradient with other nodes of the target neural network model to obtain a first average gradient.

6. A readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in any one of the methods described in claims 1 to 4 are implemented.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When executed, the processor implements the steps of the method according to any one of claims 1 to 4.

Images