Gradient synchronization method in distributed training and distributed training system

Abstract

The embodiments of the present specification provide a gradient synchronization method in distributed training and a distributed training system, wherein the gradient synchronization method in distributed training includes: inputting first training data into a neural network model to be trained for training; monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained within a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point; sending the gradient set to a communication library pre-configured in a current training node, wherein the communication library is used to communicate with other training nodes in a distributed training cluster; and obtaining gradient sets in other training nodes in the distributed training cluster through the communication library.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a gradient synchronization method in distributed training. One or more embodiments of this specification also relate to two gradient synchronization methods in distributed training, three distributed training systems, two computing devices, and two computer-readable storage media. Background Art

[0002] At present, with the rapid development of computer technology, deep learning technology has also made rapid progress. With the deepening of deep learning technology, more and more complex algorithms have been developed. These algorithms require a large amount of data and take a lot of time to effectively complete training. Therefore, distributed training has been developed.

[0003] In the distributed training scenario of deep learning, neural network models are mostly trained using the synchronous update training method in data parallelism. In this training method, the model parameters in each training node are consistent. After the gradient on a training node is calculated, it is necessary to wait for the corresponding gradient on other training nodes to be calculated, and then negotiate the order of gradient synchronization between the parameter server and the training node or between the training nodes. Gradient synchronization can only be performed after the negotiation is completed. In the process of waiting for the corresponding gradient on other training nodes and negotiating the gradient synchronization order, the network bandwidth is wasted in the waiting process, which causes the training time of the neural network model to be prolonged, seriously slowing down the training speed of the neural network model.

[0004] Therefore, how to solve the above problems has become a problem that needs to be solved urgently by technical personnel. Summary of the invention

[0005] In view of this, the present specification provides a method for synchronizing gradients in distributed training. One or more embodiments of the present specification simultaneously involve two methods for synchronizing gradients in distributed training, three distributed training systems, two computing devices, and two computer-readable storage media to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a gradient synchronization method in distributed training is provided, including:

[0007] Inputting the first training data into the neural network model to be trained for training;

[0008] Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point;

[0009] Sending the gradient set to a communication library pre-configured in the current training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster;

[0010] The gradient sets in other training nodes in the distributed training cluster are obtained through the communication library.

[0011] Optionally, after obtaining the gradient sets in other training nodes in the distributed training cluster through the communication library, the method further includes:

[0012] The parameters of the neural network model to be trained in the current training node are updated according to the gradient set of each training node in the distributed training cluster.

[0013] Optionally, updating the parameters of the neural network model to be trained in the current training node according to the gradient set of each training node in the distributed training cluster includes:

[0014] Accumulating the gradients in the gradient set of each training node in the distributed training cluster to obtain an accumulated gradient;

[0015] Calculating an average gradient based on the accumulated gradient and the number of training nodes in the distributed training cluster;

[0016] Calculating the loss value of the neural network model to be trained according to the average gradient;

[0017] Update the parameters of the neural network model to be trained according to the loss value.

[0018] Optionally, the time point sequence includes n time points, where n>1;

[0019] The training time is monitored, and when the training time is the same as the first time point in the preset time point sequence, a gradient set generated by the neural network model to be trained in the first time interval is obtained, including:

[0020] The training time is monitored, and when the training time is the same as the i-th time point, the gradients generated between the i-1-th time point and the i-th time point are packaged to generate a gradient set, where 2<=i<=n;

[0021] Increment i by 1 and determine whether i is greater than n. If so, end the gradient synchronization. If not, continue to perform the operation when the training time is the same as the i-th time point.

[0022] Optionally, the preset time point sequence is generated by the following steps, including:

[0023] Receiving second training data, and inputting the second training data into the neural network model to be trained for training;

[0024] Obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient;

[0025] Clustering multiple gradients of the neural network model to be trained according to the generation time of each gradient;

[0026] Determine the time point sequence based on the clustering results.

[0027] Optionally, a time point sequence is determined based on the clustering result, including:

[0028] Determine the start time and end time corresponding to the current category according to the generation time of the gradient in each category of the clustering result;

[0029] Determine the time interval between adjacent categories according to the start time and end time of each category of the clustering result;

[0030] A time point sequence is produced according to the time interval.

[0031] Optionally, after determining the time interval between adjacent categories according to the start time and end time of each category of the clustering result, the method further includes:

[0032] When the time interval is greater than a preset threshold, receiving third training data, and inputting the third training data into the neural network model to be trained for training;

[0033] Obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient;

[0034] Clustering multiple gradients of the neural network model to be trained according to the generation time of each gradient;

[0035] Determine the time point sequence based on the clustering results.

[0036] According to a second aspect of an embodiment of this specification, a method for synchronizing gradients in distributed training is provided, the method being applied to a distributed training cluster of a neural network model to be trained, the distributed training cluster comprising at least one parameter server and at least two training nodes, the method comprising:

[0037] Each of the training nodes inputs the first training data into the neural network model to be trained for training;

[0038] Each of the training nodes monitors the training time, and when the training time is the same as the first time point in the preset time point sequence, obtains a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point;

[0039] Each of the training nodes sends the gradient set to the parameter server;

[0040] The parameter server receives a gradient set sent by each training node.

[0041] Optionally, after the parameter server receives the gradient set sent by each training node, the method further includes:

[0042] The parameter server updates the parameters of the neural network model to be trained according to the gradient set sent by each training node, and sends the updated parameters to each training node;

[0043] Each of the training nodes receives the updated parameters sent by the parameter server, and updates the parameters of the neural network model to be trained in the current training node.

[0044] Optionally, the parameter server updates the parameters of the neural network model to be trained according to the gradient set sent by each training node, including:

[0045] Accumulating the gradients in the gradient set sent by each training node to obtain an accumulated gradient;

[0046] Calculating an average gradient based on the accumulated gradient and the number of training nodes in the distributed training cluster;

[0047] Calculating the loss value of the neural network model to be trained according to the average gradient;

[0048] Update the parameters of the neural network model to be trained according to the loss value.

[0049] Optionally, the time point sequence includes n time points, where n>1;

[0050] Each of the training nodes monitors the training time, and when the training time is the same as the first time point in the preset time point sequence, obtains a gradient set generated by the neural network model to be trained in the first time interval, including:

[0051] The training time of each training node is monitored, and when the training time is the same as the i-th time point, the gradients generated between the i-1-th time point and the i-th time point are packaged to generate a gradient set, where 2<=i<=n;

[0052] Increment i by 1 and determine whether i is greater than n. If so, end the gradient synchronization. If not, continue to perform the operation when the training time is the same as the i-th time point.

[0053] Optionally, the preset time point sequence is generated by the following steps and applied to each of the training nodes, including:

[0054] Receiving second training data, and inputting the second training data into the neural network model to be trained for training;

[0055] Obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient;

[0056] Clustering multiple gradients of the neural network model to be trained according to the generation time of each gradient;

[0057] Determine the time point sequence based on the clustering results.

[0058] Optionally, a time point sequence is determined based on the clustering result, including:

[0059] Determine the start time and end time corresponding to the current category according to the generation time of the gradient in each category of the clustering result;

[0060] Determine the time interval between adjacent categories according to the start time and end time of each category of the clustering result;

[0061] A time point sequence is produced according to the time interval.

[0062] Optionally, after determining the time interval between adjacent categories according to the start time and end time of each category of the clustering result, the method further includes:

[0063] When the time interval is greater than a preset threshold, receiving third training data, and inputting the third training data into the neural network model to be trained for training;

[0064] Obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient;

[0065] Clustering multiple gradients of the neural network model to be trained according to the generation time of each gradient;

[0066] Determine the time point sequence based on the clustering results.

[0067] According to a third aspect of an embodiment of this specification, a method for synchronizing gradients in distributed training is provided, which is applied to each training node in a distributed training cluster, and the method includes:

[0068] Inputting the first training data into the neural network model to be trained for training;

[0069] Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point;

[0070] The gradient set is sent to a parameter server in the distributed training cluster.

[0071] According to a fourth aspect of an embodiment of this specification, there is provided a distributed training system, the distributed training system comprising at least three training nodes and the training nodes being connected in a ring shape;

[0072] Each of the training nodes is configured to input the first training data into the neural network model to be trained for training; monitor the training time, and when the training time is the same as the first time point in a preset time point sequence, obtain the gradient set generated by the neural network model to be trained within the first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point; send the gradient set to a communication library pre-configured in the current training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster; obtain the gradient sets in other training nodes in the distributed training cluster through the communication library.

[0073] According to a fifth aspect of an embodiment of this specification, a distributed training system is provided, the distributed training system comprising at least one parameter server and at least two training nodes, each of the training nodes being connected to the parameter server respectively;

[0074] Each of the training nodes is configured to input the first training data into the neural network model to be trained for training; monitor the training time, and when the training time is the same as the first time point in the preset time point sequence, obtain the gradient set generated by the neural network model to be trained in the first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point; send the gradient set to the parameter server;

[0075] The parameter server is configured to receive a gradient set sent by each training node.

[0076] According to a sixth aspect of the embodiments of this specification, a distributed training system is provided, wherein the distributed training system includes at least two training nodes.

[0077] Each of the training nodes is configured to input the first training data into the neural network model to be trained for training; monitor the training time, and when the training time is the same as the first time point in a preset time point sequence, obtain the gradient set generated by the neural network model to be trained within the first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point; and send the gradient set to the parameter server in the distributed training cluster.

[0078] According to a seventh aspect of an embodiment of this specification, there is provided a computing device, including:

[0079] Memory and processor;

[0080] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions:

[0081] Inputting the first training data into the neural network model to be trained for training;

[0082] Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point;

[0083] Sending the gradient set to a communication library pre-configured in the current training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster;

[0084] The gradient sets in other training nodes in the distributed training cluster are obtained through the communication library.

[0085] According to an eighth aspect of the embodiments of this specification, a computing device is provided, including:

[0086] Memory and processor;

[0087] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions:

[0088] Inputting the first training data into the neural network model to be trained for training;

[0089] Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point;

[0090] The gradient set is sent to a parameter server in the distributed training cluster.

[0091] According to a ninth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of a gradient synchronization method in distributed training provided in the first aspect above are implemented.

[0092] According to a tenth aspect of an embodiment of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of a gradient synchronization method in distributed training provided in the third aspect above are implemented.

[0093] An embodiment of the present specification implements a gradient synchronization method in distributed training, wherein first training data is input into a neural network model to be trained for training; the training time is monitored, and when the training time is the same as a first time point in a preset time point sequence, a gradient set generated by the neural network model to be trained in the first time interval is obtained; through a preset time point sequence, gradient synchronization is triggered in each training node according to a time point in a preset time point sequence, thereby saving time for negotiating synchronized gradients and improving the training efficiency of the neural network model; during gradient synchronization, the gradient set generated between two adjacent time points is synchronized between the training nodes in the distributed training cluster, and multiple gradients can be synchronized at one time, thereby saving time for frequent gradient synchronization and further improving the training efficiency of the neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0094] Figure 1 is a processing flow chart of a gradient synchronization method in distributed training provided in the first embodiment of this specification;

[0095] Figure 2 is a gradient time distribution diagram provided by an embodiment of this specification;

[0096] Figure 3 is a schematic diagram of a preset time point sequence provided by an embodiment of this specification;

[0097] Figure 4 It is a schematic diagram of the structure of the distributed training cluster provided in the first embodiment of this specification;

[0098] Figure 5is a processing flow chart of a gradient synchronization method in distributed training provided in the second embodiment of this specification;

[0099] Figure 6 is a schematic diagram of gradient clustering provided by an embodiment of this specification;

[0100] Figure 7 is a processing flow chart of a gradient synchronization method in distributed training provided in the third embodiment of this specification;

[0101] Figure 8 is a schematic diagram of the structure of a distributed training cluster provided in the third embodiment of this specification;

[0102] Fig. 9 is a processing flow chart of a gradient synchronization method in distributed training provided by the fourth embodiment of this specification;

[0103] Fig.10 is a processing flow chart of a gradient synchronization method in distributed training provided in the fifth embodiment of this specification;

[0104] Fig.11 is a schematic diagram of a distributed training system provided by an embodiment of this specification;

[0105] Fig.12 is a schematic diagram of a distributed training system provided by yet another embodiment of the present specification;

[0106] Fig.13 is a structural block diagram of a computing device provided by one embodiment of this specification;

[0107] Fig.14 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0108] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0109] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0110] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0111] First, the terms involved in one or more embodiments of this specification are explained.

[0112] Gradient: Gradient is a vector that indicates that the directional derivative of a function at that point has the maximum value along that direction, that is, the function changes fastest along that direction (the direction of this gradient) at that point, and the rate of change is the largest. In model training, gradient is used to find the minimum loss function, train the model, and accelerate the convergence of the model. The number of model training times is the number of gradient steps.

[0113] Distributed training: A method of training using multiple training nodes.

[0114] Clustering: The process of grouping a collection of physical or abstract objects into clusters of similar objects.

[0115] In this specification, a gradient synchronization method in distributed training is provided. This specification also involves two gradient synchronization methods in distributed training, three distributed training systems, two computing devices, and two computer-readable storage media, which are described in detail one by one in the following embodiments.

[0116] Figure 1 A processing flow chart of a gradient synchronization method in distributed training provided according to the first embodiment of this specification is shown. The distributed training cluster in the gradient synchronization method in distributed training provided in this embodiment is a Ring-allreduce architecture. In the Ring-allreduce architecture, at least 3 training nodes are included, and the at least 3 training nodes are connected in a ring.

[0117] In the distributed training scenario of the Ring-allreduce architecture, a synchronous update training method in data parallelism is adopted. Under this training method, the model parameters in each training node are consistent, and the time error of each training node in generating the corresponding gradient during the training process is small.

[0118] like Figure 1 As shown, the gradient synchronization method in distributed training includes the following steps 102 to 108.

[0119] Step 102: Input the first training data into the neural network model to be trained for training.

[0120] The first training data is a training batch of training data for the neural network model to be trained. The neural network model to be trained requires multiple first training data for training. In a distributed training scenario, the first training data of each training node can be the same or different. The first training data is input into the neural network model to be trained of the training node for training.

[0121] Step 104: Monitor the training time, and when the training time is the same as the first time point in a preset time point sequence, obtain a gradient set generated by the neural network model to be trained within a first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point.

[0122] The preset time point sequence includes n time points, where n>1. In the embodiment provided in the specification, the first time point is the time point in the time point sequence that is the same as the training time. The first time point is the time point in the time point sequence excluding the first time point. The first time point in the time point sequence is 0, which is the training start time. Therefore, the first time point does not need to trigger the gradient synchronization operation. The first time interval is the time interval between the previous time point of the first time point and the first time point.

[0123] The training time is monitored. When the training time is the same as the i-th time point in the time point sequence, where 2<=i<=n, the i-th time point is the first time point, and the time interval between the i-1-th time point and the i-th time point is the first time interval, the gradient set generated by the neural network model to be trained between the i-1-th time point and the i-th time point is obtained.

[0124] Increment i by 1 and determine whether i is greater than n. If so, end the gradient synchronization. If not, continue to perform the operation when the training time is the same as the i-th time point.

[0125] For example, if the training time is 5 milliseconds and the second time point is 5 milliseconds, that is, the training time is the same as the second time point, then the second time point is the first time point, and the time interval between the first time point and the second time point is the first time interval. At this time, the gradient set generated by the neural network model to be trained between the first time point and the second time point is obtained.

[0126] In the first embodiment provided in this specification, combined with Figure 2 and Figure 3A gradient synchronization method in distributed training provided in the first embodiment of this specification is explained in detail. Figure 2 The gradient time distribution diagram of the gradient generated during a training process is shown as Figure 2 As shown in the figure, the horizontal axis indicates the number of gradients generated in one training, which is 21, and the vertical axis indicates the time corresponding to the generation of each gradient, and the time unit is milliseconds; Figure 3 A schematic diagram of a preset time point sequence is shown. Figure 3 As shown, the time point sequence includes 8 time points, namely 0 milliseconds, 4 milliseconds, 8 milliseconds, 12 milliseconds, 17 milliseconds, 25 milliseconds, 29 milliseconds and 32 milliseconds.

[0127] Taking the training time of 4 milliseconds as an example, Figure 3 It can be seen that the training time is the same as the second time point in the time point sequence. In this case, the second time point is the first time point, and the time interval between the first time point and the second time point is the first time interval. Figure 2 It can be seen that the gradient set generated in the first time interval at this time is gradients 1-7. Gradients 1-7 are obtained. At this time, i=2, and i is incremented by 1 to 3. At this time, 3 is less than the number of time points in the time point sequence 8. Then continue to monitor the training time and perform the operation when the training time is the same as the third time point.

[0128] Taking the training time of 17 milliseconds as an example, Figure 3 It can be seen that the training time is the same as the fifth time point in the time point sequence. At this time, the fifth time point is the first time point, and the time interval between the fourth time point and the fifth time point is the first time interval. Figure 2 It can be seen that the gradient set generated in the first time interval at this time is gradients No. 12-13, and gradients No. 12-13 are obtained.

[0129] Step 106: Send the gradient set to a communication library pre-configured in the current training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster.

[0130] The communication library is a communication library pre-configured on each training node for communication between training nodes, such as Nvidia Collective multi-GPU Communication Library (NCCL), Message Passing Interface (MPI), etc. The gradient set is sent to the communication library of the current training node, and the communication libraries on other training nodes in the distributed training cluster can obtain the gradient set of the current training node. At the same time, the communication library of the current training node can also obtain the gradient set on other training nodes.

[0131] In the first embodiment provided in this specification, see Figure 4 , Figure 4 The structure diagram of the distributed training cluster based on the Ring-allreduce architecture is shown in FIG. Figure 4 As shown, the distributed training cluster includes three training nodes, which are connected in a ring shape. Each training node is configured with a communication library. For training node 1, the gradient set 1 generated in training node 1 is sent to the communication library 1 of training node 1. For training node 2, the gradient set 2 generated in training node 2 is sent to the communication library 2 of training node 2. For training node 3, the gradient set 3 generated in training node 3 is sent to the communication library 3 of training node 3.

[0132] Step 108: Obtain the gradient sets in other training nodes in the distributed training cluster through the communication library.

[0133] Obtain the gradient sets of other training nodes through the communication library to complete the gradient synchronization at the first time point.

[0134] In the first embodiment provided in this specification, training node 1 obtains gradient set 2 in training node 2 and gradient set 3 in training node 3 through communication library 1, training node 2 obtains gradient set 3 in training node 3 and gradient set 1 in training node 1 through communication library 2, and training node 3 obtains gradient set 1 in training node 1 and gradient set 2 in training node 2 through communication library 3, thereby completing the gradient synchronization at the first time point.

[0135] As the training time goes by, whenever the training time is the same as the first time point in the time point sequence, the above gradient synchronization operation is performed until the training time is the same as the last time point in the time point sequence, and the last gradient synchronization operation is completed.

[0136] The first embodiment of the present specification provides a gradient synchronization method in distributed training, which is applied to a distributed training cluster based on a Ring-allreduce architecture. The training time is monitored in each training node, and the gradient synchronization operation is triggered when the training time is the same as the first time point in a preset time point sequence, thereby saving the time for negotiating synchronized gradients and the time for waiting for corresponding gradients in other training nodes, thereby improving the training efficiency of the neural network model. During the gradient synchronization process, the gradients generated between two adjacent time points are packaged to generate a gradient set, and then the gradient set is synchronized between the training nodes in the distributed training cluster. Multiple gradients can be synchronized at one time, thereby saving the time for frequent gradient synchronization, thereby further improving the training efficiency of the neural network model.

[0137] Figure 5A processing flow chart of a gradient synchronization method in distributed training provided according to the second embodiment of this specification is shown. The distributed training cluster in the gradient synchronization method in distributed training provided in this embodiment is a Ring-allreduce architecture. In the Ring-allreduce architecture, at least 3 training nodes are included, and the at least 3 training nodes are connected in a ring.

[0138] The gradient synchronization method in distributed training includes the following steps 502 to 510.

[0139] Step 502: Input the first training data into the neural network model to be trained for training.

[0140] Step 504: Monitor the training time, and when the training time is the same as the first time point in a preset time point sequence, obtain a gradient set generated by the neural network model to be trained within a first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point.

[0141] Step 506: Send the gradient set to a communication library pre-configured in the current training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster.

[0142] Step 508: Obtain the gradient sets in other training nodes in the distributed training cluster through the communication library.

[0143] Steps 502 to 508 are consistent with the methods of steps 102 to 108 described above. For specific explanations of steps 502 to 508, please refer to the detailed contents of steps 102 to 108 in the aforementioned embodiment, which will not be repeated here.

[0144] Step 510: Update the parameters of the neural network model to be trained in the current training node according to the gradient set of each training node in the distributed training cluster.

[0145] Each training node obtains the gradient sets in other training nodes and the gradient set of the current training node, and updates the parameters of the neural network model to be trained in the current training node. Since the gradient sets on each training node are the same, it can be ensured that the updated parameters of the neural network model to be trained are also the same.

[0146] Optionally, step 510 can be implemented through the following steps S5102 to S5108.

[0147] S5102: Accumulate the gradients in the gradient set of each training node in the distributed training cluster to obtain an accumulated gradient.

[0148] In the second embodiment provided in this specification, see Figure 2 , Figure 3 and Figure 4 ,by Figure 3 Take the third time point shown as an example, and then combine Figure 2 , at this point in time, Figure 4 The gradient in the gradient set of training node 1 is T 108 , the gradient in the gradient set in training node 2 is T 208 , the gradient in the gradient set in training node 2 is T 308 The accumulated gradient is T after accumulating the gradients in the gradient set. 108 +T 208 +T 308 .

[0149] S5104. Calculate an average gradient according to the accumulated gradient and the number of training nodes in the distributed training cluster.

[0150] In the second embodiment provided in this specification, the number of training nodes in the distributed training cluster is 3, so the average gradient T 008 =(T 108 +T 308 +T 308 ) / 3.

[0151] S5106. Calculate the loss value of the neural network model to be trained based on the average gradient.

[0152] S5108. Update the parameters of the neural network model to be trained according to the loss value.

[0153] Because the average gradient in each training node of the distributed training cluster is the same, the loss value of the neural network model to be trained calculated is also the same, and then the parameters of the neural network model to be trained in the current training node updated according to the loss value are also the same, thereby ensuring that the parameters of the neural network model to be trained on each training node are consistent.

[0154] Optionally, a second embodiment provided in this specification further provides a method for generating a preset time point sequence, including steps S5122 to S5128.

[0155] S5122. Receive second training data, and input the second training data into the neural network model to be trained for training.

[0156] S5124. Obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient.

[0157] In the examples provided in this specification, see Figure 2 , based on the obtained multiple gradients and the generation time of each gradient, a gradient time distribution graph is generated. The horizontal axis in the graph indicates that there are 21 gradients generated in one training, and the vertical axis indicates the time corresponding to the generation of each gradient, and the time unit is milliseconds.

[0158] S5126. Clustering multiple gradients of the neural network model to be trained according to the generation time of each gradient.

[0159] Specifically, clustering is the process of dividing a collection of physical or abstract objects into multiple classes consisting of similar objects. There are many clustering methods, such as k-means clustering algorithm (k-means), mean shift method, etc. In practical applications, Bayesian optimization methods can also be used to optimize clustering methods.

[0160] In the examples provided in this specification, see Figure 6 , Figure 6 A schematic diagram of gradient clustering is shown, in which 21 generated gradients are clustered according to the generation time of each gradient to obtain 7 categories.

[0161] S5128. Determine a time point sequence according to the clustering results.

[0162] In practical applications, there are many ways to determine the time point sequence based on the clustering results, such as determining the time point sequence based on the time interval between two adjacent categories in the clustering results, determining the time point sequence based on a preset time correspondence table, etc. In this specification, the specific method for determining the time point sequence is not limited.

[0163] Optionally, the start time and end time corresponding to the current category are determined according to the generation time of the gradient in each category of the clustering result; the time interval between adjacent categories is determined according to the start time and end time of each category of the clustering result; and a time point sequence is produced according to the time interval.

[0164] In the embodiments provided in this specification, an example of determining a time point in a time point sequence according to an average value of a time interval is used for explanation. Referring to Table 1 below, according to the generation time of the gradients in the seven categories, the start time of the first category is determined to be 0 ms, and the end time is 3 ms. The start time of the second category is 5 ms, and the end time is 5 ms. By analogy, the start time of the seventh category is 30 ms, and the end time is 30 ms.

[0165] According to the start time and end time of each category, it can be determined that the time interval between the 1st category and the 2nd category is 2ms, and the average time interval is 1ms; the time interval between the 2nd category and the 3rd category is 5ms, and the time interval at this time is 2.5ms. The time interval is rounded off, that is, the average time interval between the 2nd category and the 3rd category is 3ms; and so on, the time interval between the 6th category and the 7th category is 3ms, and the average time interval is 2ms.

[0166] The first time point is determined to be the training start time, i.e., 0ms; the second time point is determined to be 4ms based on the average time interval of 1ms between the first category and the second category and the end time of 3ms of the first category; the third time point is determined to be 8ms based on the average time interval of 3ms between the second category and the third category and the end time of 5ms of the second category; and so on, the seventh time point is determined to be 29ms based on the average time interval of 2ms between the sixth category and the seventh category and the end time of 27ms of the sixth category; it should be noted that, in this embodiment, the last time point in the time point sequence is determined based on the end time of the last category and the average time interval of the last two adjacent categories, that is, the eighth time point is determined to be 32ms based on the average time interval of 2ms between the sixth category and the seventh category and the end time of 30ms of the seventh category.

[0167]

[0168]

[0169] Table 1

[0170] Optionally, when the time interval is greater than a preset threshold, third training data is received and the third training data is input into the neural network model to be trained for training; multiple gradients generated by the neural network model to be trained during the training process are obtained, and the generation time of each gradient is recorded; multiple gradients of the neural network model to be trained are clustered according to the generation time of each gradient; and a time point sequence is determined according to the clustering results.

[0171] In practical applications, a preset threshold is set during the training process. When the time interval between any two adjacent categories exceeds the preset threshold, it indicates that the clustering algorithm is unreasonable. At this time, clustering can be restarted to obtain a new time point sequence.

[0172] The second embodiment of the present specification provides a gradient synchronization method in distributed training, which monitors the training time in each training node, and triggers the gradient synchronization operation when the training time is the same as the first time point in a preset time point sequence, thereby saving the time for negotiating synchronized gradients and the time for waiting for corresponding gradients in other training nodes, and improving the training efficiency of the neural network model. During the gradient synchronization process, the gradients generated between two adjacent time points are packaged to generate a gradient set, and then the gradient set is synchronized between the training nodes in the distributed training cluster. Multiple gradients can be synchronized at one time, saving the time for frequent gradient synchronization, and further improving the training efficiency of the neural network model. After gradient synchronization, each training node updates the parameters of the neural network model to be trained in the current training node according to the gradient set in each training node, thereby ensuring the consistency of the parameters of the neural network model to be trained on each training node and improving the stability of the training of the neural network model to be trained.

[0173] Figure 7 A processing flow chart of a gradient synchronization method in distributed training provided according to the third embodiment of this specification is shown. In the gradient synchronization method in distributed training provided by this embodiment, the distributed training cluster is a Parameter-Server architecture. In the Parameter-Server architecture, at least one parameter server and at least two training nodes are included, and each training node is connected to the parameter server.

[0174] In the distributed training scenario of the Parameter-Server architecture, the gradient of each training node is synchronized to the parameter server. The parameter server updates the parameters of the neural network model to be trained according to the gradient sent by each training node, and synchronizes the parameters to each training node to ensure that the model parameters of each training node are consistent.

[0175] like Figure 7 As shown, the gradient synchronization method in distributed training includes the following steps 702 to 708.

[0176] Step 702: Each of the training nodes inputs the first training data into the neural network model to be trained for training.

[0177] Step 704: Each of the training nodes monitors the training time, and when the training time is the same as the first time point in a preset time point sequence, obtains a gradient set generated by the neural network model to be trained within a first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point.

[0178] Steps 702 to 704 are consistent with the methods of steps 102 to 104 described above. For specific explanations of steps 702 to 704, please refer to the detailed contents of steps 102 to 104 in the aforementioned embodiment, which will not be repeated here.

[0179] Step 706: Each of the training nodes sends the gradient set to the parameter server.

[0180] Each of the training nodes sends the gradient set in the current training node to the parameter server according to the first time point in the time point sequence, which saves the negotiation and synchronization time between the training node and the parameter server and improves the training efficiency.

[0181] In the third embodiment provided in this specification, see Figure 8 , Figure 8 The schematic diagram of the structure of the distributed training cluster under the Parameter-Server architecture is shown. Training node 1 sends gradient set 1 to the parameter server, training node 2 sends gradient set 2 to the parameter server, and training node 3 sends gradient set 3 to the parameter server.

[0182] Step 708: The parameter server receives a gradient set sent by each training node.

[0183] The parameter server receives the gradient set sent by each training node and completes the gradient synchronization at the first time point.

[0184] In the third embodiment provided in this specification, the parameter server receives gradient set 1 sent by training node 1, gradient set 2 sent by training node 2, and gradient set 3 sent by training node 3.

[0185] The third embodiment of the present specification provides a method for synchronizing gradients in distributed training, which is applied to a distributed training cluster based on a Parameter-Server architecture. The training time is monitored in each training node, and a gradient synchronization operation is triggered when the training time is the same as the first time point in a preset time point sequence. This saves the time for negotiating gradient synchronization between the training node and the parameter server and the time for waiting for corresponding gradients in other training nodes, thereby improving the training efficiency of the neural network model. During the gradient synchronization process, the gradients generated between two adjacent time points are packaged to generate a gradient set, and then the gradient set is sent to the parameter server in the distributed training cluster. Multiple gradients can be synchronized at one time, saving the time for frequent gradient synchronization, and further improving the training efficiency of the neural network model.

[0186] Fig. 9A processing flow chart of a gradient synchronization method in distributed training provided according to the fourth embodiment of this specification is shown. The distributed training cluster in the gradient synchronization method in distributed training provided in this embodiment is a Parameter-Server architecture. In the Parameter-Server architecture, at least one parameter server and at least two training nodes are included, and each of the training nodes is connected to the parameter server.

[0187] In the distributed training scenario of the Parameter-Server architecture, the gradient of each training node is synchronized to the parameter server. The parameter server updates the parameters of the neural network model to be trained according to the gradient sent by each training node, and synchronizes the parameters to each training node to ensure that the model parameters of each training node are consistent.

[0188] like Fig. 9 As shown, the gradient synchronization method in distributed training includes the following steps 902 to 912.

[0189] Step 902: Each of the training nodes inputs the first training data into the neural network model to be trained for training.

[0190] Step 904: Each of the training nodes monitors the training time, and when the training time is the same as the first time point in a preset time point sequence, obtains a gradient set generated by the neural network model to be trained within a first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point.

[0191] Step 906: Each of the training nodes sends the gradient set to the parameter server.

[0192] Step 908: The parameter server receives a gradient set sent by each training node.

[0193] Steps 902 to 908 are consistent with the methods of steps 702 to 708 described above. For specific explanations of steps 902 to 908, please refer to the detailed contents of steps 702 to 708 in the aforementioned embodiment, which will not be repeated here.

[0194] Step 910: The parameter server updates the parameters of the neural network model to be trained according to the gradient set sent by each training node, and sends the updated parameters to each training node.

[0195] The parameter server updates the parameters of the neural network model to be trained in the parameter server according to the gradient set sent by each training node, and then sends the updated parameters to each training node to ensure that the parameters of the neural network model to be trained in each training node are consistent.

[0196] Optionally, updating the parameters of the neural network model to be trained according to the gradient set sent by each training node can be achieved through the following steps S9102 to S9108.

[0197] S9102. Accumulate the gradients in the gradient set sent by each training node to obtain an accumulated gradient.

[0198] In the fourth embodiment provided in this specification, see Figure 2 , Figure 3 and Figure 8 ,by Figure 3 Take the fifth time point shown as an example, and then combine Figure 2 , at this point in time, Figure 8 The parameter server in the training node 1 is based on the gradient set (T 112 、T 113 ), the gradient set of training node 2 (T 212 、T 213 ), the gradient set of training node 3 (T 312 , T 313 ), the gradients in the gradient set are accumulated to obtain the accumulated gradient T 112 +T 212 +T 312 and T 113 +T 213 +T 313 .

[0199] S9104. Calculate an average gradient according to the accumulated gradient and the number of training nodes in the distributed training cluster.

[0200] In the fourth embodiment provided in this specification, the number of training nodes is 3, so the average gradient T 012 =(T 112 +T 212 +T 312 ) / 3,T 013 =(T 113 +T 213 +T 313 ) / 3.

[0201] S9106. Calculate the loss value of the neural network model to be trained based on the average gradient.

[0202] S9108. Update the parameters of the neural network model to be trained according to the loss value.

[0203] Step 912: Each of the training nodes receives the updated parameters sent by the parameter server, and updates the parameters of the neural network model to be trained in the current training node.

[0204] Optionally, a method for generating a preset time point sequence is also provided in the fourth embodiment provided in this specification. The method is consistent with the methods of steps S5122 to S5128 in the aforementioned embodiment. The specific operating steps of the method for generating a preset time point sequence refer to the detailed contents of steps S5122 to S5128 in the aforementioned embodiment, which will not be repeated here.

[0205] The fourth embodiment of the present specification provides a method for synchronizing gradients in distributed training. The method monitors the training time in each training node, and triggers the gradient synchronization operation when the training time is the same as the first time point in a preset time point sequence, thereby saving the time for negotiating gradient synchronization between the training node and the parameter server and the time for waiting for corresponding gradients in other training nodes, thereby improving the training efficiency of the neural network model. During gradient synchronization, the gradients generated between two adjacent time points are packaged to generate a gradient set, and then the gradient set is sent to the parameter server. Multiple gradients can be synchronized at one time, saving the time for frequent gradient synchronization, further improving the training efficiency of the neural network model. The parameter server updates the neural network model to be trained according to the gradients in the gradient set of each training node to obtain updated parameters, and then sends the updated model parameters to each training node, thereby ensuring the consistency of the parameters of the neural network model to be trained on each training node, and improving the stability of the training of the neural network model to be trained.

[0206] Fig.10 A processing flow chart of a gradient synchronization method in distributed training provided according to the fifth embodiment of this specification is shown. In the gradient synchronization method in distributed training provided in this embodiment, the distributed training cluster is a Parameter-Server architecture. The method is applied to each training node in the distributed training cluster, including the following steps 1002 to 1006.

[0207] Step 1002: Input the first training data into the neural network model to be trained for training.

[0208] Step 1004: Monitor the training time, and when the training time is the same as the first time point in a preset time point sequence, obtain a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point.

[0209] Step 1006: Send the gradient set to a parameter server in the distributed training cluster.

[0210] Steps 1002 to 1006 are consistent with the methods of steps 702 to 706 described above. For specific explanations of steps 1002 to 1006, please refer to the detailed contents of steps 702 to 706 in the aforementioned embodiment, which will not be repeated here.

[0211] The fifth embodiment of the present specification provides a gradient synchronization method in distributed training, which is applied to a distributed training cluster based on a Parameter-Server architecture. The training time is monitored in each training node, and gradient synchronization is triggered when the training time is the same as the first time point in a preset time point sequence, thereby saving the time for negotiating gradient synchronization between the training node and the parameter server and the time for waiting for corresponding gradients in other training nodes, thereby improving the training efficiency of the neural network model. During gradient synchronization, the gradients generated between two adjacent time points are packaged to generate a gradient set, and then the gradient set is synchronized between the training nodes in the distributed training cluster. Multiple gradients can be synchronized at one time, thereby saving the time for frequent gradient synchronization, thereby further improving the training efficiency of the neural network model.

[0212] Corresponding to the embodiment of the gradient synchronization method in distributed training provided in the second embodiment above, see Fig.11 This specification also provides a distributed training system 1100, which includes at least three training nodes and the training nodes are connected in a ring. Fig.11 Only 5 training nodes are shown, such as Fig.11 As shown, the distributed training system includes training node 1101, training node 1102, training node 1103, training node 1104, and training node 1105.

[0213] Each of the training nodes is configured to input the first training data into the neural network model to be trained for training; monitor the training time, and when the training time is the same as the first time point in a preset time point sequence, obtain the gradient set generated by the neural network model to be trained within the first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point; send the gradient set to a communication library pre-configured in the current training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster; obtain the gradient sets in other training nodes in the distributed training cluster through the communication library.

[0214] Optionally, each of the training nodes is further configured to update the parameters of the neural network model to be trained in the current training node according to the gradient set of each of the training nodes in the distributed training cluster.

[0215] Optionally, each of the training nodes is further configured to accumulate the gradients in the gradient set of each training node in the distributed training cluster to obtain an accumulated gradient; calculate an average gradient based on the accumulated gradient and the number of training nodes in the distributed training cluster; calculate a loss value of the neural network model to be trained based on the average gradient; and update the parameters of the neural network model to be trained based on the loss value.

[0216] Optionally, each of the training nodes is also configured to receive second training data, and input the second training data into the neural network model to be trained for training; obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient; cluster the multiple gradients of the neural network model to be trained according to the generation time of each gradient; and determine the time point sequence according to the clustering results.

[0217] Optionally, each of the training nodes is further configured to determine the start time and end time corresponding to the current category based on the generation time of the gradient in each category of the clustering results; determine the time interval between adjacent categories based on the start time and end time of each category of the clustering results; and create a time point sequence based on the time interval.

[0218] Optionally, each of the training nodes is also configured to receive third training data when the time interval is greater than a preset threshold, and input the third training data into the neural network model to be trained for training; obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient; cluster the multiple gradients of the neural network model to be trained according to the generation time of each gradient; and determine the time point sequence according to the clustering results.

[0219] The distributed training system provided in this specification is applied to a distributed training cluster based on a Ring-allreduce architecture. The training time is monitored in each training node. When the training time is the same as the first time point in a preset time point sequence, gradient synchronization is triggered, which saves the time for negotiating synchronized gradients and the time for waiting for corresponding gradients in other training nodes, thereby improving the training efficiency of the neural network model. During the gradient synchronization process, the gradients generated between two adjacent time points are packaged to generate a gradient set, and then the gradient set is synchronized between the training nodes in the distributed training cluster. Multiple gradients can be synchronized at one time, which saves the time for frequent gradient synchronization and further improves the training efficiency of the neural network model.

[0220] Corresponding to the embodiment of the gradient synchronization method in distributed training provided in the fourth embodiment above, see Fig.12 This specification also provides a distributed training system 1200, which includes at least one parameter server and at least two training nodes, each of which is connected to the parameter server. Fig.12 Only one parameter server and five training nodes are shown. Fig.12 As shown, the distributed training system includes a parameter server 12101, a training node 12201, a training node 12202, a training node 12203, a training node 12204, and a training node 12205.

[0221] Each of the training nodes is configured to input the first training data into the neural network model to be trained for training; monitor the training time, and when the training time is the same as the first time point in the preset time point sequence, obtain the gradient set generated by the neural network model to be trained in the first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point; send the gradient set to the parameter server;

[0222] The parameter server 12101 is configured to receive a gradient set sent by each training node.

[0223] Optionally, the parameter server 12101 is further configured to update the parameters of the neural network model to be trained according to the gradient set sent by each training node, and send the updated parameters to each training node.

[0224] Each of the training nodes is further configured to receive updated parameters sent by the parameter server and update the parameters of the neural network model to be trained in the current training node.

[0225] Optionally, the parameter server 12101 is further configured to accumulate the gradients in the gradient set sent by each of the training nodes to obtain an accumulated gradient; calculate an average gradient based on the accumulated gradient and the number of training nodes in the distributed training cluster; calculate a loss value of the neural network model to be trained based on the average gradient; and update the parameters of the neural network model to be trained based on the loss value.

[0226] Optionally, each of the training nodes is also configured to receive second training data, and input the second training data into the neural network model to be trained for training; obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient; cluster the multiple gradients of the neural network model to be trained according to the generation time of each gradient; and determine the time point sequence according to the clustering results.

[0227] Optionally, each of the training nodes is further configured to determine the start time and end time corresponding to the current category based on the generation time of the gradient in each category of the clustering results; determine the time interval between adjacent categories based on the start time and end time of each category of the clustering results; and create a time point sequence based on the time interval.

[0228] Optionally, each of the training nodes is also configured to receive third training data when the time interval is greater than a preset threshold, and input the third training data into the neural network model to be trained for training; obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient; cluster the multiple gradients of the neural network model to be trained according to the generation time of each gradient; and determine the time point sequence according to the clustering results.

[0229] The distributed training system provided in the embodiments of this specification is applied to a distributed training cluster based on a Parameter-Server architecture. The training time is monitored in each training node. When the training time is the same as the first time point in a preset time point sequence, gradient synchronization is triggered, which saves the time for negotiating gradient synchronization between the training node and the parameter server and the time for waiting for corresponding gradients in other training nodes, thereby improving the training efficiency of the neural network model. During the gradient synchronization process, the gradients generated between two adjacent time points are packaged to generate a gradient set, and then the gradient set is synchronized between the training nodes in the distributed training cluster. Multiple gradients can be synchronized at one time, which saves the time for frequent gradient synchronization and further improves the training efficiency of the neural network model.

[0230] Corresponding to the embodiment of the gradient synchronization method in distributed training provided in the fifth embodiment above, the present specification also provides a distributed training system, wherein the distributed training system includes at least two training nodes, each of the training nodes is configured to input first training data into a neural network model to be trained for training; monitor the training time, and when the training time is the same as a first time point in a preset time point sequence, obtain a gradient set generated by the neural network model to be trained within a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point; and send the gradient set to a parameter server in the distributed training cluster.

[0231] Fig.13 The block diagram of a computing device 1300 according to one embodiment of the present specification is shown. The components of the computing device 1300 include but are not limited to a memory 1310 and a processor 1320. The processor 1320 is connected to the memory 1310 via a bus 1330, and the database 1350 is used to store data.

[0232] The computing device 1300 also includes an access device 1340 that enables the computing device 1300 to communicate via one or more networks 1360. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1340 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0233] In one embodiment of the present specification, the above components of the computing device 1300 and Fig.13 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.13 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0234] Computing device 1300 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 1300 may also be a mobile or stationary server.

[0235] The memory 1310 is used to store computer executable instructions, and the processor 1320 is used to execute the following computer executable instructions:

[0236] Inputting the first training data into the neural network model to be trained for training;

[0237] Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point;

[0238] Sending the gradient set to a communication library pre-configured in the current training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster;

[0239] The gradient sets in other training nodes in the distributed training cluster are obtained through the communication library.

[0240] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the gradient synchronization method in distributed training provided in the second embodiment above belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the gradient synchronization method in distributed training provided in the second embodiment above.

[0241] Fig.14 The structure block diagram of the electronic device 1400 according to an embodiment of the present application is shown. The components of the electronic device 1400 include but are not limited to a memory 1410 and a processor 1420. The processor 14620 is connected to the memory 1410 via a bus 1430, and the database 1450 is used to store data.

[0242] The electronic device 1400 also includes an access device 1440 that enables the electronic device 1400 to communicate via one or more networks 1460 .

[0243] The access method of the electronic device 1400 is the same as that of the above-mentioned electronic device 1400, which will not be repeated here.

[0244] The processor 1420 is used to execute the following computer executable instructions:

[0245] Inputting the first training data into the neural network model to be trained for training;

[0246] Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point;

[0247] The gradient set is sent to a parameter server in the distributed training cluster.

[0248] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the gradient synchronization method in distributed training provided in the fourth embodiment above belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the gradient synchronization method in distributed training provided in the fourth embodiment above.

[0249] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the steps of the gradient synchronization method in distributed training provided in the second embodiment.

[0250] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the gradient synchronization method in distributed training provided in the second embodiment above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the gradient synchronization method provided in the second embodiment in distributed training above.

[0251] An embodiment of the present specification also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the steps of the gradient synchronization method in distributed training provided in the fourth embodiment.

[0252] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the gradient synchronization method in distributed training provided in the fourth embodiment above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the gradient synchronization method provided in the fourth embodiment in distributed training above.

[0253] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0254] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0255] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0256] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0257] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A method for synchronizing gradients in distributed training, the method comprising: Inputting the first training data into the neural network model to be trained for training; Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point, and the preset time point sequence is determined by the generation time of each gradient of the neural network model to be trained during the training process; Sending the gradient set to a communication library pre-configured in the current training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster; The gradient sets in other training nodes in the distributed training cluster are obtained through the communication library.

2. The method for synchronizing gradients in distributed training according to claim 1, after obtaining the gradient sets in other training nodes in the distributed training cluster through the communication library, further comprising: The parameters of the neural network model to be trained in the current training node are updated according to the gradient set of each training node in the distributed training cluster.

3. The gradient synchronization method in distributed training according to claim 2, updating the parameters of the neural network model to be trained in the current training node according to the gradient set of each training node in the distributed training cluster, comprising: Accumulating the gradients in the gradient set of each training node in the distributed training cluster to obtain an accumulated gradient; Calculating an average gradient based on the accumulated gradient and the number of training nodes in the distributed training cluster; Calculating the loss value of the neural network model to be trained according to the average gradient; Update the parameters of the neural network model to be trained according to the loss value.

4. The gradient synchronization method in distributed training according to claim 1, wherein the time point sequence includes n time points, wherein: n>1； The training time is monitored, and when the training time is the same as the first time point in the preset time point sequence, a gradient set generated by the neural network model to be trained in the first time interval is obtained, including: The training time is monitored, and when the training time is the same as the i-th time point, the gradients generated between the i-1-th time point and the i-th time point are packaged to generate a gradient set, where 2<=i<=n; Increment i by 1 and determine whether i is greater than n. If so, end the gradient synchronization. If not, continue to perform the operation when the training time is the same as the i-th time point.

5. The gradient synchronization method in distributed training according to claim 1, wherein the preset time point sequence is generated by the following steps, including: Receiving second training data, and inputting the second training data into the neural network model to be trained for training; Obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient; Clustering multiple gradients of the neural network model to be trained according to the generation time of each gradient; Determine the time point sequence based on the clustering results.

6. The method for synchronizing gradients in distributed training according to claim 5, wherein determining the time point sequence according to the clustering result comprises: Determine the start time and end time corresponding to the current category according to the generation time of the gradient in each category of the clustering result; Determine the time interval between adjacent categories according to the start time and end time of each category of the clustering result; A time point sequence is produced according to the time interval.

7. The method for synchronizing gradients in distributed training according to claim 6, after determining the time interval between adjacent categories according to the start time and end time of each category of the clustering result, further comprising: When the time interval is greater than a preset threshold, receiving third training data, and inputting the third training data into the neural network model to be trained for training; Obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient; Clustering multiple gradients of the neural network model to be trained according to the generation time of each gradient; Determine the time point sequence based on the clustering results.

8. A method for synchronizing gradients in distributed training, the method being applied to a distributed training cluster of a neural network model to be trained, the distributed training cluster comprising at least one parameter server and at least two training nodes, the method comprising: Each of the training nodes inputs the first training data into the neural network model to be trained for training; Each of the training nodes monitors the training time, and when the training time is the same as the first time point in the preset time point sequence, obtains a gradient set generated by the neural network model to be trained in the first time interval, wherein the first time interval is determined by the previous time point of the first time point and the first time point, and the preset time point sequence is determined by the generation time of each gradient of the neural network model to be trained during the training process; Each of the training nodes sends the gradient set to the parameter server; The parameter server receives a gradient set sent by each training node.

9. The method for synchronizing gradients in distributed training according to claim 8, further comprising: after the parameter server receives the gradient set sent by each training node: The parameter server updates the parameters of the neural network model to be trained according to the gradient set sent by each training node, and sends the updated parameters to each training node; Each of the training nodes receives the updated parameters sent by the parameter server, and updates the parameters of the neural network model to be trained in the current training node.

10. The method for synchronizing gradients in distributed training according to claim 9, wherein the parameter server updates the parameters of the neural network model to be trained according to the gradient set sent by each training node, comprising: Accumulating the gradients in the gradient set sent by each training node to obtain an accumulated gradient; Calculating an average gradient based on the accumulated gradient and the number of training nodes in the distributed training cluster; Calculating the loss value of the neural network model to be trained according to the average gradient; Update the parameters of the neural network model to be trained according to the loss value.

11. The gradient synchronization method in distributed training according to claim 8, wherein the time point sequence comprises n time points, wherein: n>1； Each of the training nodes monitors the training time, and when the training time is the same as the first time point in the preset time point sequence, obtains a gradient set generated by the neural network model to be trained in the first time interval, including: The training time of each training node is monitored, and when the training time is the same as the i-th time point, the gradients generated between the i-1-th time point and the i-th time point are packaged to generate a gradient set, where 2<=i<=n; Increment i by 1 and determine whether i is greater than n. If so, end the gradient synchronization. If not, continue to perform the operation when the training time is the same as the i-th time point.

12. The gradient synchronization method in distributed training according to claim 8, wherein the preset time point sequence is generated by the following steps and applied to each of the training nodes, including: Receiving second training data, and inputting the second training data into the neural network model to be trained for training; Obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient; Clustering multiple gradients of the neural network model to be trained according to the generation time of each gradient; Determine the time point sequence based on the clustering results.

13. The method for synchronizing gradients in distributed training according to claim 12, wherein determining the time point sequence according to the clustering result comprises: Determine the start time and end time corresponding to the current category according to the generation time of the gradient in each category of the clustering result; Determine the time interval between adjacent categories according to the start time and end time of each category of the clustering result; A time point sequence is produced according to the time interval.

14. The method for synchronizing gradients in distributed training according to claim 13, after determining the time interval between adjacent categories according to the start time and end time of each category of the clustering result, further comprising: When the time interval is greater than a preset threshold, receiving third training data, and inputting the third training data into the neural network model to be trained for training; Obtain multiple gradients generated by the neural network model to be trained during the training process, and record the generation time of each gradient; Clustering multiple gradients of the neural network model to be trained according to the generation time of each gradient; Determine the time point sequence based on the clustering results.

15. A gradient synchronization method in distributed training, applied to each training node in a distributed training cluster, the method comprising: Inputting the first training data into the neural network model to be trained for training; Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point, and the preset time point sequence is determined by the generation time of each gradient of the neural network model to be trained during the training process; The gradient set is sent to a parameter server in the distributed training cluster.

16. A distributed training system, comprising at least three training nodes and the training nodes are connected in a ring; Each of the training nodes is configured to input the first training data into the neural network model to be trained for training; The training time is monitored, and when the training time is the same as the first time point in a preset time point sequence, a gradient set generated by the neural network model to be trained in a first time interval is obtained, wherein the first time interval is determined by the previous time point of the first time point and the first time point; the gradient set is sent to a communication library preconfigured in the current training node, wherein the communication library is used to communicate with other training nodes in a distributed training cluster; the gradient sets in other training nodes in the distributed training cluster are obtained through the communication library, and the preset time point sequence is determined by the generation time of each gradient of the neural network model to be trained during the training process.

17. A distributed training system, comprising at least one parameter server and at least two training nodes, each of the training nodes being connected to the parameter server respectively; Each of the training nodes is configured to input the first training data into the neural network model to be trained for training; Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point; sending the gradient set to the parameter server, wherein the preset time point sequence is determined by the generation time of each gradient of the neural network model to be trained during the training process; The parameter server is configured to receive a gradient set sent by each training node.

18. A distributed training system, comprising at least two training nodes, Each of the training nodes is configured to input the first training data into the neural network model to be trained for training; monitor the training time, and when the training time is the same as the first time point in the preset time point sequence, obtain the gradient set generated by the neural network model to be trained in the first time interval, wherein, The first time interval is determined by the previous time point of the first time point and the first time point; the gradient set is sent to the parameter server in the distributed training cluster, and the preset time point sequence is determined by the generation time of each gradient of the neural network model to be trained during the training process.

19. A computing device comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions: Inputting the first training data into the neural network model to be trained for training; Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point, and the preset time point sequence is determined by the generation time of each gradient of the neural network model to be trained during the training process; Sending the gradient set to a communication library pre-configured in the current training node, wherein the communication library is used to communicate with other training nodes in the distributed training cluster; The gradient sets in other training nodes in the distributed training cluster are obtained through the communication library.

20. A computing device comprising: Memory and processor; The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions: Inputting the first training data into the neural network model to be trained for training; Monitoring the training time, and when the training time is the same as a first time point in a preset time point sequence, obtaining a gradient set generated by the neural network model to be trained in a first time interval, wherein the first time interval is determined by a previous time point of the first time point and the first time point, and the preset time point sequence is determined by the generation time of each gradient of the neural network model to be trained during the training process; The gradient set is sent to a parameter server in a distributed training cluster.

21. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the gradient synchronization method in distributed training as described in any one of claims 1 to 7.

22. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the gradient synchronization method in distributed training described in claim 15.

Images