Distributed training method and device for neural network

Abstract

The present application provides a distributed training method and device for a neural network, which can effectively reduce the number and time of determining a gradient aggregation scheme, thereby improving training efficiency. The method includes: before starting a training task, determining a gradient aggregation threshold once, and then dividing multiple gradient calculation operators in the neural network into multiple gradient aggregation sets each including at least one gradient calculation operator according to the gradient aggregation threshold, and when all gradient calculation operators in any gradient aggregation set complete a gradient calculation, then taking the any gradient aggregation set as a whole, performing a gradient aggregation operation.

Description

Technical Field

[0001] The present application relates to the field of neural networks, and in particular to a distributed training method and device for neural networks. Background Art

[0002] At present, in view of the increasing complexity of neural networks and the increasing number of layers, distributed training methods are usually used to train neural networks to improve the training efficiency of neural networks. Exemplarily, the computing nodes N0-N4 are connected end to end to form a ring-shaped distributed training system as an example. For each computing node in the ring-shaped distributed training system, each computing node uses a training sample to complete a gradient calculation of the neural network. Each computing node must determine the gradient aggregation scheme and complete the gradient aggregation based on the amount of gradient data calculated by the computing node and the amount of gradient data calculated by the upstream adjacent nodes of the computing node. Among them, gradient aggregation refers to merging the gradient data obtained by multiple computing nodes in a preset manner to obtain a merged result.

[0003] That is to say, in the existing distributed training, for each training, each computing node must determine a gradient aggregation plan after completing a gradient calculation, which wastes a lot of computing resources and communication resources between computing nodes, resulting in increased training time of the neural network and low training efficiency. Summary of the invention

[0004] The present application provides a distributed training method and device for a neural network, which can effectively reduce the number and time consumption of determining a gradient aggregation scheme, thereby improving training efficiency.

[0005] In a first aspect, a distributed training method for a neural network is provided. The method includes: determining a gradient aggregation threshold of the neural network. Then, according to the gradient aggregation threshold, multiple gradient calculation operators in the neural network are divided into multiple gradient aggregation sets; wherein each gradient aggregation set includes at least one gradient calculation operator. Afterwards, a training task for the neural network is started. wherein the training task is used to perform multiple rounds of training iterations on the neural network, and each round of training iterations is used to perform multiple gradient calculations corresponding to multiple gradient calculation operators respectively. Finally, gradient aggregation is performed for the gradient data of any gradient aggregation set. wherein the gradient data is obtained by performing at least one gradient calculation corresponding to at least one gradient calculation operator in any gradient aggregation set, and the gradient aggregation is used to aggregate the above-mentioned gradient data with other gradient data of other computing nodes in the distributed training system, and the other gradient data is the gradient data of the gradient aggregation set corresponding to any one of the above-mentioned gradient aggregation sets in other computing nodes.

[0006] The distributed training method of a neural network provided in the present application can determine a gradient aggregation threshold before starting a training task, and then divide multiple gradient calculation operators in the neural network into multiple gradient aggregation sets each including at least one gradient calculation operator according to the gradient aggregation threshold, and when all gradient calculation operators in any gradient aggregation set each complete a gradient calculation, the any gradient aggregation set is taken as a whole to perform a gradient aggregation operation, which can avoid the situation where the coordinator node needs to determine the gradient aggregation scheme for each training sample multiple times in the process of using multiple training samples to perform training tasks, and can effectively reduce the number of times and time consumption of determining the gradient aggregation scheme, thereby improving training efficiency.

[0007] In a possible design method, the above-mentioned determination of the gradient aggregation threshold of the neural network may include: determining the gradient aggregation threshold according to a first correspondence between the gradient data statistic and the gradient calculation time, and a second correspondence between the gradient data statistic and the gradient aggregation time. For example, the first correspondence is a first functional relationship, and the second correspondence is a second functional relationship.

[0008] Exemplarily, the above-mentioned determination of the gradient aggregation threshold value based on the first corresponding relationship between the gradient data statistic and the gradient calculation time consumption, and the second corresponding relationship between the gradient data statistic and the gradient aggregation time consumption, may include: determining a preferred gradient data amount in the gradient data statistic, and determining the preferred gradient data amount as the gradient aggregation threshold value. The time deviation between the gradient calculation time consumption corresponding to the preferred gradient data amount and the gradient aggregation time consumption is less than the time deviation threshold value.

[0009] Optionally, it is preferred that the gradient calculation time corresponding to the amount of gradient data is equal to the gradient aggregation time.

[0010] Furthermore, the above-mentioned distributed training method of the neural network may also include: calculating a first correspondence between gradient data statistics and gradient calculation time by online calculation or offline calculation, and a second correspondence between gradient data statistics and gradient aggregation time.

[0011] In a second aspect, a device in a distributed training system is provided. The device includes: a determination module, a training module and an aggregation module. The determination module is used to determine the gradient aggregation threshold of the neural network, and divide the multiple gradient calculation operators in the neural network into multiple gradient aggregation sets according to the gradient aggregation threshold. Each gradient aggregation set includes at least one gradient calculation operator. The training module is used to start the training task of the neural network. The training task is used to perform multiple rounds of training iterations on the neural network, and each round of training iteration is used to perform multiple gradient calculations corresponding to multiple gradient calculation operators. The aggregation module is used to perform gradient aggregation for the gradient data of any gradient aggregation set. The gradient data is obtained by executing at least one gradient calculation corresponding to at least one gradient calculation operator in any gradient aggregation set, and the gradient aggregation is used to aggregate the above-mentioned gradient data with other gradient data of other computing nodes in the distributed training system, and the other gradient data is the gradient data of the gradient aggregation set corresponding to any one of the above-mentioned gradient aggregation sets in other computing nodes.

[0012] In a possible design, the determination module is further used to determine the gradient aggregation threshold according to a first correspondence between the gradient data statistic and the gradient calculation time, and a second correspondence between the gradient data statistic and the gradient aggregation time. For example, the first correspondence is a first functional relationship, and the second correspondence is a second functional relationship.

[0013] Exemplarily, the determination module is further used to determine the preferred gradient data amount in the gradient data statistics, and to determine the preferred gradient data amount as the gradient aggregation threshold. The time deviation between the gradient calculation time corresponding to the preferred gradient data amount and the gradient aggregation time is less than the time deviation threshold;

[0014] Optionally, it is preferred that the gradient calculation time corresponding to the amount of gradient data is equal to the gradient aggregation time.

[0015] Furthermore, the above-mentioned determination module is used to calculate the first corresponding relationship between the gradient data statistics and the gradient calculation time, and the second corresponding relationship between the gradient data statistics and the gradient aggregation time by online calculation or offline calculation.

[0016] In a third aspect, an apparatus is provided, comprising: a processor coupled to a memory; and the memory is used to store a computer program. The processor is used to execute the computer program stored in the memory, so that the communication device executes the distributed training method of a neural network described in the first aspect or any possible implementation of the first aspect.

[0017] In one possible design, the device described in the third aspect also includes one or more transceivers. The transceiver is used to support the communication between the device and other devices to realize the receiving and / or sending functions between computing nodes. For example, receiving gradient data sent by upstream adjacent nodes, sending gradient data to downstream adjacent nodes, receiving training samples, etc. The processor includes one or more processors, which are configured to support the device described in the third aspect to perform the functions of the above-mentioned computing nodes. For example, determining a gradient aggregation threshold, dividing multiple gradient calculation operators in a neural network into multiple gradient aggregation sets according to the gradient aggregation threshold, performing gradient aggregation on the gradient data of any gradient aggregation set, etc.

[0018] Optionally, the memory may also include one or more memories, which are coupled to the processor and are used to store program instructions and / or data necessary for the device. The memory is coupled to the processor, which means that there is a signal connection between the memory and the processor. The one or more memories may be integrated with the processor or may be separately provided from the processor, which is not limited in this application.

[0019] The device may be a computing node in a distributed network. The transceiver may be a transceiver circuit of the computing node. Optionally, the transceiver may also be an input / output circuit or an interface.

[0020] The device may also be one or more communication chips arranged in the computing node. The transceiver may be an input / output circuit or an interface of the communication chip.

[0021] In a fourth aspect, a distributed training system for a neural network is provided, the system comprising one or more ring links, each ring link comprising at least two computing nodes, any computing node being used to execute the distributed training method for a neural network as described in the first aspect or any possible implementation of the first aspect.

[0022] In a fifth aspect, a readable storage medium is provided, storing a program or instruction. When the program or instruction is executed on a computer or a processor, the computer or the processor executes the distributed training method of a neural network as described in the first aspect or any possible implementation of the first aspect.

[0023] In a sixth aspect, a computer program product is provided, comprising a computer program code. When the computer program code runs on a computer or a processor, the computer or the processor executes the distributed training method for a neural network as described in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1A schematic diagram of the structure of a distributed training system for a neural network to which the distributed training method for a neural network provided in the present application is applicable;

[0025] Figure 2 A schematic diagram of a flow chart of a distributed training method for a neural network provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of a neural network before optimization provided in an embodiment of the present application;

[0027] Figure 4 A schematic diagram of a scenario of a method for determining a gradient aggregation threshold provided in an embodiment of the present application;

[0028] Figure 5 A schematic diagram of the structure of an optimized neural network provided in an embodiment of the present application;

[0029] Figure 6 A schematic diagram of the structure of the device in the distributed training system provided in the embodiment of the present application Figure 1 ;

[0030] Figure 7 A schematic diagram of the structure of the device in the distributed training system provided in the embodiment of the present application Figure 2 ;

[0031] Figure 8 A schematic diagram of the structure of the distributed training software and hardware based on the tensor flow framework provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in this application will be described below in conjunction with the accompanying drawings. The technical solutions of the embodiments of the present application can be applied to a distributed training system of a ring-allreduce neural network based on a tensorflow framework, but are not limited thereto. This application will present various aspects, embodiments or features around a system that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. In addition, a combination of these solutions may also be used.

[0033] In the embodiment of the present application, a neural network is an algorithmic network that can learn, summarize and generalize, and can be built into a computing node in the form of neural network software or hardware, such as a neural network program, an executable script, etc. The neural network can learn and summarize by experimental application of known data (hereinafter referred to as training samples) to improve the recognition ability of the neural network. Generally speaking, a neural network consists of multiple layers of neurons (operators), each layer has multiple inputs and outputs, and the input or output is a multidimensional array, also known as a tensor. Each layer has one or more weighted values, called weights. The output result of a certain layer is equal to the result of mathematical operations such as the addition or multiplication of the input and weight of the layer. The process of creating a model by learning the weights of each layer through training samples is called a training process. After executing a training, the process of correcting the weights of each layer is called completing a training iteration. In the process of training iteration, multiple gradient calculations corresponding to multiple gradient calculation operators are required to obtain gradient data for each gradient calculation operator. Among them, the neural network may include multiple gradient calculation operators, each of which corresponds to an operation or processing of the neural network. Any gradient calculation operator can be located on a layer of the neural network to implement a gradient calculation. The gradient data obtained by the gradient calculation is also called a gradient, which reflects the deviation of the calculated weight. Therefore, the corresponding weight update can be performed later.

[0034] In the embodiments of the present application, "example" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present concepts in a concrete way. In the embodiments of the present application, "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction is not emphasized, the meanings to be expressed are similar or consistent.

[0035] The network architecture and business scenarios described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. It is known to those skilled in the art that with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems. Figure 1 The distributed training system of the ring neural network shown in the figure is used as an example to explain in detail the distributed training system of the neural network applicable to the embodiments of the present application.

[0036] like Figure 1As shown, the distributed training system of the neural network includes computing nodes N1-N6. Among them, computing nodes N1-N4 form a ring link 1, and computing nodes N4-N6 form a ring link 2. It is easy to understand that the same computing node can belong to only one ring link, such as computing nodes N1-N3, or computing nodes N5 and N6. Of course, the same computing node can also belong to different ring links at the same time, such as computing node N4. This is not limited here. Therefore, the function of neural network calculation exists in each computing node in the distributed training system.

[0037] It should be noted that the above-mentioned ring distributed network can be considered to be for a certain training task. That is to say, for any specific training task, there will only be one specific ring link. Specifically, different training tasks can be executed on different ring links. For example, training task A is executed on ring link 1, and training task B is executed on ring link 2. Furthermore, different training tasks can also be executed in different data transmission directions of the same ring link. Figure 1 As shown, the training task A can be performed in the data transmission direction 1 (counterclockwise direction) on the ring link, and the training task B can be performed in the data transmission direction 2 (clockwise direction) of the ring link 1 at the same time.

[0038] The computing nodes can be directly or indirectly connected via wired or wireless means, which is not limited here. It should be noted that for each computing node in any transmission direction of each ring link, there is an upstream adjacent node and a downstream adjacent node. For example, in the counterclockwise direction of ring link 1, the upstream adjacent node of computing node N4 is computing node N3, and the downstream adjacent node of computing node N4 is computing node N1.

[0039] For example, Figure 1As shown, the computing node N4 in the clockwise direction of the ring link 2 is taken as an example for explanation. After starting the training task, the computing node N4 sends the gradient data calculated by it to its downstream adjacent node N5. The computing node N5 aggregates the gradient value calculated by itself with the gradient data received from its upstream computing node N4, and then sends the aggregated gradient data to the downstream adjacent computing node N6 of the computing node N5. The computing node N6 aggregates the gradient data calculated by itself with the gradient data received from the computing node N5. So far, the gradient data calculated by all computing nodes in the clockwise direction of the ring link 2 have been aggregated. Finally, the computing node N6 also needs to send the gradient data after the gradient aggregation to the computing nodes N4 and N5 in turn in the clockwise direction of the ring link 2, so that each computing node calculates the new weight corresponding to the computing node according to the gradient data aggregated by the computing node N6, and completes the weight update of the neural network copy stored locally in each computing node. In other words, the gradient data obtained by each computing node in the loop will be accumulated through aggregation to update the weight. For example, when each computing node includes multiple gradient computing operators, the gradient data of the corresponding gradient computing operators in the multiple computing nodes will be accumulated to achieve aggregation and weight update.

[0040] Exemplarily, the computing nodes in the above-mentioned distributed training system can be network devices in a distributed network, such as large computers, servers, etc., or chips arranged in the above-mentioned network devices, such as central processing units (CPU), graphic processing units (GPU), neural network processing units (NPU), etc., which are not limited in this application. It should be understood that Figure 1 This is a simplified schematic diagram for ease of understanding. The distributed training system of the neural network may also include other computing nodes or other ring links. Figure 1 Not drawn in.

[0041] The following describes in detail the distributed training method of the neural network provided by the embodiment of the present application in conjunction with the accompanying drawings. Figure 2 A schematic diagram of a distributed training method for a neural network provided in an embodiment of the present application. The method can be applied to Figure 1 The distributed training system of the neural network shown is specifically used to perform the following Figure 3 The distributed training task of the neural network shown in FIG. The method is specifically run on each computing node in the distributed training system, such as N1.

[0042] like Figure 3As shown, the neural network located in any computing node includes: a forward computing operator, a loss computing operator, 5 gradient computing operators AE, and 5 gradient aggregation operators and 5 weight update operators corresponding to the above 5 gradient computing operators. Each operator represents an operation or operation in the neural network. All operators are combined to form the neural network of the computing node to perform one or more training iterations for the neural network. Among them, the gradient computing operator is used to calculate the gradient data amount of the gradient computing operator, for example, the gradient computing operator A obtains its gradient data amount a. The gradient aggregation operator is used to aggregate the gradient data amount obtained by the corresponding gradient computing operator with the corresponding corresponding gradient data amount obtained from the previous computing node, that is, to perform gradient accumulation. For example, the gradient aggregation operator A accumulates the gradient data amount a with the gradient data amount a' obtained by the corresponding gradient computing operator A in the previous one or more computing nodes to obtain the accumulated value. The weight update operator A updates the weight corresponding to the gradient computing operator A according to the accumulated value.

[0043] The following takes the operation of computing node N2 in ring link 1 as an example for explanation. Figure 2 As shown, the method may include S201-S204 to reduce the number and time of gradient aggregation in the system. S201, computing node N2 determines the gradient aggregation threshold of the neural network. In a possible design method, S201 may include the following steps: determining the gradient aggregation threshold according to a first corresponding relationship between the gradient data statistic and the gradient calculation time, and a second corresponding relationship between the gradient data statistic and the gradient aggregation time. Any of the above corresponding relationships is a functional relationship, such as a first functional relationship between the gradient data statistic and the gradient calculation time or a second functional relationship between the gradient data statistic and the gradient aggregation time.

[0044] Figure 4 An example of the first correspondence relationship and the second correspondence relationship is given. Figure 4 As shown in the figure, the gradient calculation curve is used to represent the corresponding relationship between the gradient data statistic (horizontal axis m) and the gradient calculation time (vertical axis t), that is, the first corresponding relationship, which reflects the functional change of the gradient calculation time corresponding to the gradient data statistic. The gradient aggregation curve is used to represent the corresponding relationship between the gradient data statistic (horizontal axis m) and the gradient aggregation time, that is, the second corresponding relationship, which reflects the functional change of the gradient aggregation time corresponding to the gradient data statistic. Figure 4As shown, as the gradient data statistic increases, the changing trend of the gradient calculation curve is different from the changing trend of the gradient aggregation curve, and there is an intersection between the two. At this intersection, the gradient calculation time is the same as the gradient aggregation time. In the present application, the gradient data statistic corresponding to the intersection can be selected as the gradient aggregation threshold. Therefore, optionally, the gradient calculation time corresponding to the above-mentioned gradient aggregation threshold is equal to the gradient aggregation time.

[0045] It should be noted that Figure 4 The gradient calculation curve and gradient aggregation curve shown are both continuous curves. In practical applications, the gradient data statistics and gradient calculation time in the gradient calculation curve, and the gradient data statistics and gradient aggregation time in the gradient aggregation curve are usually discrete values. Therefore, optionally, the above-mentioned determination of the gradient aggregation threshold according to the first corresponding relationship between the gradient data statistics and the gradient calculation time, and the second corresponding relationship between the gradient data statistics and the gradient aggregation time, may include the following steps: determining the preferred gradient data amount in the gradient data statistics, and determining the preferred gradient data amount as the gradient aggregation threshold. Among them, the time deviation between the gradient calculation time corresponding to the preferred gradient data amount and the gradient aggregation time is less than or equal to the time deviation threshold. It should be noted that the above-mentioned time deviation is usually a small value, such as not exceeding 5% of the gradient calculation time, or not exceeding 5% of the gradient aggregation time. For example, the gradient calculation time corresponding to the preferred gradient data amount and the gradient aggregation time may be the same, and at this time the time deviation threshold is equivalent to 0. It is easy to understand that it is also possible to use the fitting algorithm to obtain the above-mentioned discrete values. Figure 4 The continuous curve is then selected as the gradient aggregation threshold value.

[0046] In practical applications, Figure 4 The gradient calculation curve and the gradient aggregation curve in can be obtained by offline calculation or online calculation. Therefore, in a possible design method, the distributed training method of the neural network can also include the following steps: calculating the first corresponding relationship between the gradient data statistic and the gradient calculation time, and the second corresponding relationship between the gradient data statistic and the gradient aggregation time by online calculation or offline calculation.

[0047] Among them, the first correspondence between the gradient data statistics and the gradient calculation time consumed by online calculation refers to the use of different types of training samples (corresponding to different amounts of training sample data) to perform the gradient calculation operation of the neural network, and to count the amount of gradient data and the gradient calculation time consumed by each gradient calculation operator, and then to count the correspondence between the gradient data statistics and the gradient calculation time consumed by the neural network corresponding to different types of training samples.

[0048] The above-mentioned first correspondence between the gradient data statistics and the gradient calculation time consumed by offline calculation means that before receiving the current training task, based on the historical training records of different types of training samples (corresponding to different amounts of training sample data) in the process of executing the training task (including gradient calculation operations) of the neural network, the correspondence between the gradient data statistics of the neural network corresponding to each of the different types of training samples and the gradient calculation time consumed is obtained by statistics, and then pre-stored in a public storage space or locally on each computing node in the form of a spreadsheet, configuration file, executable script, software program, etc. for backup.

[0049] The second correspondence between the gradient data statistics and the gradient aggregation time consumed by online calculation refers to using simulated gradient data of different data amounts, such as random numbers, to perform neural network gradient aggregation operations, and to counting the correspondence between the simulated gradient data statistics of the neural network corresponding to the simulated gradient data of different data amounts and the gradient aggregation time consumed.

[0050] The second correspondence between the gradient data statistics and the gradient aggregation time consumed by offline calculation means that, before receiving the current training task, based on the historical training records of different types of training samples (corresponding to different amounts of training sample data) in the process of executing the training task (including gradient aggregation operation) of the neural network, the correspondence between the gradient data statistics of the neural network corresponding to different amounts of gradient data and the gradient aggregation time consumed is statistically obtained, and then pre-stored in a public storage space or locally on each computing node in the form of a spreadsheet, configuration file, executable script, software program, etc. for backup.

[0051] It should be noted that the above-mentioned offline calculation method can also be executed before the product corresponding to the computing node is actually launched on the market, for example, the manufacturer performs relevant development tests or pre-training operations during the debugging, testing or development stage to obtain the two corresponding relationships. The two corresponding relationships are pre-configured in the product by the manufacturer who produces or sells the product corresponding to the computing node, and are directly used to perform actual training during the subsequent use of the product. For example, when the neural network receives a training task, the preferred gradient data volume with a time deviation less than the time deviation threshold is determined by looking up the above-mentioned spreadsheet or configuration file. Optionally, the gradient calculation time corresponding to the preferred gradient data volume is equal to the gradient aggregation time.

[0052] In addition, for the neural network determined in the offline calculation method, the algorithm corresponding to each gradient calculation operator is determined, that is, relative to the training sample of the specified sample type, the amount of gradient data output by each gradient calculation operator is also determined. Therefore, the gradient data statistics in the above two corresponding relationships can also be replaced by the data type of the training sample or the data amount of the training sample. Correspondingly, the above gradient aggregation threshold can also be replaced by the data amount of a training sample, or the data amount or sample type of the training sample corresponding to each gradient data statistic can be added to the above two corresponding relationships. For example, the first corresponding relationship between the above gradient calculation time-consuming statistics and the gradient data statistics can be replaced by the corresponding relationship between the gradient calculation time-consuming statistics and the data amount of a training sample, or the corresponding relationship between the gradient calculation time-consuming statistics and the data type of a training sample, or the data amount or data type of a training sample can be added to the above first corresponding relationship. It is easy to understand that for the second corresponding relationship between the above gradient aggregation time-consuming statistics and the gradient data statistics, the same method as the above first corresponding relationship can also be used for adaptive processing, which will not be repeated here.

[0053] It should be noted that the above-mentioned online calculation method or offline calculation method for counting the first corresponding relationship and the second corresponding relationship can be used in combination. For example, the first corresponding relationship can be counted by online calculation method, and the second corresponding relationship can be counted by offline calculation method. For another example, the first corresponding relationship can be counted by offline calculation method, and the second corresponding relationship can be counted by online calculation method.

[0054] It can be understood that when the first correspondence relationship and the second correspondence relationship are obtained online or offline, the neural network used can be the one mentioned in the previous embodiment. Figure 3 The corresponding neural network, that is, the neural network that executes the distributed training method, can also be other neural networks similar to the above neural network, which can be used to simulate the Figure 3 The corresponding neural network is not limited in this embodiment.

[0055] Alternatively, after obtaining the gradient aggregation threshold by the above method, the gradient aggregation threshold can also be configured in the memory of the computing node or configured in the computing node as hardware logic. Therefore, computing node N2 determines the gradient aggregation threshold may include: obtaining a preconfigured gradient aggregation threshold from the computing node, for example, reading the gradient aggregation threshold from the memory, or obtaining the gradient aggregation threshold configured in the computing node in a hardware logic manner. It can be understood that in addition to obtaining the gradient aggregation threshold in the above-mentioned online or offline manner, those skilled in the art can set the gradient aggregation threshold according to experience, and configure the gradient aggregation threshold in the computing node, so that the computing node directly calls the gradient aggregation threshold and uses it when executing the method. This operation eliminates the need for the computing node to recalculate or simulate the gradient aggregation threshold multiple times, but directly preconfigures the appropriate gradient aggregation threshold in the computing node, thereby simplifying the calculation operation.

[0056] S202, the computing node N2 divides multiple gradient computing operators in the neural network into multiple gradient aggregation sets according to the gradient aggregation threshold, wherein each gradient aggregation set includes at least one gradient computing operator. Figure 3 A schematic diagram of the structure of a neural network before optimization provided in an embodiment of the present application. Figure 5 The schematic diagram of the structure of the optimized neural network provided in the embodiment of the present application is as follows. Figure 3 and Figure 5 , a method for dividing a gradient aggregation set provided in an embodiment of the present application is described in detail.

[0057] Assume that for a given type of training sample, such as a 2-megabyte image, Figure 3 The gradient calculation operators A, B, C, D, and E in the neural network shown in the figure correspond to the gradient data amounts a, b, c, d, and e, respectively. Then the gradient aggregation set can be determined according to the following steps 1 to 4: Step 1, starting from the input layer operator, identify all paths in the neural network according to the topological structure and data flow of the neural network. For example, Figure 3As shown, with the gradient calculation operator A as the starting point, the neural network has two paths: A->D->E and A->B->C. Step two, accumulate the gradient data of all identified paths and clear them to 0. Step three, according to the topological structure and data flow of the neural network, for each path, accumulate the gradient data of the traversed gradient calculation operators from front to back. For example, for the path A->D->E, the cumulative sum of the gradient data after the 1st, 2nd, and 3rd accumulations is: a, a+d, a+d+e. For another example, for the path A->B->C, the cumulative sum of the gradient data after the 1st, 2nd, and 3rd accumulations is: a, a+b, a+b+c. Step four, when there is at least one path whose corresponding cumulative sum of the gradient data is equal to or greater than the gradient aggregation threshold for the first time, all the gradient calculation operators traversed by any one of the above at least one path are divided into a gradient aggregation set. For example, as Figure 3 and Figure 5 As shown in , for the path A->D->E, if the cumulative sum of the gradient data of the gradient calculation operators A and D is less than the gradient aggregation threshold, and the cumulative sum of the gradient data a+d+e is equal to or greater than the gradient aggregation threshold, then Figure 5 As shown, the gradient calculation operators A, D, and E traversed by the path can be divided into gradient aggregation sets ADE. That is, the gradient calculation operators A, D, and E share a gradient aggregation operator ADE. After obtaining the gradient aggregation set, if there are still other gradient calculation operators on the path, the above process is repeated along the path to continue to divide the gradient aggregation set. Whenever the accumulated gradient data amount corresponding to at least one gradient calculation operator reaches the gradient aggregation threshold, the at least one gradient calculation operator is divided into a gradient aggregation set.

[0058] It should be noted that when traversing multiple paths at the same time, the gradient calculation operators traversed by different paths may have intersections, such as the above paths A->D->E and A->B->C both include gradient calculation operator A. Therefore, when the cumulative sum of the gradient data corresponding to at least one path is equal to or greater than the gradient aggregation threshold, in addition to dividing the gradient calculation operators traversed by any of the above at least one path into a gradient aggregation set, it is also necessary to subtract the gradient data corresponding to the gradient calculation operators of the above intersection from the cumulative sum of the gradient data of other paths, and delete the gradient calculation operators of the intersection from other paths. That is, when obtaining the gradient aggregation set, the calculation operators whose gradient data have been counted in the previous gradient aggregation set division will no longer be counted in the subsequent gradient aggregation set division. For example, assuming that the cumulative sum of the gradient data volume of the above path A->D->E is greater than or equal to the gradient aggregation threshold, the gradient calculation operators A, D, and E can be divided into gradient aggregation sets ADE, and the gradient calculation operator A in the path A->B->C is eliminated, and a is subtracted from the cumulative sum of the gradient data volume a+b+c. For another example, assuming that the cumulative sum of the gradient data volume of the path A->D->E and the path A->B->C is greater than or equal to the gradient aggregation threshold, the gradient calculation operators traversed by any path, such as the path A->B->C, can be divided into a gradient aggregation set ABC, and for other paths, such as the path A->D->E, the gradient calculation operator A in the intersection is deducted, and the gradient data volume a corresponding to the gradient calculation operator A in the intersection is subtracted from its cumulative sum of gradient data volume a+d+e. For another example, after the gradient calculation operators A, D, and E are divided into gradient aggregation sets ADE, the gradient aggregation sets are divided again for the subsequent multiple calculation operators on the path, and the gradient data accumulation sum a+d+e is no longer included, that is, the gradient data accumulation sum of the subsequent multiple calculation operators is compared with the gradient aggregation threshold again.

[0059] Based on the above analysis, further, in step 5, after completing the division of the gradient aggregation set, the starting gradient calculation operator of each path can be determined again according to the topological relationship of the remaining gradient calculation operators in the neural network, and the above steps 2 to 4 are repeated until all the gradient calculation operators in the neural network are uniquely divided into a certain gradient aggregation set. For example, Figure 3 and Figure 5As shown, after the gradient calculation operators A, D, and E are divided into the gradient aggregation set ADE, the above steps 2 to 4 can be repeated for the new path B->C or other gradient calculation operators on the path where the gradient calculation operators A, D, and E are located. For example, if the first cumulative sum of the amount of gradient data b of the path B->C is less than the gradient aggregation threshold, and the second cumulative sum of the amount of gradient data b+c of the path B->C is equal to or greater than the gradient aggregation threshold, the gradient calculation operators B and C are divided into the gradient aggregation set BC and share the same gradient aggregation operator BC.

[0060] It should be noted that when a path includes an output layer operator, and the output layer operator has also been traversed and accumulated, that is, all the gradient calculation operators included in the path have been traversed and accumulated, even if the cumulative sum of the gradient data of all the gradient calculation operators traversed by the path is still less than the gradient aggregation threshold, it is necessary to divide all the gradient calculation operators traversed by the path into a gradient aggregation set. For example, assuming that the gradient calculation operator C in the path B->C is the output layer operator, that is, the final operator on this path, even if the cumulative sum of the gradient data of the path B->C, b+c, is still less than the gradient aggregation threshold, it is necessary to divide the gradient aggregation operators B and C into a gradient aggregation set.

[0061] S203, computing node N2 starts the training task of the neural network. The training task is used to perform multiple rounds of training iterations on the neural network, and each round of training iteration is used to perform multiple gradient calculations corresponding to multiple gradient calculation operators. One round of training iteration refers to using a training sample to complete a gradient calculation, gradient aggregation and weight update of the neural network. In the present application, a gradient calculation in the above-mentioned multiple gradient calculations refers to a gradient calculation operator using a training sample for a gradient calculation operation. Therefore, for the above-mentioned gradient aggregation set ADE, in the process of using 1 training sample to complete a round of training iterations, 3 gradient calculations need to be completed, corresponding to 3 gradient calculation operators A, D, and E respectively.

[0062] The above-mentioned training sample can be a training data or a group of training data. For example, in a face recognition application, a training sample can be a face picture of 2 megabytes in size. For another example, in an identity recognition application, the above-mentioned training sample can also be a group of training data, such as a training sample including a 2-megabyte picture of a person with a known identity, such as a fingerprint pattern, a 1-minute voice or an iris pattern. In the present application, each computing node uses a training sample to complete a gradient calculation, gradient aggregation and weight update of the neural network, which is regarded as a training iteration.

[0063] S204, computing node N2 performs gradient aggregation on the gradient data of any gradient aggregation set. The gradient data is obtained by executing at least one gradient calculation corresponding to at least one gradient calculation operator in any gradient aggregation set, and the gradient aggregation is used to aggregate the above gradient data with other gradient data of other computing nodes in the distributed training system, and the other gradient data is the gradient data of the gradient aggregation set corresponding to any one of the above gradient aggregation sets in other computing nodes. In practical applications, the time for different gradient aggregation sets to complete a gradient calculation may be different. Therefore, optionally, if all gradient calculation operators in any gradient aggregation set complete a gradient calculation, the gradient aggregation set can be taken as a whole to perform a gradient aggregation operation. Combined Figure 1 and Figure 5 , assuming that computing nodes N1 and N2 have completed the gradient data calculation of all gradient calculation operators in the gradient aggregation set ADE, computing node N2 can start the gradient aggregation operation of the gradient aggregation set ADE regardless of whether computing nodes N1 and N2 have completed the gradient data calculation of all gradient calculation operators in the gradient aggregation set BC.

[0064] Executing a gradient aggregation operation on a gradient aggregation set includes executing gradient aggregation for each gradient calculation operator in the gradient aggregation set. For the gradient aggregation set ADE of any calculation node, taking calculation operator A as an example, the gradient data of calculation operator A of the calculation node will be accumulated with the corresponding gradient data transmitted from the previous calculation node to achieve aggregation. The corresponding gradient data transmitted from the previous calculation node belongs to the accumulated gradient data obtained by one or more calculation operators A in one or more previous calculation nodes. The gradient aggregation of calculation operators B and C is similar to this and will not be repeated. Therefore, the gradient aggregation operation for the gradient aggregation set ADE actually includes gradient aggregation for calculation operators A, B and C respectively.

[0065] Specifically, each computing node can set a gradient calculation completion indication for each gradient aggregation set in the neural network. When the upstream adjacent node completes a gradient calculation of a gradient aggregation set, the gradient calculation completion indication of the gradient aggregation set can be sent to the downstream adjacent node. Accordingly, the downstream adjacent node can obtain the gradient data of the gradient aggregation set from the upstream adjacent node according to the indication when it also completes a gradient calculation of the same gradient aggregation set, and completes a gradient aggregation operation between computing nodes with the gradient aggregation set as a whole.

[0066] It is easy to understand that in the present application, after two adjacent computing nodes have completed a gradient calculation of the entire neural network, the gradient aggregation operation of multiple gradient aggregation sets can be started simultaneously. Similarly, the upstream adjacent node can also send a gradient calculation completion indication of the neural network to the downstream adjacent node. Correspondingly, the downstream adjacent node can also obtain the gradient data of the gradient aggregation set from the upstream adjacent node according to the indication, and complete a gradient aggregation operation with the gradient aggregation set as a whole. For example, in combination Figure 1 and Figure 5 , assuming that computing nodes N1 and N2 have completed a gradient calculation of the entire neural network and sent a gradient calculation completion indication of the entire neural network to their respective downstream adjacent nodes, then after receiving the completion indication sent by computing node N1, computing node N2 obtains the gradient data of gradient aggregation sets ADE and BC from computing node N1, and performs a gradient aggregation operation on gradient aggregation sets ADE and BC respectively.

[0067] Optionally, the gradient calculation completion indication of the above-mentioned gradient aggregation set may include the storage location of the gradient data of the gradient aggregation set, such as the starting storage address and the size of the storage space. The storage location may be located in the memory of the computing node that sends the gradient calculation completion indication, or in the memory of the computing node that receives the gradient calculation completion indication, or in a third-party memory, which is not limited in this embodiment. Similarly, the gradient calculation completion indication of the above-mentioned neural network may also include the storage location of the gradient data of all gradient aggregation sets in the neural network, such as the starting storage address and the size of the storage space of the gradient data of each gradient aggregation set.

[0068] It should be noted that Figure 2 The distributed training method of the neural network shown is based on Figure 1 The example of the computing node N2 in the ring link 1 is used for explanation. It is easy to understand that the above method flow is also applicable to other computing nodes in the ring link 1, such as computing node N3, and will not be repeated here.

[0069] Easy to understand, such as Figure 1 In the ring link 1 shown, assuming that the training task uses 1000 training samples, each of the four computing nodes can use 250 training samples to perform 250 training iterations. In each training iteration, each computing node needs to aggregate each gradient set of the neural network, such as Figure 5The gradient aggregation sets ADE and BC in the figure each perform 250 gradient aggregation operations. Of course, for each training iteration, after the gradient data calculated by all computing nodes complete a gradient aggregation, such as the last computing node in the ring link 1 in the figure completes a gradient aggregation of the neural network, it is also necessary to transmit the aggregated gradient data to all other computing nodes on the ring link in sequence according to the data transmission direction of the ring link, so that each computing node on the ring link performs gradient splitting, new weight calculation and weight update of the neural network according to the aggregated gradient data and the divided gradient aggregation set, which will not be repeated here. After each computing node completes a round of training iterations based on the current training sample (including the updated new weights), it can read in the next training sample and perform the next round of training iterations. The specific operations of the next round of training iterations can be found in S201-S204 above, which will not be repeated here.

[0070] The distributed training method of a neural network provided in the present application can determine a gradient aggregation threshold before starting a training task, and then divide multiple gradient calculation operators in the neural network into multiple gradient aggregation sets each including at least one gradient calculation operator according to the gradient aggregation threshold, and when all gradient calculation operators in any gradient aggregation set complete a gradient calculation, the any gradient aggregation set is taken as a whole to perform a gradient aggregation operation, which can avoid the situation where the coordinator node needs to determine the gradient aggregation scheme for each training sample multiple times in the process of using multiple training samples to perform training tasks, and can effectively reduce the number of times and time consumption of determining the gradient aggregation scheme, thereby improving training efficiency.

[0071] In addition, in the present application, a gradient aggregation set can be taken as a whole, and continuous storage space can be allocated for the gradient data of all gradient calculation operators in the gradient aggregation set, so that all gradient data corresponding to the gradient aggregation set can be directly obtained in the gradient aggregation operation, so as to reduce the increase in the number of data transfers caused by the different storage locations of the gradient data of different gradient calculation operators in the same gradient aggregation set, which is conducive to reducing the time consumption of gradient aggregation, thereby improving training efficiency. This is because in the prior art, the gradient data of the same gradient aggregation set needs to come from a continuous storage space, so the above operation of allocating continuous storage space can avoid the secondary replication of different gradient data of different gradient calculation operators in the same gradient aggregation set, and there is no need to allocate additional sending buffers required for secondary replication, but directly use continuous storage space for operations, which can reduce the storage space occupied by gradient data without increasing the number of transmission times of gradient data, so as to reduce the cost of computing nodes, or use the saved storage space to process more gradient data, so as to further improve training efficiency.

[0072] Combination of the above Figure 2-Figure 5 The distributed training method of the neural network in the embodiment of the present application is described in detail. Figure 6 and Figure 7 The apparatus in the distributed training system of the embodiment of the present application is described in detail. Figure 6 A device for distributed training of a neural network provided in an embodiment of the present application. The device may be as follows Figure 1 The computing nodes shown can also be set at Figure 1 The communication chip, chipset or mainboard in the computing node shown may also be a software program or a module formed by a software program in the computing node, or a module formed by a combination of software and hardware, which is not limited here. The device includes: a determination module 601, a training module 602 and an aggregation module 603. Among them, the determination module 601 is used to determine the gradient aggregation threshold of the neural network, and according to the gradient aggregation threshold, multiple gradient calculation operators in the neural network are divided into multiple gradient aggregation sets. Among them, each gradient aggregation set includes at least one gradient calculation operator. The training module 602 is used to start the training task of the neural network. Among them, the training task is used to perform multiple rounds of training iterations on the neural network, and each round of training iteration is used to perform multiple gradient calculations corresponding to multiple gradient calculation operators. The aggregation module 603 is used to perform gradient aggregation for the gradient data of any gradient aggregation set. The gradient data is obtained by executing at least one gradient calculation corresponding to at least one gradient calculation operator in any one of the gradient aggregation sets, and the gradient aggregation is used to aggregate the above-mentioned gradient data with other gradient data of other computing nodes in the distributed training system, and the other gradient data is the gradient data of the gradient aggregation set corresponding to the above-mentioned any one of the gradient aggregation sets in other computing nodes.

[0073] In one possible design, the determination module 601 is also used to determine the gradient aggregation threshold value based on the first corresponding relationship between the gradient data statistic and the gradient calculation time, and the second corresponding relationship between the gradient data statistic and the gradient aggregation time. Optionally, the determination module 601 is also used to determine the preferred gradient data amount in the gradient data statistic, and to determine the preferred gradient data amount as the gradient aggregation threshold value. Among them, the time deviation between the gradient calculation time corresponding to the preferred gradient data amount and the gradient aggregation time is less than the time deviation threshold. Optionally, the gradient calculation time corresponding to the preferred gradient data amount is equal to the gradient aggregation time. In one possible design, the above-mentioned determination module 601 is used to calculate the first corresponding relationship between the gradient data statistic and the gradient calculation time, and the second corresponding relationship between the gradient data statistic and the gradient aggregation time by online calculation or offline calculation.

[0074] One or more of the above modules can be implemented by software, hardware or a combination of the two. When any of the above modules is implemented by software, the software exists in the form of computer program instructions and is stored in a memory, and the processor can be used to execute the program instructions to implement the above method flow. The processor may include but is not limited to at least one of the following: a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a microcontroller unit (MCU), or an artificial intelligence processor and other types of computing devices that run software, each of which may include one or more cores for executing software instructions to perform calculations or processing. The processor can be a separate semiconductor chip, or it can be integrated into a semiconductor chip with other circuits. For example, it can form a SoC (system on chip) with other circuits (such as a codec circuit, a hardware acceleration circuit or various buses and interface circuits), or it can also be integrated into the ASIC as a built-in processor of an application-specific integrated circuit (ASIC). The ASIC with an integrated processor can be packaged separately or with other circuits. In addition to the core for executing software instructions to perform operations or processing, the processor may further include necessary hardware accelerators, such as field programmable gate arrays (FPGA), PLDs (programmable logic devices), or logic circuits for implementing dedicated logic operations. When the above modules are implemented in hardware, the hardware can be any one or any combination of CPU, microprocessor, DSP, MCU, artificial intelligence processor, ASIC, SoC, FPGA, PLD, dedicated digital circuit, hardware accelerator or non-integrated discrete device, which can run the necessary software or not rely on software to execute the above method flow.

[0075] Figure 7 Another device in a distributed training system provided in an embodiment of the present application can be applied to Figure 1 The distributed training system of the neural network shown in FIG. The device may be a computing node or a component thereof, such as a chip, a chipset or a circuit board. Figure 7 As shown, the apparatus 700 includes: a processor 701, a transceiver 702 and a memory 703. The processor 701 is coupled to the transceiver 702 and the memory 703; the memory 703 is used to store a computer program. The processor 701 is used to execute the computer program stored in the memory 703, so that the apparatus 700 performs the following steps: Figure 2The functions of computing nodes in the distributed training method of the neural network shown in FIG.

[0076] In one possible design, the device 700 includes one or more processors and one or more transceivers. The one or more processors are configured to support the device 700 to perform the functions of computing nodes in the above-mentioned distributed training method of the neural network. For example, determine the gradient aggregation threshold, divide the multiple gradient calculation operators in the neural network into multiple gradient aggregation sets according to the gradient aggregation threshold, and when all the gradient calculation operators in any gradient aggregation set complete the gradient calculation, perform the gradient aggregation of any gradient aggregation set for the calculated gradient data, etc. The transceiver is used to support the device 700 to communicate with other devices and realize the receiving and / or sending functions. For example, receive training tasks, receive gradient data sent by upstream adjacent nodes, send gradient data to downstream adjacent nodes, receive training samples, etc.

[0077] Optionally, the device 700 may further include one or more memories, which are coupled to the processor and are used to store program instructions and / or data necessary for the device 700. The one or more memories may be integrated with the processor or may be separated from the processor, which is not limited in this application. Figure 1 The computing node shown in the figure, the transceiver may be a transceiver circuit of the computing node. Optionally, the transceiver may also be an input / output circuit or interface of the computing node. The apparatus 700 may also be arranged at Figure 1 The communication chip or chipset in the computing node shown. The transceiver may be an input / output circuit or interface of the communication chip.

[0078] To facilitate further description Figure 7 The working principle of the device 700 is, illustratively, Figure 8 The schematic diagram of the structure of the training hardware and software based on the tensor flow framework provided in the embodiment of the present application specifically describes Figure 7 The software may be installed in the computing nodes N1-N6 and may be specifically run on a processor to execute the distributed training method of the neural network described in the above method embodiment. Figure 8As shown, the software includes modules such as an application interface (API), a graph optimizer module, a graph execution module, an operator layer, a collection communication library, and a neural network acceleration library. Among them, the API is used to provide an access interface to the neural network for the upper-level user application, such as receiving training tasks, receiving recognition tasks, and outputting instructions for recognition results. The graph optimization module is used to optimize and adjust the neural network defined by the user through the application, such as determining the gradient aggregation threshold online or offline, and dividing the gradient calculation operators in the neural network defined by the application into multiple gradient aggregation sets according to the gradient aggregation threshold to generate an optimized neural network and save it in the memory of the computing node, such as the above-mentioned memory 703. The graph execution module, that is, the training execution module, is used to perform gradient calculation operations and gradient aggregation operations on any one or more gradient aggregation sets in the optimized neural network. The operator layer is used to call the calculation interface according to the dependency relationship between different gradient calculation operators in the optimized neural network during the process of the graph execution module performing the gradient calculation operation, such as calling the neural network acceleration library to calculate the gradient data corresponding to one or more gradient calculation operators. The specific example of the operator layer can be referred to Figure 3 or Figure 5 . The collection communication library is used to obtain the gradient data calculated by other computing nodes when the graph execution module performs a gradient aggregation operation with the gradient data corresponding to one or more gradient aggregation sets as a whole. For example, the gradient data corresponding to the one or more gradient aggregation sets of the upstream adjacent nodes can be obtained through a network card or a high-speed peripheral device interconnect (PCI-E) bus. The neural network acceleration library is used to provide a computing operation interface, such as calling a neural network computing chip, such as calling the above-mentioned processor 701 to implement a gradient calculation operation, when the graph execution module performs a gradient calculation operation. Among them, the network card and the PCI-E bus can be Figure 7 The specific implementation of the transceiver 702 in the embodiment of the present invention is as follows: Figure 7 The specific implementation of the processor 701. In addition to the network card, PCI-E bus and neural network computing chip are hardware, Figure 8 Other modules are implemented in software and drive related hardware. Figure 8 The specific method flow completed by the illustrated architecture may refer to the previous method embodiments.

[0079] The present application provides a distributed training system for a neural network, which includes one or more ring links as mentioned above, each ring link includes multiple computing nodes. The present application provides a readable storage medium, which stores a program or instruction, and when the program or instruction is run on a computer, the computer executes the distributed training method for a neural network described in the above method embodiment. The present application provides a computer program product, including a computer program code, which, when the computer program code is run on a computer, enables the computer to execute the distributed training method for a neural network described in the above method embodiment.

[0080] It should be understood that the processor in the embodiment of the present application may be a CPU, and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0081] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), high bandwidth memory (HBM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM).

[0082] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0083] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0084] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean one of the following items: a; b; c; a and b; a and c; b and c; a, b, and c, where a, b, and c can be single or multiple.

[0085] It should be understood that in various embodiments of the present application, the size of the sequence number of each process does not mean the order of execution, and the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but this implementation should not be considered to be beyond the scope of the present application.

[0086] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the systems, devices and units described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0087] If the functions described in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0088] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A distributed training method for a neural network, characterized in that: include: Determine the gradient aggregation threshold of the neural network; According to the gradient aggregation threshold, a plurality of gradient calculation operators in the neural network are divided into a plurality of gradient aggregation sets; wherein each gradient aggregation set includes at least one gradient calculation operator; Starting a training task for the neural network; wherein the training task is used to perform multiple rounds of training iterations on the neural network, each round of training iteration is used to perform multiple gradient calculations corresponding to the multiple gradient calculation operators, and the training samples used in each round of training iteration are one or more of the following: pictures, voices; Performing gradient aggregation on the gradient data of any gradient aggregation set; wherein the gradient data is obtained by executing at least one gradient calculation corresponding to at least one gradient calculation operator in the any gradient aggregation set, and the gradient aggregation is used to aggregate the gradient data with other gradient data of other computing nodes in the distributed training system, and the other gradient data is the gradient data of the gradient aggregation set corresponding to the any gradient aggregation set in the other computing nodes; Determining the gradient aggregation threshold of the neural network includes: Determining a preferred gradient data amount in the gradient data statistics; wherein a time deviation between a gradient calculation time and a gradient aggregation time corresponding to the preferred gradient data amount is less than or equal to a time deviation threshold; The preferred gradient data amount is determined as the gradient aggregation threshold.

2. The distributed training method of a neural network according to claim 1, characterized in that: The gradient calculation time corresponding to the preferred gradient data volume is equal to the gradient aggregation time.

3. The distributed training method for a neural network according to claim 1 or 2, characterized in that: Also includes: A first corresponding relationship between the gradient data statistic and the gradient calculation time consumption, and a second corresponding relationship between the gradient data statistic and the gradient aggregation time consumption are statistically calculated through an online calculation method or an offline calculation method.

4. A device in a distributed training system, characterized in that: include: Determine module, training module and aggregation module; wherein, The determination module is used to determine the gradient aggregation threshold of the neural network; The determination module is further used to divide the multiple gradient calculation operators in the neural network into multiple gradient aggregation sets according to the gradient aggregation threshold; wherein each gradient aggregation set includes at least one gradient calculation operator; The training module is used to start the training task of the neural network; wherein the training task is used to perform multiple rounds of training iterations on the neural network, each round of training iteration is used to perform multiple gradient calculations corresponding to the multiple gradient calculation operators, and the training samples used in each round of training iteration are one or more of the following: pictures, voices; The aggregation module is used to perform gradient aggregation on the gradient data of any gradient aggregation set, wherein the gradient data is obtained by performing at least one gradient calculation corresponding to at least one gradient calculation operator in the any gradient aggregation set, and the gradient aggregation is used to aggregate the gradient data with other gradient data of other computing nodes in the distributed training system, and the other gradient data is the gradient data of the gradient aggregation set corresponding to the any gradient aggregation set in the other computing nodes; Among them, the determination module is also used to determine a preferred gradient data amount in the gradient data statistics, and determine the preferred gradient data amount as the gradient aggregation threshold; the time deviation between the gradient calculation time corresponding to the preferred gradient data amount and the gradient aggregation time is less than the time deviation threshold.

5. The device according to claim 4, characterized in that The gradient calculation time corresponding to the preferred gradient data volume is equal to the gradient aggregation time.

6. The device according to claim 4 or 5, characterized in that The determination module is used to calculate the first corresponding relationship between the gradient data statistics and the gradient calculation time, and the second corresponding relationship between the gradient data statistics and the gradient aggregation time by online calculation or offline calculation.

7. A device in a distributed training system, characterized in that: include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the device performs the distributed training method of the neural network according to any one of claims 1 to 3.

8. A readable storage medium, characterized in that: A program or instruction is stored, and when the program or instruction is executed on a computer or a processor, the computer or the processor executes the distributed training method for a neural network as claimed in any one of claims 1 to 3.

9. A computer program product, characterized in that The method comprises a computer program code, and when the computer program code is run on a computer or a processor, the computer or the processor executes the distributed training method for a neural network according to any one of claims 1 to 3.

Images