Gradient sparsification method and gradient sparsification device for neural networks

Abstract

The application provides a gradient sparsification method and related device of a neural network in the field of artificial intelligence. The application proposes to calculate a sparse threshold by using a data operation method, or to compare an actual sparsity with a preset sparsity and update the sparse threshold based on the comparison result, thereby avoiding using a sorting operation to determine the sparse threshold, greatly reducing the operation amount, and ultimately improving the training efficiency of the neural network. Further, the application proposes to update the sparse threshold only once every m rounds of iteration, so that the sparse threshold update time of m-1 rounds can be saved for every m rounds of training, thereby improving the training efficiency of the neural network, and m is an integer greater than 1. Further, the application also proposes that the sparse threshold update and gradient sparsification processing can be processed in parallel, thereby improving the training efficiency of the neural network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a gradient sparseness method and device of a neural network. BACKGROUND

[0002] With the continuous development of artificial intelligence (AI) in recent years, artificial intelligence has caused a new round of innovation and change in the world, and has begun to promote the development and transformation of human beings from the information age to the intelligent age. At the same time, new changes are also constantly occurring in the field of artificial intelligence. The development of artificial intelligence technology gradually moves from the academic stage to the experimental stage, and continues to develop towards the industrialization stage of the cooperation between academia and industry.

[0003] The key technology of artificial intelligence is neural network. Neural network simulates the connection of human brain nerve cells, and connects a large number of simple processing units (called neurons) to form a complex network system. At present, large deep neural networks generally use a distributed training method to speed up the training. During the training process, frequent gradient transmission is needed between nodes in the distributed training system to synchronize the model parameters on each node.

[0004] At present, when using the distributed training method to train the neural network, on the one hand, the node needs to be configured with a high-bandwidth network, which increases the hardware cost; on the other hand, as the number of nodes in the distributed training system increases, the linear degree of acceleration of the distributed training decreases greatly, which seriously affects the performance of the distributed training. In order to solve the above problems, in the training process of the neural network, the following method is proposed: the gradient is first sparsely processed by each node in the distributed training system and then transmitted, so as to save bandwidth and accelerate the training.

[0005] At present, when each node in the distributed training system sparsely processes the gradient, the gradient to be transmitted each time is sorted, then the K largest absolute values are selected, and the remaining gradient values are set to zero, so as to realize gradient sparseness.

[0006] In the above gradient sparseness method, since the gradient needs to be sorted each time, and the sorting operation takes a lot of time, the efficiency of gradient sparseness processing is low, thereby affecting the training efficiency of the neural network. SUMMARY

[0007] The present application provides a gradient sparseness method and device of a neural network, which can improve the training efficiency of the neural network.

[0008] In a first aspect, the present application provides a gradient sparsification method of a neural network. The gradient sparsification method is applied to a distributed training system, the distributed training system being configured to perform N rounds of iterative training on the neural network, and the distributed training system comprising M training devices, each of the M training devices being configured to deploy the neural network, wherein N and M are positive integers greater than 1.

[0009] The method comprises: a first training device of the M training devices updating a sparsity threshold in a memory only in an i th round of iterative training of the N rounds of iterative training, the i th round of iterative training being performed on the neural network by the first training device, the sparsity threshold being updated according to a current gradient set, the current gradient set comprising a parameter gradient calculated by the first training device in the i th round of iterative training, i being a positive integer less than N, and i being an integer multiple of m, m being an integer less than N and greater than 1; and the first training device performing sparsity processing on a gradient in the current gradient set using the sparsity threshold in the memory in the i th round of iterative training, to obtain the sparse gradient.

[0010] In the method, the sparsity threshold is not updated in each round of iterative training, but is updated every m rounds, so that the time for updating the sparsity threshold in m-1 rounds of iterative training can be saved every m rounds, thereby improving the training efficiency of the neural network.

[0011] In combination with the first aspect, in a first possible implementation, the first training device updates the sparsity threshold in the memory according to the current gradient set in the i th round of iterative training, and performs the sparsity processing on the gradient in the current gradient set using the sparsity threshold in the memory in the i th round of iterative training in parallel.

[0012] In the method of the implementation, the sparsity threshold used by the first training device for gradient sparsification in the current round of iterative training is the sparsity threshold calculated in the previous round, and the gradient sparsification is not performed after the new sparsity threshold is updated in the current round, so that the updating of the sparsity threshold and the gradient sparsification can be processed in parallel in the current round, thereby reducing the time delay of the gradient sparsification, and further improving the training efficiency of the neural network.

[0013] In a second possible implementation manner of the first aspect or the first possible implementation manner, the first training device of the M training devices updates the sparse threshold in the memory according to the current gradient set only in the i th iteration training of the N iteration trainings, including: the first training device only in the i th iteration training, calculates a new sparse threshold according to a current gradient mean value, a historical gradient mean value, a current gradient standard deviation, a historical gradient standard deviation and the sparse threshold in the memory, and updates the sparse threshold in the memory as the new sparse threshold, the current gradient mean value is a mean value of gradients in the current gradient set, the historical gradient mean value is a mean value of gradients in a historical gradient set, the historical gradient set includes parameter gradients used by the first training device to calculate the sparse threshold in the memory before updating, the current gradient standard deviation is a standard deviation of gradients in the current gradient set, the historical gradient standard deviation is a standard deviation of gradients in the historical gradient set, and a mapping relationship between the sparse threshold in the memory before updating and the historical gradient mean value is the same as a mapping relationship between the sparse threshold after updating in the memory and the current gradient mean value.

[0014] In the method of the present implementation, because the gradient is no longer sorted when calculating the sparse threshold, but the current round sparse threshold is calculated based on the previous sparse threshold by using a data operation method, the time and computing resources used for sorting a large number of gradients can be saved, the efficiency of calculating the sparse threshold can be improved, the efficiency of gradient sparsification can be improved, and finally the training efficiency of the neural network can be improved.

[0015] In the third possible implementation manner of the second possible implementation manner, the new sparse threshold, the current gradient mean value, the historical gradient mean value, the current gradient standard deviation, the historical gradient standard deviation and the sparse threshold in the memory before updating satisfy the following relationship:

[0016]

[0017] wherein, th new represents the new sparse threshold, th represents the sparse threshold in the memory before updating, μ represents the historical gradient mean value, μ new represents the current gradient mean value, σ represents the historical gradient standard deviation, σ new represents the current gradient standard deviation.

[0018] In a fourth possible implementation manner of the first aspect or the first possible implementation manner, the first training device in the M training devices updates the sparse threshold in the storage according to a proportion of target gradients in a current gradient set only in the i-th iteration training in the N iteration trainings, including: the first training device updates the sparse threshold in the storage according to the proportion of the target gradients in the current gradient set only in the i-th iteration training, wherein if the proportion is greater than or equal to a first threshold, the first training device decreases the sparse threshold in the storage, and if the proportion is less than or equal to a second threshold, the first training device increases the sparse threshold in the storage, the current gradient set includes a parameter gradient calculated by the first training device for the neural network in the i-th iteration training, and the target gradient includes a gradient less than or equal to the sparse threshold in the storage before being updated.

[0019] In the implementation manner, because the gradient is no longer sorted when the sparse threshold is calculated, but the actual sparsity is calculated, the actual sparsity is compared with the preset sparsity, and the sparse threshold is updated based on the comparison result, the time and the computing resource used for sorting a large number of gradients can be saved, the efficiency of calculating the sparse threshold can be improved, the efficiency of performing the gradient sparsity processing can be improved, and finally the training efficiency of the neural network can be improved.

[0020] The first threshold or the second threshold is the preset sparsity, and the proportion of the target gradient is the actual sparsity.

[0021] In a fifth possible implementation manner of the fourth possible implementation manner, the first training device updates the sparse threshold in the storage according to the proportion of the target gradients in the current gradient set only in the i-th iteration training, including: the first training device generates a sparse mask matrix corresponding to the current gradient set according to the current gradient set and the sparse threshold in the storage, a value of each element in the sparse mask matrix is 1, indicating that a gradient corresponding to the each element is greater than the sparse threshold in the storage, and a value of the each element is 0, indicating that the gradient corresponding to the each element is less than or equal to the sparse threshold in the storage; and the first training device calculates the proportion according to a number of elements with a value of 0 in the sparse mask matrix, wherein the following relationship is satisfied among a sum of all elements in the sparse mask matrix, a number of all elements in the sparse mask matrix, and the proportion:

[0022] sparse ratio = 1 - sum(mask) / size(mask)

[0023] If sparse ratio - sr > ε, th newth = th - a*th; if sparse ratio - sr < -e, then th = th + a*th new th = th - a*th; if sparse ratio - sr < -e, then th = th + a*th

[0024] wherein, sparse ratio represents a proportion of elements with value 0 in the sparse mask matrix, sum(mask) represents a sum of all elements in the sparse mask matrix, size(mask) represents a number of all elements in the sparse mask matrix, th new represents the new sparse threshold, th represents the sparse threshold in the memory before updating, sr is a preset sparsity, a is a step size of updating the threshold each time, and e is a preset hyperparameter.

[0025] wherein, the sum of sr and e is an example of the first threshold, and the difference obtained by subtracting e from sr is an example of the second threshold.

[0026] In a second aspect, a gradient sparsification method of a neural network is provided. The gradient sparsification method is applied to a distributed training system, and the distributed training system is configured to perform N rounds of iterative training on the neural network. The distributed training system includes M training devices, and each of the M training devices is configured to deploy the neural network. N and M are positive integers greater than 1.

[0027] The method includes: in the i th round of iterative training, a first training device in the M training devices updates a sparse threshold in a memory according to a current gradient set, the current gradient set including a parameter gradient calculated by the first training device for the neural network in the i th round of iterative training, i is a positive integer less than or equal to N, and i is greater than 1; in the i th round of iterative training, the first training device performs sparse processing on the gradients in the current gradient set using the sparse threshold in the memory to obtain sparse gradients; and in the i th round of iterative training, the first training device updates the sparse threshold in the memory according to the current gradient set, and the sparse processing of the first training device on the gradients in the current gradient set using the sparse threshold in the memory is performed in parallel.

[0028] In the method, because the sparse threshold used by the first training device for gradient sparsification in the current round of iteration is the sparse threshold calculated in the previous round, the gradient sparsification does not have to wait until a new sparse threshold is obtained in the current round before performing sparse processing on the gradients. Therefore, the updating of the sparse threshold and the gradient sparsification can be performed in parallel in the current round, thereby reducing the time delay of the gradient sparsification and improving the training efficiency of the neural network.

[0029] With reference to the second aspect, in a first possible implementation manner, the first training device updates the sparse threshold in the memory according to the current gradient set, including: the first training device calculates a new sparse threshold according to a current gradient mean, a historical gradient mean, a current gradient standard deviation, a historical gradient standard deviation, and the sparse threshold in the memory, and updates the sparse threshold in the memory as the new sparse threshold, the current gradient mean is a mean of gradients in the current gradient set, the historical gradient mean is a mean of gradients in a historical gradient set, the historical gradient set includes parameter gradients used by the first training device to calculate the sparse threshold in the memory before updating, the current gradient standard deviation is a standard deviation of gradients in the current gradient set, the historical gradient standard deviation is a standard deviation of gradients in the historical gradient set, and a mapping relationship between the sparse threshold in the memory before updating and the historical gradient mean is the same as a mapping relationship between the sparse threshold after updating in the memory and the current gradient mean.

[0030] In the present implementation manner, because the gradient is no longer sorted when calculating the sparse threshold, but the current round of sparse threshold is calculated by using a data operation method based on the previous sparse threshold, the time and computing resources used for sorting a large number of gradients can be saved, so that the efficiency of calculating the sparse threshold can be improved, and the efficiency of gradient sparsification can be improved, and finally the training efficiency of the neural network can be improved.

[0031] With reference to the first possible implementation manner, in a second possible implementation manner, the new sparse threshold, the current gradient mean, the historical gradient mean, the current gradient standard deviation, the historical gradient standard deviation, and the sparse threshold in the memory before updating satisfy the following relationship:

[0032]

[0033] wherein, th new represents the new sparse threshold, th represents the sparse threshold in the memory before updating, μ represents the historical gradient mean, μ new represents the current gradient mean, σ represents the historical gradient standard deviation, σ new represents the current gradient standard deviation.

[0034] In a third possible implementation manner, in combination with the second aspect, the first training device updates the sparse threshold in the memory according to a proportion of target gradients in the current gradient set, including: updating the sparse threshold in the memory according to a proportion of target gradients in the current gradient set, wherein if the proportion is greater than or equal to a first threshold, the first training device decreases the sparse threshold in the memory, and if the proportion is less than or equal to a second threshold, the first training device increases the sparse threshold in the memory, the target gradient including a gradient less than or equal to the sparse threshold in the memory before updating.

[0035] In the implementation manner, because when the sparse threshold is calculated, the gradients are no longer sorted, but the current actual sparsity and the preset sparsity are compared, and the sparse threshold is updated based on the comparison result, the time and the computing resources used for sorting a large number of gradients can be saved, the efficiency of calculating the sparse threshold can be improved, the efficiency of gradient sparsity processing can be improved, and finally the training efficiency of the neural network can be improved.

[0036] The first threshold or the second threshold is a preset sparsity, and the proportion of the target gradient is an actual sparsity.

[0037] In combination with the third possible implementation manner, in a fourth possible implementation manner, the first training device updates the sparse threshold in the memory according to a proportion of target gradients in the current gradient set, including: the first training device generates a sparse mask matrix corresponding to the current gradient set according to the current gradient set and the sparse threshold in the memory, each element in the sparse mask matrix has a value of 1, indicating that a gradient corresponding to the element is greater than the sparse threshold in the memory, and the element has a value of 0, indicating that the gradient corresponding to the element is less than or equal to the sparse threshold in the memory; the first training device calculates the proportion according to a number of elements with a value of 0 in the sparse mask matrix, wherein the following relationship is satisfied between a sum of all elements in the sparse mask matrix, a number of all elements in the sparse mask matrix, and the proportion:

[0038] sparse ratio = 1 - sum(mask) / size(mask)

[0039] If sparse ratio-sr > ε, th = th - α*th; if sparse ratio-sr < -ε, th = th + α*th. new If sparse ratio-sr > ε, th = th - α*th; if sparse ratio-sr < -ε, th = th + α*th. new If sparse ratio-sr > ε, th = th - α*th; if sparse ratio-sr < -ε, th = th + α*th.

[0040] wherein, the sparse ratio represents a proportion of elements with a value of 0 in the sparse mask matrix, sum(mask) represents a sum of all elements in the sparse mask matrix, size(mask) represents a number of all elements in the sparse mask matrix, th represents the sparse threshold in the memory new wherein, th represents the new sparse threshold, th represents the sparse threshold in the memory before being updated, sr is a preset sparsity, and a is a step length of updating the threshold each time, and e is a preset hyperparameter.

[0041] wherein, the sum of sr and e is an example of the first threshold, and the difference between sr and e is an example of the second threshold.

[0042] With reference to the second aspect or any possible implementation manner of the above, in a fifth possible implementation manner, the i is an integer multiple of m, and m is an integer greater than 1 and less than N.

[0043] In the implementation manner, because the sparse threshold does not need to be updated in each iteration, but is updated every m rounds, the time for updating the sparse threshold in m-1 rounds of iteration training can be saved every m rounds, so that the training efficiency of the neural network can be improved.

[0044] In a third aspect, the present application provides a gradient sparsification method of a neural network. The gradient sparsification method is applied to a distributed training system, the distributed training system being configured to perform N rounds of iterative training on the neural network, the distributed training system comprising M training devices, each of the M training devices being configured to deploy the neural network, N and M being positive integers greater than 1, the method comprising: in the i th round of iterative training of the N rounds of iterative training, the first training device calculating a new sparsity threshold based on a current gradient mean, a historical gradient mean, a current gradient standard deviation, a historical gradient standard deviation, and a sparsity threshold in a memory, and updating the sparsity threshold in the memory to the new sparsity threshold, the current gradient mean being a mean of gradients in a current gradient set, the historical gradient mean being a mean of gradients in a historical gradient set, the historical gradient set comprising parameter gradients used by the first training device to calculate the sparsity threshold in the memory before the update, the current gradient set comprising parameter gradients calculated by the first training device for the neural network in the i th round of iterative training, the current gradient standard deviation being a standard deviation of the gradients in the current gradient set, the historical gradient standard deviation being a standard deviation of the gradients in the historical gradient set, a mapping relationship between the sparsity threshold in the memory before the update and the historical gradient mean being the same as a mapping relationship between the updated sparsity threshold in the memory and the current gradient mean, i being a positive integer less than or equal to N; in the i th round of iterative training, the first training device performing sparsity processing on the gradients in the current gradient set using the sparsity threshold in the memory to obtain sparse gradients.

[0045] In the method, because the gradients are not sorted when calculating the sparsity threshold, but the current round of sparsity threshold is calculated based on the previous sparsity threshold using a data operation method, the time and computing resources used for sorting a large number of gradients can be saved, the efficiency of calculating the sparsity threshold can be improved, the efficiency of performing gradient sparsity processing can be improved, and finally the training efficiency of the neural network can be improved.

[0046] In combination with the third aspect, in a first possible implementation, the new sparsity threshold, the current gradient mean, the historical gradient mean, the current gradient standard deviation, the historical gradient standard deviation, and the sparsity threshold in the memory before the update satisfy the following relationship:

[0047]

[0048] wherein th new represents the new sparsity threshold, th represents the sparsity threshold in the memory before the update, μ represents the historical gradient mean, μ newdenotes the current gradient mean value, and σ denotes the historical gradient standard deviation, σ new denotes the current gradient standard deviation.

[0049] With reference to the third aspect or the first possible implementation manner of the first aspect, in a second possible implementation manner, the first training apparatus calculates a new sparse threshold according to the current gradient mean value, the historical gradient mean value, the current gradient standard deviation, the historical gradient standard deviation, and the sparse threshold in the memory, and performs the sparse processing on the gradients in the current gradient set in parallel with the first training apparatus using the sparse threshold in the memory.

[0050] In this implementation manner, because the first training apparatus uses the sparse threshold calculated in the previous round for the gradient sparsification in the current round iteration, and does not have to wait until the new sparse threshold is updated in the current round before performing the sparse processing on the gradients, the updating of the sparse threshold and the gradient sparsification can be processed in parallel in the current round, thereby reducing the time delay of the gradient sparsification, and further improving the training efficiency of the neural network.

[0051] With reference to the third aspect or any of the possible implementation manners of the third aspect, in a third possible implementation manner, the i is an integer multiple of m, and m is an integer greater than 1 and less than N.

[0052] In this implementation manner, because the sparse threshold is not updated in each round of iteration, but is updated once every m rounds, the time for updating the sparse threshold in m-1 rounds of iteration can be saved every m rounds, thereby improving the training efficiency of the neural network.

[0053] In a fourth aspect, the present application provides a gradient sparsification method of a neural network. The gradient sparsification method is applied to a distributed training system, the distributed training system being configured to perform N rounds of iterative training on the neural network, the distributed training system comprising M training devices, each of the M training devices being configured to deploy the neural network, N and M being positive integers greater than 1. The method comprises: in the i th round of iterative training, updating, by the first training device, a sparsity threshold in a memory according to a proportion of target gradients in a current gradient set, wherein if the proportion is greater than or equal to a first threshold, the first training device decreases the sparsity threshold in the memory, and if the proportion is less than or equal to a second threshold, the first training device increases the sparsity threshold in the memory, the current gradient set comprising parameter gradients calculated by the first training device for the neural network in the i th round of iterative training, the target gradients comprising gradients less than or equal to the sparsity threshold in the memory before being updated, i being a positive integer less than N and greater than 1, and N being a positive integer; and in the i th round of iterative training, performing, by the first training device, sparsity processing on the gradients in the current gradient set using the sparsity threshold in the memory to obtain sparse gradients.

[0054] In this implementation, because the gradients are no longer sorted when calculating the sparsity threshold, but the actual sparsity is calculated, the actual sparsity is compared with the preset sparsity, and the sparsity threshold is updated based on the comparison result, so that the time and computing resources used for sorting a large number of gradients can be saved, the efficiency of calculating the sparsity threshold can be improved, the efficiency of performing gradient sparsity processing can be improved, and finally the training efficiency of the neural network can be improved.

[0055] In this implementation, the first threshold or the second threshold is the preset sparsity, and the proportion of the target gradients is the actual sparsity.

[0056] In combination with the fourth aspect, in a first possible implementation, the first training device only updates the sparsity threshold in the memory according to the proportion of the target gradients in the current gradient set in the i th round of iterative training, comprising: generating, by the first training device, a sparse mask matrix corresponding to the current gradient set according to the current gradient set and the sparsity threshold in the memory, each element in the sparse mask matrix having a value of 1 indicating that the gradient corresponding to the element is greater than the sparsity threshold in the memory, and each element in the sparse mask matrix having a value of 0 indicating that the gradient corresponding to the element is less than or equal to the sparsity threshold in the memory; and calculating, by the first training device, the proportion according to the number of elements with a value of 0 in the sparse mask matrix, wherein the sum of all elements in the sparse mask matrix, the number of all elements in the sparse mask matrix, and the proportion satisfy the following relationship:

[0057] sparse ratio = 1 - sum(mask) / size(mask)

[0058] If sparse ratio - sr > ε, then th new = th - a * th; If sparse ratio - sr < -ε, then th new = th + a * th;

[0059] wherein sparse ratio represents a proportion of elements with a value of 0 in the sparse mask matrix, sum(mask) represents a sum of all elements in the sparse mask matrix, size(mask) represents a number of all elements in the sparse mask matrix, th new represents the new sparse threshold, th represents a sparse threshold in the memory before updating, sr is a preset sparsity, a is a step length of updating the threshold each time, and ε is a preset hyperparameter.

[0060] wherein the sum of sr and ε is an example of the first threshold, and the difference between sr and ε is an example of the second threshold.

[0061] With reference to the fourth aspect or the first possible implementation manner, in a second possible implementation manner, the first training apparatus updates the sparse threshold in the memory according to a proportion of target gradients in the current gradient set, and the first training apparatus performs, in parallel, the sparse processing of the gradients in the current gradient set using the sparse threshold in the memory in the i-th round of iterative training.

[0062] In this implementation manner, because the first training apparatus uses the sparse threshold calculated in the previous round for the sparse processing of the gradients in the current round of iteration, the first training apparatus can perform the updating of the sparse threshold and the sparse processing of the gradients in parallel in the current round, thereby reducing the time delay of the sparse processing of the gradients and improving the training efficiency of the neural network.

[0063] With reference to the fourth aspect or any of the possible implementation manners described above, in a third possible implementation manner, the i is an integer multiple of m, and m is an integer greater than 1 and less than N.

[0064] In a fifth aspect, the present application provides a training apparatus of a neural network, which can include various functional modules for implementing the method in any of the first aspect to the fourth aspect or any of the possible implementation manners, and these functional modules can be implemented in a software and / or hardware manner.

[0065] In a sixth aspect, the present application provides a training device of a neural network, which can include a processor coupled with a memory. The memory is configured to store program codes, and the processor is configured to execute the program codes in the memory to implement the method in any one of the first aspect to the fourth aspect or any possible implementation manner thereof.

[0066] Optionally, the device can further include the memory.

[0067] When the device is an electronic device, in some implementation manners, the device can further include a transceiver configured to communicate with other devices.

[0068] When the device is a chip for an electronic device, in some implementation manners, the device can further include a communication interface configured to communicate with other devices in the electronic device, for example, to communicate with a transceiver of the electronic device.

[0069] In a seventh aspect, the present application provides a computer readable storage medium storing program codes for execution by a processor, the program codes including instructions for implementing the method in any one of the first aspect to the fourth aspect or any possible implementation manner thereof.

[0070] In an eighth aspect, the present application provides a computer program product including instructions, which, when executed on a processor, cause the processor to implement the method in any one of the first aspect to the fourth aspect or any possible implementation manner thereof. BRIEF DESCRIPTION OF DRAWINGS

[0071] Figure 1 a schematic diagram of a distributed training system provided by an embodiment of the present application;

[0072] Figure 2 a schematic structural diagram of a training device of an embodiment of the present application;

[0073] Figure 3 a schematic flowchart of a gradient sparsification method of a neural network of an embodiment of the present application;

[0074] Figure 4 a schematic flowchart of a gradient sparsification method of a neural network of another embodiment of the present application;

[0075] Figure 5 a schematic flowchart of a gradient sparsification method of a neural network of yet another embodiment of the present application;

[0076] Figure 6 a schematic flowchart of a gradient sparsification method of a neural network of another embodiment of the present application;

[0077] Figure 7A schematic structural diagram of a training device according to another embodiment of the present application. DETAILED DESCRIPTION

[0078] In order to better introduce the embodiments of the present application, the related concepts in the embodiments of the present application are introduced first as follows.

[0079] 1. Neural network

[0080] Artificial intelligence (AI) is the theory, method, technology and application system for using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

[0081] The key technology of today's artificial intelligence is neural network (NN). Neural network simulates the connection of nerve cells in the brain, and connects a large number of simple processing units (called neurons) extensively to form a complex network system.

[0082] A simple neural network contains three layers, namely input layer, output layer and hidden layer (also called intermediate layer), and the connection between each layer corresponds to a weight (the value is called weight, parameter). Neural network has excellent performance in the fields of computer vision, natural language processing, etc., because the weight is adjusted by training algorithm, so that the prediction result of neural network is optimal.

[0083] The training of neural network generally includes two calculation steps, the first step is forward calculation, and the second step is backward calculation. Among them, the forward calculation is that the input value and the parameter are calculated, and then the output value is generated through a nonlinear function. The output value is either the final output of the network or will be used as the subsequent input value to continue similar calculation. The deviation between the output value of the network and the actual label value of the corresponding sample is measured by a model loss function, which is expressed as a function f(x, w) of input sample x and network parameter W. In order to minimize the loss function, the parameter W of the network needs to be adjusted constantly, and the backward calculation is to obtain the updated value of the parameter W. In the algorithm based on gradient descent, the backward calculation starts from the last layer of the neural network, calculates the partial derivative of the loss function with respect to the parameter of each layer, and finally obtains the partial derivative of all parameters, which is called gradient. In each iteration, the parameter W is updated in the opposite direction of the gradient with a certain step size η to obtain a new parameter W, that is, one step of training is completed. The updating process is represented by the following formula:

[0084]

[0085] where w t denotes the parameter used at the t-th iteration, w t+1 denotes the updated parameter, η is called learning rate, B t denotes the sample set input at the t-th iteration.

[0086] The process of training a neural network is also the process of learning the weights corresponding to the neurons, and the ultimate goal is to obtain the weights corresponding to each layer of neurons of the trained neural network model.

[0087] 2. Distributed training

[0088] The method of training using multiple training nodes. Distributed training is usually divided into two parallelization methods: data parallelism and model parallelism.

[0089] In the model parallelism mode, the layers and parameters of the model will be distributed on multiple nodes, and the model will pass through multiple cross-node communications in the forward and backward training of a mini-batch. Each node only saves a part of the entire model.

[0090] In the data parallelism mode, each node saves the layers and parameters of the complete model, each node completes the forward and backward calculations independently, and then completes the aggregation of the gradients and synchronously updates the parameters on all nodes.

[0091] 3. Gradient

[0092] The gradient is a vector that represents the directional derivative of a function at that point along the direction that takes the maximum value, that is, the function changes most quickly along the direction of the gradient (the direction of the gradient) at that point, and the rate of change is the largest. In model training, the gradient is used to find the minimum loss function, train the model, and accelerate the convergence of the model. The number of model training times is the number of gradient steps.

[0093] 4. Gradient sparsity

[0094] Gradient sparsity is to reduce the amount of data actually communicated by sparsifying and further encoding and compressing gradient data, thereby alleviating the communication bandwidth limitation problem of distributed training. The general method of gradient sparsity is to use a threshold as an indication of sparsity, and when the amplitude of the gradient is less than the threshold, it is set to zero. Only the gradients with amplitudes greater than the threshold will be transmitted. Generally, the gradient of the model contains a large number of small amplitude values, and these values have little effect on the update of the model parameters. Setting these values to zero hardly affects the final accuracy of the model. Generally, 95% to 99% of the gradient values can be set to zero while maintaining the accuracy.

[0095] The embodiments of the application will be described in detail below with reference to the accompanying drawings.

[0096] Figure 1 A schematic diagram of a distributed training system is provided for an embodiment of the present application. As shown, the application scenario is described by way of example as shown in Figure 1 Figure 1 The neural network distributed training system mainly consists of multiple training devices, which can communicate with each other. The training device in this embodiment can be any device, system or chip with computing capability, etc.

[0097] Figure 2 A schematic structural diagram of the training device of an embodiment of the present application is shown. As shown, the training device 200 can include an AI chip 201. The AI chip 201 can include a sparse module 202 for performing gradient sparsification, a data compression module 203, an interface 204 for communication, a decompression module 205, a gradient accumulation module 206 and a cache unit 207. Figure 2

[0098] The AI chip 201 can include a matrix calculation unit (AI cube) and a vector calculation unit. The AI chip can be a neural network processing unit (NPU) and / or a graphics processing unit (GPU), etc. The sparse module, the data compression module, the decompression module and the gradient accumulation module can be implemented by a central processing unit (CPU) or other types of processors.

[0099] The AI chip 201 is the main calculation unit for training neural networks, mainly used for performing forward and backward calculations in neural network training. The matrix calculation unit is mainly used to complete forward and backward calculations of neural network training and store the calculated gradients in the cache unit. The vector calculation unit is mainly used to read gradient data from the matrix calculation unit, further calculate and correct the gradient, and send the result to the sparse module. In addition, the vector calculation unit is also used to update the sparse threshold based on the gradient and send the updated threshold to the sparse module.

[0100] The sparse module 202 is mainly used to read gradient data and perform sparse operations on the gradient data. For example, the sparse module sets the gradient below the threshold to zero according to the sparse threshold configured in the register, and then sends the sparse gradient to the data compression module.

[0101] The data compression module 203 is mainly used to encode and compress the sparse gradient data, and then send it to the communication interface. ​​

[0102] Communication interface 204 is used to send and receive data.

[0103] The decompression module 205 is mainly used to decode and decompress the gradient data received by the communication interface 204 and then send it to the gradient accumulation module 206.

[0104] The gradient accumulation module 206 is mainly used to accumulate the decoded sparse gradient with the local sparse gradient.

[0105] The most prevalent distributed training method currently is data parallelism. Figure 1 Taking the distributed training system shown as an example, the distributed training method mainly involves synchronizing the same network parameters W on multiple training devices. In each iteration, each training device reads a batch of sample data and calculates the corresponding gradient. To ensure that the parameter update values ​​on each training device are the same, the gradients calculated by each training device need to be aggregated and synchronized to all training devices in each iteration. Then, each training device updates its local weights w based on the aggregated gradient. The deviation between the network's output value and the actual label value of the corresponding sample is measured by the model loss function, which is expressed as a function f(x,w) of the input sample x and the network parameters W, with its partial derivative being ▽f(x,w). This weight update process can be expressed as the following formula:

[0106]

[0107] Where x is the input value (also called the activation value), b represents the number of input sample groups, and w t w represents the parameter used in the t-th iteration. t+1 The updated parameters are represented by η, where η is called the learning rate, N represents the number of training devices, and B... k,t Let represent the set of input samples for the k-th device in the t-th iteration.

[0108] During the training process of a neural network, each training device in a distributed training system can first perform sparse processing on the gradients before transmission, in order to save bandwidth and accelerate training.

[0109] In existing technologies, when each training device performs gradient sparsity compression, firstly, each training device copies the same neural network parameters and stores them locally. Then, each device reads a batch of training samples and obtains the model's gradient data through forward and backward computation. Subsequently, each training device sorts its local gradient values ​​and determines a sparsity threshold based on the sorting results. Gradient values ​​below the sparsity threshold are set to 0, resulting in the sparse gradient. Finally, each training device encodes and compresses the sparse gradient and sends it to other training devices through interconnection devices such as network interface cards, while simultaneously receiving gradients sent from other training devices.

[0110] In the above gradient sparsification method, since the sorting operation of the gradient is required each time, the time consumption is large, and thus the efficiency of the gradient sparsification processing is low, thereby affecting the training efficiency of the neural network.

[0111] To solve the technical problem of low efficiency of the gradient sparsification processing, an embodiment of the present application provides a scheme of adaptive dynamic adjustment of the threshold value, which greatly reduces the calculation amount, and the threshold value calculation and updating operation can be asynchronously executed with other operations, thereby avoiding the additional time consumption introduced by the threshold value calculation and updating.

[0112] Figure 3 A schematic flowchart of the gradient sparsification method of the neural network of an embodiment of the present application. As shown in the figure, the method can include S301 and S302. Figure 3

[0113] Figure 3 The method shown in the figure can be applied to a distributed training system for N-round iterative training of a neural network, the distributed training system including M training devices, each of the M training devices deploying the same neural network, and N and M being positive integers greater than 1. N can be pre-set or determined by a training end condition set in the training device.

[0114] S301, in the i-th round of iterative training in the N-round iterative training, a first training device calculates a new sparse threshold value according to a current gradient mean value, a historical gradient mean value, a current gradient standard deviation, a historical gradient standard deviation, and a sparse threshold value in a memory of the first training device, and updates the sparse threshold value in the memory to the new sparse threshold value, the current gradient mean value being a mean value of gradients in a current gradient set, the historical gradient mean value being a mean value of gradients in a historical gradient set, the historical gradient set including a parameter gradient used by the first training device to calculate the sparse threshold value in the memory before updating, the current gradient set including a parameter gradient calculated by the first training device for the neural network in the i-th round of iterative training, the current gradient standard deviation being a standard deviation of the gradients in the current gradient set, the historical gradient standard deviation being a standard deviation of the gradients in the historical gradient set, a mapping relationship between the sparse threshold value in the memory before updating and the historical gradient mean value being the same as a mapping relationship between the sparse threshold value in the memory after updating and the current gradient mean value, and i being a positive integer less than or equal to N.

[0115] ​For example, the vector calculation unit in the AI chip in the first training device reads the sparse threshold, the historical gradient mean and the historical gradient standard deviation from the register, calculates the current gradient mean and the current gradient standard deviation based on the current gradient set calculated by the matrix calculation unit, and calculates the new sparse threshold based on the read sparse threshold, the historical gradient mean, the historical gradient standard deviation and the calculated current gradient mean and current gradient standard deviation.

[0116] In this embodiment, the first training device can be any one of the M training devices. As an example, any one of the M training devices is the first training device, in other words, each of the M training devices performs the same operation as the first training device.

[0117] As an example, the sparse threshold in the memory before updating can be the sparse threshold calculated by the first training device in the i-mth iteration training. At this time, the historical gradient set is a set of gradients calculated by the first training device in the i-mth iteration training, m is a positive integer, and m is less than or equal to N.

[0118] S302, in the i-th iteration training, the first training device uses the sparse threshold in the memory to perform sparse processing on the gradients in the current gradient set to obtain sparse gradients.

[0119] For example, the sparse module in the first training device reads the sparse threshold from the memory of the first training device, and performs sparse processing on the current gradient set calculated by the matrix calculation unit based on the sparse threshold.

[0120] The implementation of the first training device using the sparse threshold read from the memory to perform sparse processing on the gradients in the current gradient set and the subsequent processing flow can refer to the prior art.

[0121] For example, the sparse module can set the gradient values smaller than the sparse threshold in the current gradient set to 0, and send the sparse gradients to the data compression module; the data compression module encodes and compresses the sparse gradients, and sends the compressed data to the communication interface; the communication interface sends the compressed data to other training devices; after the communication interface of the first training device receives the data sent by other training devices, the decompression module restores the sparse gradients on other training devices and sends them to the gradient accumulation module, and the gradient accumulation module accumulates the local sparse gradients of the first training device with the gradients from other training devices, and stores the accumulation result in the cache unit. Steps S301 and S302 can be performed on each training device until the gradient aggregation on each training device is completed, and then each training device updates the network parameters of the local neural network with the accumulated gradients.

[0122] Because the embodiment calculates the sparse threshold in the current round of data operation method based on the previous sparse threshold instead of sorting the gradients, the time and computing resources used for sorting a large number of gradients can be saved, thereby improving the efficiency of calculating the sparse threshold, and further improving the efficiency of gradient sparsification, and finally improving the training efficiency of the neural network.

[0123] In the embodiment, when the first training device calculates the new sparse threshold based on the previous sparse threshold, the current gradient mean and the historical gradient mean, the mapping relationship between the previous sparse threshold and the historical gradient mean should be the same as the mapping relationship between the new sparse threshold and the current gradient mean.

[0124] As an example, the mapping relationship between the previous sparse threshold and the historical gradient mean can be The mapping relationship between the new sparse threshold and the current gradient mean is also Where th represents the new sparse threshold, th represents the previous sparse threshold, μ represents the historical gradient mean, μ new represents the current gradient mean, σ represents the historical gradient standard deviation, and the historical gradient standard deviation is the standard deviation of the gradients in the historical gradient set, σ new represents the current gradient standard deviation, and the current gradient standard deviation is the standard deviation of the gradients in the current gradient set. new

[0125] Based on the above mapping relationship, when calculating the new sparse threshold, the new sparse threshold, the current gradient mean, the historical gradient mean and the previous sparse threshold can satisfy the following relationship:

[0126]

[0127] It can be understood that the above relationship is only an example, and the relationship obtained by any linear transformation of the above relationship belongs to the protection scope of the present application.

[0128] In some implementations of the embodiment, S301 and S302 can be executed in series, that is, the first training device executes S302 after executing S301. That is, the first training device uses the updated sparse threshold to perform gradient sparsification after updating the sparse threshold in the i-th round.

[0129] ​In another implementation of the embodiment, S301 and S302 can be performed in parallel, i.e., the first training device can perform S301 to update the sparse threshold while using the existing sparse threshold in the memory to perform gradient sparsification. That is, the first training device can use the sparse threshold calculated in the previous round when performing gradient sparsification in the i-th round, and the sparse threshold updated in the i-th round can be used for gradient sparsification in the following rounds.

[0130] In this implementation, because the first training device uses the sparse threshold calculated in the previous round to perform gradient sparsification in the current round, the gradient sparsification can be performed in parallel with the updating of the sparse threshold, thereby reducing the time delay of gradient sparsification and improving the training efficiency of the neural network.

[0131] In one implementation of the embodiment, the first training device can use the method in S301 to update the sparse threshold only in part of the round iterations. In this implementation, because the updating of the sparse threshold is not required in each round, the training efficiency of the neural network can be further improved.

[0132] As an example, the sparse threshold can be obtained using the sorting method in the prior art in the first round of iteration, and the method in S301 can be used to calculate and update the sparse threshold in the iteration of all rounds from the second round to the N-th round.

[0133] As another example, the first training device can use the method in S301 to calculate and update the sparse threshold every m rounds. In this example, i can be an integer multiple of m, i.e., when the training round is an integer multiple of m, the method in S301 is used to update the sparse threshold in the training round. m is an integer greater than 1, and m is less than N. The size of m can be set according to experience.

[0134] In this example, because the sparse threshold is not updated in each round of iteration, but is updated only every m rounds, the time for updating the sparse threshold in m-1 rounds of iteration can be saved every m rounds, thereby improving the training efficiency of the neural network.

[0135] Figure 4 The schematic flowchart of the gradient sparsification method of the neural network of another embodiment of the present application is shown in FIG. 4. Figure 4 As shown in FIG. 4, the method can include S401 and S402.

[0136] Figure 4The method shown can be applied to a distributed training system for training a neural network for N rounds of iterative training, the distributed training system comprising M training devices, each of the M training devices deploying the same neural network, and N and M being positive integers greater than 1. N can be pre-set or determined by a training end condition set in the training device.

[0137] S401, in the i th round of iterative training in the N rounds of iterative training, the first training device updates the sparse threshold in the memory according to the proportion of the target gradient in the current gradient set, wherein if the proportion is greater than or equal to a first threshold, the first training device reduces the sparse threshold in the memory, and if the proportion is less than or equal to a second threshold, the first training device increases the sparse threshold in the memory, the current gradient set contains the parameter gradient calculated by the first training device for the neural network in the i th round of iterative training, the target gradient includes the gradient less than or equal to the sparse threshold in the memory before updating, i is a positive integer less than N, and i is greater than 1, and N is a positive integer.

[0138] For example, Figure 7 The threshold updating module 208 in the first training device reads the sparse threshold from the register, calculates a new sparse threshold based on the mask matrix corresponding to the current gradient set obtained from the sparse module, and writes the new sparse threshold into the memory to update the sparse threshold in the memory, or sends the new sparse threshold to the sparse module, and the sparse module writes the new sparse threshold into the memory.

[0139] In this embodiment, the proportion of the target gradient can be referred to as the current actual sparsity. In this embodiment, the first training device can be any one of the M training devices. As an example, any one of the M training devices is the first training device, in other words, each of the M training devices performs the same operation as the first training device.

[0140] As an example, the sparse threshold before updating in the memory can be the sparse threshold calculated by the first training device in the iteration training before the i th round.

[0141] For example, the sparse threshold before updating in the memory can be the sparse threshold calculated by the first training device in the i-1 th round of iterative training.

[0142] In this embodiment, the first threshold is greater than the second threshold. The first threshold and the second threshold can be pre-set based on experience.

[0143] S402, in the i-th round of iterative training, the first training device uses the sparse threshold in the memory to perform sparse processing on the gradients in the current gradient set, to obtain sparse gradients.

[0144] Because the embodiment no longer sorts the gradients when calculating the sparse threshold, but compares the current actual sparsity and the preset sparsity, and updates the sparse threshold based on the comparison result, the time and computing resources used for sorting a large number of gradients can be saved, the efficiency of calculating the sparse threshold can be improved, the efficiency of gradient sparse processing can be improved, and finally the training efficiency of the neural network can be improved.

[0145] In the embodiment, when the first training device determines the proportion of the target gradient in the current gradient set according to the sparse threshold in the memory, in one implementation, the first training device can compare the gradients in the current gradient set with the sparse threshold in the memory one by one, and count the total number of gradients in the current gradient set that are less than or equal to the sparse threshold in the memory, and calculate the proportion of the total number to the total number of gradients in the current gradient set.

[0146] As an example, the first training device can generate a sparse mask matrix corresponding to the current gradient set according to the current gradient set and the sparse threshold in the memory, where the value of each element in the sparse mask matrix is 1, indicating that the gradient corresponding to the element is greater than the sparse threshold in the memory, and the value of each element is 0, indicating that the gradient corresponding to the element is less than or equal to the sparse threshold in the memory; then, the first training device can calculate the proportion according to the number of elements with a value of 0 in the sparse mask matrix.

[0147] For example, the proportion of the target gradient in the current gradient set can be obtained in the following way: the first training device generates a sparse mask matrix corresponding to the current gradient set, where the value of each element in the sparse mask matrix is initialized to 1; according to the sparse threshold in the memory, compare the size of each gradient in the current gradient set with the sparse threshold in the memory one by one, if the compared gradient is greater than the sparse threshold in the memory, set the value of the element corresponding to the gradient in the sparse mask matrix to 0. The final sparse mask matrix obtained has the following characteristics: the value of the element corresponding to the gradient greater than the sparse threshold in the memory is 1, and the value of the element corresponding to the gradient less than or equal to the sparse threshold in the memory is 0.

[0148] One of the relationships that should be satisfied between the sum of all elements in the sparse mask matrix, the number of all elements in the sparse mask matrix, and the proportion is as follows:

[0149] sparse ratio = 1 - sum(mask) / size(mask)

[0150] wherein, sparse ratio represents a proportion of elements with value 0 in the sparse mask matrix, sum(mask) represents a sum of all elements in the sparse mask matrix, and size(mask) represents a number of all elements in the sparse mask matrix.

[0151] It can be understood that the above relationship is only an example, for example, the number of elements with value 1 in the sparse matrix can be counted directly, and then the aforementioned proportion is calculated according to the counting result and the number of gradients in the current gradient set.

[0152] In the embodiment, an implementation manner of updating the sparse threshold in the memory based on the proportion of the target gradient is as follows:

[0153] If sparse ratio-sr > ε, th new = th-α*th; if sparse ratio-sr < -ε, th new = th+α*th.

[0154] wherein, th new represents the new sparse threshold, th represents the sparse threshold before updating in the memory, sr is a preset sparsity, α is a step length of updating the threshold each time, and ε is a preset hyperparameter. The value of α can also be preset.

[0155] In the embodiment, the sum of sr and ε is an example of the first threshold, and the difference obtained by subtracting ε from sr is an example of the second threshold.

[0156] In some implementation manners of the embodiment, S401 and S402 can be executed in series, that is, the first training device executes S402 after executing S401. That is, the first training device uses the updated sparse threshold to perform gradient sparsification after updating the sparse threshold in the i-th round.

[0157] In another implementation manner of the embodiment, S401 and S402 can be executed in parallel, that is, the first training device can use the existing sparse threshold in the memory to perform gradient sparsification while executing S401 to update the sparse threshold. That is, the first training device can use the sparse threshold calculated in the previous round to perform gradient sparsification in the i-th round, and the sparse threshold updated in the i-th round is used for gradient sparsification in the subsequent round.

[0158] In this implementation, because the first training device uses the sparse threshold calculated in the previous round for gradient sparsification in the current round of iteration, the update of the sparse threshold and the gradient sparsification can be processed in parallel in the current round, thereby reducing the time delay of gradient sparsification and improving the training efficiency of the neural network.

[0159] In one implementation of the embodiment, the first training device can use the method in S401 to update the sparse threshold only in part of the round iteration training. In this implementation, because the update of the sparse threshold is not required in each round, the training efficiency of the neural network can be further improved.

[0160] As an example, in the first round of iteration training, the sparse threshold can be obtained using the sorting method in the prior art, and in the iteration training in all rounds from the second round to the Nth round, the sparse threshold can be calculated and updated using the method in S401.

[0161] As another example, the first training device can calculate and update the sparse threshold using the method in S401 every m rounds. In this example, i can be an integer multiple of m, that is, when the training round is an integer multiple of m, the sparse threshold is updated using the method in S401 in the training round. m is an integer greater than 1, and m is less than N. The size of m can be set empirically.

[0162] In this example, because the sparse threshold is not updated in each round of iteration, but is updated only every m rounds, the time for updating the sparse threshold in m-1 rounds of iteration can be saved every m rounds, thereby improving the training efficiency of the neural network.

[0163] Figure 5 The figure shows the schematic flowchart of the gradient sparsification method of the neural network of another embodiment of the present application. As shown in the figure, the method can include S501 and S502. Figure 5

[0164] Figure 5 The method shown in the figure can be applied to a distributed training system for N rounds of iteration training of a neural network. The distributed training system includes M training devices, each of which is deployed with the same neural network. N and M are positive integers greater than 1. N can be pre-set or determined by the training end condition set in the training device.

[0165] ​S501, in the i th iteration training in N iterations of training, the first training device updates the sparse threshold in the memory according to the current gradient set, the current gradient set contains the parameter gradient calculated by the first training device for the neural network in the i th iteration training, i is a positive integer less than or equal to N, and i is greater than 1.

[0166] In some implementations of the embodiment, the implementation of updating the sparse threshold according to the current gradient set can refer to the related content in S301, or can refer to the related content in S401, or can also use the method in the prior art, for example, the sparse threshold is obtained by sorting.

[0167] S502, in the i th iteration training, the first training device uses the sparse threshold in the memory to perform sparse processing on the gradient in the current gradient set to obtain sparse gradient, and the first training device updates the sparse threshold in the memory according to the current gradient set, which is performed in parallel with the first training device using the sparse threshold in the memory to perform sparse processing on the gradient in the current gradient set.

[0168] For example, the sparse module in the first training device reads the sparse threshold from the memory of the first training device, and performs sparse processing on the current gradient set calculated by the matrix calculation unit based on the sparse threshold.

[0169] The implementation of the first training device using the sparse threshold read from the memory to perform sparse processing on the gradient in the current gradient set and the subsequent processing flow can refer to the prior art.

[0170] The first training device updates the sparse threshold in the memory according to the current gradient set, which is performed in parallel with the first training device using the sparse threshold in the memory to perform sparse processing on the gradient in the current gradient set, which can be understood as that the first training device uses the sparse threshold updated in the iteration round before the i th round to perform gradient sparse processing in the i th round, and calculates a new sparse threshold in the i th round, which is used for gradient sparse processing in the iteration round after the i th round.

[0171] In the method of the embodiment, because the first training device uses the sparse threshold calculated in the previous round to perform gradient sparse processing in the current round, it is not necessary to wait until a new sparse threshold is updated in the current round before performing gradient sparse processing, so the update of the sparse threshold and the gradient sparse processing in the current round can be processed in parallel, thereby reducing the time delay of gradient sparse processing, and further improving the training efficiency of the neural network.

[0172] In some implementations of the embodiment, in the i th iteration training, the first training device can use the method in S501 to calculate and update the sparse threshold every m rounds. In this example, i can be an integer multiple of m, that is, when the training round is an integer multiple of m, the sparse threshold is updated using the method in S501 at this training round. m is an integer greater than 1, and m is less than N. The size of m can be set empirically.

[0173] In the implementation, because the sparse threshold does not need to be updated every iteration, but is updated only every m rounds, the time for updating the sparse threshold in m-1 rounds of iteration can be saved every m rounds, thereby improving the training efficiency of the neural network.

[0174] Figure 6 The schematic flowchart of the gradient sparsification method of the neural network of another embodiment of the present application is shown in FIG. 6. As shown in FIG. 6, the method can include S601 and S602. Figure 6

[0175] Figure 6 The method shown in FIG. 6 can be applied to a distributed training system for N rounds of iteration training of a neural network, the distributed training system including M training devices, each of the M training devices deploying the same neural network, and N and M being positive integers greater than 1. N can be pre-set, or can be determined by a training end condition set in the training device.

[0176] In S601, the first training device only updates the sparse threshold in the memory according to the current gradient set in the i th iteration training of the N rounds of iteration training, the current gradient set including the parameter gradient calculated by the first training device for the neural network in the i th iteration training, i is a positive integer less than N, and i is an integer multiple of m, m is an integer less than N and greater than 1.

[0177] In the embodiment, the first training device only updates the sparse threshold in the memory according to the current gradient set in the i th iteration training of the N rounds of iteration training, and i is an integer multiple of m, m is greater than 1. It can be understood that in the N rounds of iteration training, the first training device updates the sparse threshold every m rounds. The size of m can be set empirically.

[0178] That is, in the method of the embodiment, in the N rounds of iteration training, when the iteration round is an integer multiple of m, the sparse threshold can be updated and gradient sparsification processing can be performed at the iteration round; when the iteration round is not an integer multiple of m, the sparse threshold can not be updated, and only gradient sparsification processing can be performed.

[0179] ​In this embodiment, the implementation method of updating the sparse threshold according to the current gradient set can refer to the relevant content in S301 or S401, and will not be repeated here.

[0180] S602, in the i-th iteration of training, the first training device uses the sparse threshold in the memory to perform sparse processing on the gradients in the current gradient set to obtain sparse gradients.

[0181] In this embodiment, since the sparse threshold does not need to be updated in every iteration, but only once every m iterations, the time for updating the sparse threshold in m-1 iterations of training can be saved every m iterations, thereby improving the training efficiency of the neural network.

[0182] In some implementations of this embodiment, when the iteration number is an integer multiple of m (i.e., i is an integer multiple of m), both gradient sparsity processing and sparse threshold updating are performed. In this implementation, updating the sparse threshold and gradient sparsity processing can be performed along the path, i.e., sequentially; or, updating the sparse threshold and gradient sparsity processing can be performed in parallel, i.e., the sparse threshold used for gradient sparsity processing is the sparse threshold calculated before the current iteration, and the updated sparse threshold for the current iteration is used for gradient sparsity processing in subsequent iterations.

[0183] In the current iteration, the first training device uses the sparse threshold calculated in the previous iteration to perform gradient sparsity, without having to wait for the current iteration to update the sparse threshold before performing gradient sparsity processing. In other words, the update of the sparse threshold and gradient sparsity processing can be performed in parallel in this iteration, which can reduce the latency of gradient sparsity and thus improve the training efficiency of the neural network.

[0184] Figure 7 This is a schematic structural diagram of a training device according to an embodiment of this application. Figure 7 As shown, the AI ​​chip 201 in the training device 700 may include a sparse module 202, a data compression module 203, an interface 204, a decompression module 205, a gradient accumulation module 206, a cache unit 207, and a threshold update module 208. Figure 7 The meanings of modules 201 to 207 can be found in [reference]. Figure 2 The meaning of the same modules in the text will not be elaborated here.

[0185] The threshold update module 208 is used to: receive a sparse mask from the sparse module 202, calculate a new threshold based on the sparse mask, and send the new threshold to the sparse module 208.

[0186] The terms "first", "second", and "third" and the like in the description and in the claims of the present application and the specification herein are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the descriptive terms used herein are to be interpreted in the context as a special embodiment.

[0187] In the embodiments of the present application, the words "exemplary" and "for example" are used to mean serving as an example, instance, or illustration, at 99 2 2 least with respect to the particular aspects of the disclosure being discussed. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application is not necessarily to be construed as preferred or advantageous over other embodiments or design schemes. Indeed, any embodiment or design scheme described as "exemplary" or "for example" is to be interpreted as an example and is not necessarily to be interpreted as preferred or advantageous over other embodiments or design schemes. Moreover, it is appreciated that any embodiment or design scheme described in the embodiments of the present application is not necessarily to be interpreted as preferred or advantageous over other embodiments or design schemes.

[0188] In the embodiments of the present application, "at least one" means one or more, "multiple" means two or two more. The "and / or" describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the existence of A alone, the existence of A and B together, and the existence of B alone, where A and B can be singular or plural. In the textual description of the present application, the character " / " generally represents that the associated objects before and after are in an "or" relationship.

Claims

1. A gradient sparsification method of a neural network, characterized by, The gradient sparse method is applied to a distributed training system in the field of artificial intelligence technology, the distributed training system is used for N-round iterative training of the neural network, the distributed training system comprises M training devices, each of the M training devices is deployed with the neural network, and the training device comprises a matrix calculation unit, a threshold updating module, a vector calculation unit, a memory, a sparse module and a communication interface; N and M are positive integers greater than 1, and the method comprises: In the i th round of iterative training in the N-round iterative training: The matrix calculation unit of the first training device calculates a current gradient set of the i th round of iterative training; the first training device is any one of the M training devices; The vector calculation unit of the first training device calculates a new sparse threshold according to a current gradient mean, a current gradient standard deviation of the current gradient set, and a historical gradient mean, a historical gradient standard deviation and a sparse threshold read from the memory of the first training device, and updates the sparse threshold in the memory to the new sparse threshold; wherein the current gradient is a gradient in the current gradient set, and the historical gradient is a gradient in the historical gradient set; Or, the threshold updating module of the first training device updates the sparse threshold in the memory according to the size relationship between the proportion of the target gradient in the current gradient set and the threshold, and the proportion of the target gradient is the proportion of the total number of gradients less than or equal to the sparse threshold in the memory before updating in the current gradient set and the total number of gradients in the current gradient set; Wherein, the current gradient set contains the parameter gradient calculated by the matrix calculation unit of the first training device for the neural network in the i th round of iterative training, and i is a positive integer less than N; In the i th round of iterative training, the sparse module of the first training device reads the sparse threshold from the memory of the first training device, and performs sparse processing on the gradients in the current gradient set calculated by the matrix calculation unit to obtain sparse gradients; The communication interface of the first training device sends the sparse gradients to other training devices in the M training devices, and receives sparse gradients from the other training devices through the communication interface; Wherein, the process of updating the sparse threshold in the memory by the vector calculation unit or the threshold updating module of the first training device according to the current gradient set in the i th round of iterative training is performed in parallel with the process of sparse processing of the gradients in the current gradient set by the sparse module of the first training device using the sparse threshold in the memory in the i th round of iterative training.

2. The method of claim 1, wherein The current gradient mean is a mean of gradients in the current gradient set, the historical gradient mean is a mean of gradients in a historical gradient set, the historical gradient set includes parameter gradients used by the first training device to calculate the sparse threshold in the memory before updating, the current gradient standard deviation is a standard deviation of gradients in the current gradient set, the historical gradient standard deviation is a standard deviation of gradients in the historical gradient set, and a mapping relationship between the sparse threshold in the memory before updating and the historical gradient mean is the same as a mapping relationship between the sparse threshold in the memory after updating and the current gradient mean.

3. The method of claim 2, wherein, The new sparse threshold, the current gradient mean, the historical gradient mean, the current gradient standard deviation, the historical gradient standard deviation and the sparse threshold in the memory before updating satisfy the following relationship: wherein th new denotes the new sparsity threshold, th denotes the sparsity threshold before update in the memory, μ denotes the historical gradient mean, μ new denotes the current gradient mean, σ denotes the current gradient standard deviation, σ new denotes the historical gradient standard deviation.

4. The method of claim 1, wherein, If the proportion of target gradients in the current gradient set is greater than or equal to a first threshold, the first training device reduces the sparse threshold in the memory; If the proportion of target gradients in the current gradient set is less than or equal to a second threshold, the first training device increases the sparse threshold in the memory.

5. The method of claim 4, wherein, The updating of the sparse threshold in the memory according to the size relationship between the proportion of target gradients in the current gradient set and the threshold comprises: generating a sparse mask matrix corresponding to the current gradient set according to the current gradient set and the sparse threshold in the memory, wherein a value of each element in the sparse mask matrix is 1, indicating that a gradient corresponding to the element is greater than the sparse threshold in the memory, and a value of each element in the sparse mask matrix is 0, indicating that a gradient corresponding to the element is less than or equal to the sparse threshold in the memory; calculating the proportion according to the number of elements with a value of 0 in the sparse mask matrix, wherein the following relationship is satisfied among a sum of all elements in the sparse mask matrix, a number of all elements in the sparse mask matrix and the proportion: sparse ratio = 1 - sum(mask) / size(mask) th = th - a*th if sparse ratio - sr > ε new th = th + a*th if sparse ratio - sr < -ε new th = th - a*th if sparse ratio - sr > ε wherein, the sparse ratio represents the proportion of elements with value 0 in the sparse mask matrix, sum(mask) represents the sum of all elements in the sparse mask matrix, size(mask) represents the number of all elements in the sparse mask matrix, th new wherein, th represents the new sparse threshold, th represents the sparse threshold before updating in the memory, sr is a preset sparsity, a is a step size of updating the threshold each time, and ε is a preset hyperparameter.

6. A gradient sparsification method of a neural network, characterized by, The gradient sparsification method is applied to a distributed training system in the field of artificial intelligence technology, the distributed training system is used for N-round iterative training of the neural network, the distributed training system includes M training devices, each training device in the M training devices is deployed with the neural network, the training device includes a matrix calculation unit, a threshold updating module, a vector calculation unit, a memory, a sparse module and a communication interface, N and M are positive integers greater than 1, and the method comprises: only in the i-th round of iterative training in the N-round iterative training, the i is an integer multiple of m, m is an integer greater than 1 and less than N: the matrix calculation unit of the first training device calculates a current gradient set of the i-th round of iteration; the first training device is any one of the M training devices; The vector calculation unit of the first training device calculates a new sparse threshold according to a current gradient mean, a current gradient standard deviation of the current gradient set, and a historical gradient mean, a historical gradient standard deviation and a sparse threshold read from the memory of the first training device, and updates the sparse threshold in the memory as the new sparse threshold; wherein the current gradient is a gradient in the current gradient set, and the historical gradient is a gradient in the historical gradient set; Alternatively, the threshold updating module of the first training device updates the sparse threshold in the memory according to a size relationship between a proportion of target gradient in the current gradient set and the threshold, and the proportion of the target gradient is a proportion of a total number of gradients in the current gradient set that are less than or equal to the sparse threshold in the memory before being updated to a total number of gradients in the current gradient set. The current gradient set contains a parameter gradient calculated by the matrix calculation unit of the first training device for the neural network in the i-th round of iterative training, i is a positive integer less than or equal to N, and i is greater than 1; In the i-th round of iterative training, the sparsity module of the first training device reads the sparse threshold from the memory of the first training device, and sparsifies the gradients in the current gradient set calculated by the matrix calculation unit to obtain sparse gradients; The communication interface of the first training device sends the sparse gradients to other training devices in the M training devices, and receives sparse gradients from the other training devices through the communication interface; The process of updating the sparse threshold in the memory by the vector calculation unit or the threshold updating module of the first training device according to the current gradient set is performed in parallel with the process of sparsifying the gradients in the current gradient set by the sparsity module of the first training device using the sparse threshold in the memory.

7. The method of claim 6, wherein, The current gradient mean is a mean of the gradients in the current gradient set, the historical gradient mean is a mean of the gradients in the historical gradient set, the historical gradient set includes a parameter gradient used by the first training device to calculate the sparse threshold in the memory before being updated, the current gradient standard deviation is a standard deviation of the gradients in the current gradient set, and the historical gradient standard deviation is a standard deviation of the gradients in the historical gradient set. The mapping relationship between the sparse threshold in the memory before being updated and the historical gradient mean is the same as the mapping relationship between the sparse threshold in the memory after being updated and the current gradient mean.

8. The method of claim 7, wherein, The new sparse threshold, the current gradient mean, the historical gradient mean, the current gradient standard deviation, the historical gradient standard deviation and the sparse threshold in the memory before being updated satisfy the following relationship: where th new denotes the new sparsity threshold, th denotes the sparsity threshold before update in the memory, μ denotes the historical gradient mean, μ new denotes the current gradient mean, σ denotes the historical gradient standard deviation, σ new denotes the current gradient standard deviation.

9. The method of claim 6, wherein, If the proportion of the target gradient in the current gradient set is greater than or equal to a first threshold, the first training device reduces the sparse threshold in the memory. If a proportion of target gradients in the current gradient set is less than or equal to a second threshold value, the first training device increases a sparse threshold value in the memory.

10. The method of claim 9, wherein The updating of the sparse threshold value in the memory according to the size relationship between the proportion of target gradients in the current gradient set and the threshold value comprises: According to the current gradient set and the sparse threshold value in the memory, a sparse mask matrix corresponding to the current gradient set is generated, each element in the sparse mask matrix has a value of 1, indicating that a gradient corresponding to the element is greater than the sparse threshold value in the memory, and each element has a value of 0, indicating that the gradient corresponding to the element is less than or equal to the sparse threshold value in the memory. The proportion is calculated according to the number of elements with a value of 0 in the sparse mask matrix, and the sum of all elements in the sparse mask matrix, the number of all elements in the sparse mask matrix and the proportion satisfy the following relationship: sparse ratio = 1 - sum(mask) / size(mask) th = th - a*th if sparse ratio - sr > e new th = th + a*th if sparse ratio - sr < -e new th = th - a*th if sparse ratio - sr > e wherein, the sparse ratio represents the proportion of elements with value 0 in the sparse mask matrix, sum(mask) represents the sum of all elements in the sparse mask matrix, size(mask) represents the number of all elements in the sparse mask matrix, th new wherein, th represents the new sparse threshold, th represents the sparse threshold before updating in the memory, sr is a preset sparsity, a is a step size of updating the threshold each time, and ε is a preset hyperparameter.

11. A method for gradient sparsification of a neural network, the method comprising: The gradient sparsification method is applied to a distributed training system in the field of artificial intelligence technology, the distributed training system is used for N-round iterative training of the neural network, the distributed training system comprises M training devices, each of the M training devices is deployed with the neural network, and the training device comprises a matrix calculation unit, a threshold value updating module, a vector calculation unit, a memory, a sparse module and a communication interface; N and M are positive integers greater than 1, and the method comprises: In the i th round of iterative training in the N-round iterative training: The matrix calculation unit of the first training device calculates a current gradient set of the i th round of iterative training; the first training device is any one of the M training devices; The vector calculation unit or the threshold value updating module of the first training device updates the sparse threshold value in the memory of the first training device to a new sparse threshold value according to the current gradient set; wherein the current gradient set comprises a parameter gradient calculated by the matrix calculation unit of the first training device for the neural network in the i th round of iterative training, i is a positive integer less than or equal to N, and i is greater than 1; In the i th round of iterative training, the sparse module of the first training device reads the sparse threshold value from the memory of the first training device, and performs sparse processing on the gradients in the current gradient set calculated by the matrix calculation unit to obtain sparse gradients; The communication interface of the first training device sends the sparse gradients to other training devices in the M training devices, and receives sparse gradients from the other training devices through the communication interface; The process of updating the sparse threshold value in the memory of the first training device according to the current gradient set in the i th round of iterative training by the vector calculation unit or the threshold value updating module of the first training device is performed in parallel with the process of performing sparse processing on the gradients in the current gradient set using the sparse threshold value in the memory by the sparse module of the first training device in the i th round of iterative training.

12. The method of claim 11, wherein, The current gradient mean is a mean of gradients in a current gradient set, the historical gradient mean is a mean of gradients in a historical gradient set, the historical gradient set includes parameter gradients used by the first training device to calculate the sparse threshold in the memory before being updated, the current gradient set includes parameter gradients calculated by the first training device for the neural network in the i th iteration training, the current gradient standard deviation is a standard deviation of gradients in the current gradient set, the historical gradient standard deviation is a standard deviation of gradients in the historical gradient set, and a mapping relationship between the sparse threshold in the memory before being updated and the historical gradient mean is the same as a mapping relationship between the sparse threshold in the memory after being updated and the current gradient mean.

13. The method of claim 12, wherein, The new sparse threshold, the current gradient mean, the historical gradient mean, the current gradient standard deviation, the historical gradient standard deviation and the sparse threshold in the memory before being updated satisfy the following relationship: where th new denotes the new sparsity threshold, th denotes the sparsity threshold before update in the memory, μ denotes the historical gradient mean, μ new denotes the current gradient mean, σ denotes the historical gradient standard deviation, σ new denotes the current gradient standard deviation.

14. The method according to any one of claims 11 to 13, characterized in that, The i is an integer multiple of m, and m is an integer greater than 1 and less than N.

15. A method for gradient sparsification of a neural network, the method comprising: The gradient sparsification method is applied to a distributed training system in the field of artificial intelligence technology, the distributed training system is used for N-round iteration training of the neural network, the distributed training system includes M training devices, each training device in the M training devices is deployed with the neural network, the training device includes a matrix calculation unit, a threshold updating module, a vector calculation unit, a memory, a sparse module and a communication interface, N and M are positive integers greater than 1, and the method comprises: Only in the i th iteration training of the N-round iteration training, the i is an integer multiple of m, and m is an integer greater than 1 and less than N: The matrix calculation unit of the first training device calculates a current gradient set of the i th iteration; the first training device is any one of the M training devices; The vector calculation unit or the threshold updating module of the first training device updates the sparse threshold in the memory of the first training device to a new sparse threshold according to the current gradient set; wherein the current gradient set includes parameter gradients calculated by the matrix calculation unit of the first training device for the neural network in the i th iteration training, i is a positive integer less than or equal to N, and i is greater than 1; In the i th iteration training, the sparse module of the first training device reads the sparse threshold from the memory of the first training device, and sparsifies the gradients in the current gradient set calculated by the matrix calculation unit to obtain sparse gradients; The communication interface of the first training device sends the sparse gradients to other training devices in the M training devices, and receives sparse gradients from the other training devices through the communication interface; The vector calculation unit or the threshold updating module of the first training device updates the sparse threshold in the memory according to the current gradient set in the i-th iteration training.

16. The method of claim 15, wherein, The updating of the sparse threshold in the memory of the first training device according to the current gradient set to a new sparse threshold comprises: updating the sparse threshold in the memory according to the size relationship between the proportion of the target gradient in the current gradient set and the threshold; wherein, if the proportion is greater than or equal to a first threshold, the first training device decreases the sparse threshold in the memory, and if the proportion is less than or equal to a second threshold, the first training device increases the sparse threshold in the memory, the current gradient set comprises the parameter gradient calculated by the first training device for the neural network in the i-th iteration training, and the proportion of the target gradient is the proportion of the total number of gradients less than or equal to the sparse threshold in the memory before updating in the current gradient set to the total number of gradients in the current gradient set.

17. The method of claim 16, wherein, The updating of the sparse threshold in the memory according to the size relationship between the proportion of the target gradient in the current gradient set and the threshold comprises: generating a sparse mask matrix corresponding to the current gradient set according to the current gradient set and the sparse threshold in the memory, wherein the value of each element in the sparse mask matrix is 1, indicating that the gradient corresponding to the element is greater than the sparse threshold in the memory, and the value of each element is 0, indicating that the gradient corresponding to the element is less than or equal to the sparse threshold in the memory; calculating the proportion according to the number of elements with a value of 0 in the sparse mask matrix, wherein the sum of all elements in the sparse mask matrix, the number of all elements in the sparse mask matrix and the proportion satisfy the following relationship: sparse ratio = 1-sum(mask) / size(mask) th = th - a*th if sparse ratio - sr > e new th = th + a*th if sparse ratio - sr < -e new th = th - a*th if sparse ratio - sr > e wherein, the sparse ratio represents the proportion of elements with value 0 in the sparse mask matrix, sum(mask) represents the sum of all elements in the sparse mask matrix, size(mask) represents the number of all elements in the sparse mask matrix, th new wherein, th represents the new sparse threshold, th represents the sparse threshold before updating in the memory, sr is a preset sparsity, a is a step size of updating the threshold each time, and ε is a preset hyperparameter.

18. A gradient sparsity device for a neural network, characterized in that, The device comprises a processor coupled with a memory, and the processor is configured to execute program codes in the memory to implement the method according to any one of claims 1 to 17.

19. A gradient sparsity device for a neural network, characterized in that, The readable medium stores computer programs or instructions, and when the computer programs or instructions are executed by the processor, the method according to any one of claims 1 to 17 is implemented.

20. A computer readable medium characterized by The readable medium stores computer programs or instructions, and when the computer programs or instructions are executed by the processor, the method according to any one of claims 1 to 17 is implemented.

Images