Layer sparse neural network distributed training system and method based on hypernetwork

Abstract

The invention discloses a super-network-based layer sparse neural network distributed training system and method, the distributed training system comprises a global node and a plurality of computer nodes, and a super network is arranged on the computer nodes; the computer node is used for acquiring a to-be-trained neural network copy, and training the to-be-trained neural network and the super network based on the local training data set to obtain a complete gradient of the current training of the to-be-trained neural network; performing layer sparsification operation according to the output of the super network, and selecting a multi-layer gradient in the neural network to be trained for transmission; the global node is used for receiving and aggregating the gradients uploaded by all the computer nodes, updating the weight of the global neural network, and judging whether the global neural network is converged or not or whether the number of training times reaches the preset number of times, if yes, determining a trained neural network model based on the network model trained by the plurality of computer nodes, and if not, determining the neural network model; otherwise, the global neural network weight is distributed to all the computer nodes for training again.

Description

Technical Field

[0001] The present invention relates to a neural network training technology, and in particular to a layer-sparse neural network distributed training system and method based on a supernet. Background Art

[0002] As the complexity of deep neural networks (DNNs) continues to increase, the size of their parameters continues to grow, making training extremely time-consuming. Distributed training strategies are introduced into the neural network training process to speed up training and reduce training time. For example, in the training of convolutional networks for image classification or natural language processing neural networks, the training time of a single computer node is 1 hour. In theory, adding one node can achieve a 2-fold acceleration and reduce the training time by 50%, but in practice we can only reduce the training time by 20-30%. The reason is that the efficiency of distributed training is reduced by the overhead introduced by the frequent communication between different computer nodes, which reduces the acceleration effect, so that the training efficiency does not reach the expected effect.

[0003] In order to solve the above problems, related research work has introduced a variety of gradient compression techniques (such as sparsification, quantization, and low rank) to reduce communication overhead and reduce overall training time. However, these methods mainly focus on numerical characteristics, such as reducing data transmission rules or changing floating-point stored gradients to data structures with fewer bits, ignoring the convergence characteristics of the neural network training process itself. In addition, these techniques need to complete operations on all tensors and perform compression operations on each element, which will consume a lot of computing time. Although the above methods can reduce communication time, they increase data compression time, and the overall time is not significantly reduced, that is, the degree of improvement in the training efficiency of the neural network is not ideal. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the layer-sparse neural network distributed training system and method based on supernet provided by the present invention solve the problem of low efficiency of the prior art methods in performing distributed training of neural networks.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0006] In a first aspect, a layer-sparse neural network distributed training system based on a supernet is provided, which includes a global node and a plurality of computer nodes, each of which is provided with a supernet;

[0007] The computer node is used to obtain a copy of the neural network to be trained, and train the neural network to be trained and the supernet based on the local training data set to obtain the complete gradient of the neural network to be trained at that time; perform a layer sparsification operation according to the output of the supernet, and select the gradients of multiple layers in the neural network to be trained for transmission;

[0008] The global node is used to receive and aggregate the gradients uploaded by all computer nodes, update the global neural network weights, and determine whether the global neural network has converged or whether the number of training times has reached a preset number of times. If so, the trained neural network model is determined based on the network models trained by multiple computer nodes, otherwise the global neural network weights are distributed to all computer nodes for retraining;

[0009] The neural network to be trained is an image classification ResNet or a natural language processing neural network.

[0010] Further, methods for performing layer sparsification operations based on the output of the supernet include:

[0011] Calculate the difference between the outputs of two consecutive supernet trainings and use the exponential moving average to smooth the difference to obtain the probability of each layer of the neural network to be trained generated by the supernet being selected;

[0012] According to the probability of each layer of the neural network to be trained generated by the supernet being selected, the gradients of the k layers with the largest probability are selected for transmission.

[0013] Furthermore, the output of the supernet is expressed as:

[0014] α i =HN i (v i , ψ i ) Among them, HN i () is the supernet of the ith computer node, υ i and ψ i are the embedding vector of the i-th computer node and the network weight of the supernet respectively; α i is the output of the supernet of the ith computer node, that is, the probability of each layer of the supernet of the ith computer node being selected for training the neural network;

[0015] The expression for smoothing the difference using exponential moving average is:

[0016]

[0017] Among them, ∈ is a hyperparameter; are the probabilities of each layer of the neural network to be trained generated by the supernet of the i-th computer node being selected at the t-1th and tth iterations; is the difference between the output of the supernet of the i-th computer node at the t-th iteration and the output at the t-1-th iteration.

[0018] Furthermore, the objective function of the supernet is expressed as:

[0019]

[0020] Among them, f i (θ i ; α i ) is the objective function value of the supernet of the i-th computer node; θ i is the weight of the neural network to be trained on the i-th computer node; and They are θ i The weights of the l1 and ln layers in L i (·) is the loss function; and Respectively α i where is the probability of selecting the l1th and lnth layers of the neural network to be trained; ⊙ is the Hadamard product.

[0021] Furthermore, the expressions for updating the gradients of the embedding vector and the supernet weights at each iteration are:

[0022]

[0023] in, For i The gradient of is i The gradient of θ i is the weight of the neural network to be trained on the i-th computer node; is the intermediate parameter; and They are θ i The weights of the l1 and ln layers in L i (·) is the loss function; and Respectively α i where is the probability of selecting the l1th and lnth layers of the neural network to be trained; ⊙ is the Hadamard product.

[0024] Furthermore, the supernet of each computer node includes an input layer, two fully connected layers and an output layer, and each fully connected layer includes a plurality of neurons.

[0025] In a second aspect, a method for distributed training of a layer-sparse neural network based on a supernet is provided, comprising the steps of:

[0026] S1. Arrange a supernet on each computer node, and each computer node obtains a copy of the neural network to be trained;

[0027] S2. Train the neural network to be trained and the supernet based on the local training data set to obtain the complete gradient of the neural network to be trained during the training;

[0028] S3, perform layer sparsification operation according to the output of the supernet, and select the gradients of multiple layers in the neural network to be trained for transmission;

[0029] S4, receiving and aggregating the gradients uploaded by all computer nodes, and updating the global neural network weights;

[0030] S5, determine whether the global neural network converges or whether the number of training times reaches a preset number, if so, proceed to step S6, otherwise distribute the global neural network weights to all computer nodes and return to step S2;

[0031] S6. Determine a trained neural network model based on the network models trained by multiple computer nodes;

[0032] The neural network to be trained is an image classification ResNet or a natural language processing neural network.

[0033] Further, the method of performing a layer sparsification operation according to the output of the supernet includes:

[0034] Calculate the difference between the outputs of two consecutive supernet trainings and use the exponential moving average to smooth the difference to obtain the probability of each layer of the neural network to be trained generated by the supernet being selected;

[0035] According to the probability of each layer of the neural network to be trained generated by the supernet being selected, the gradients of the k layers with the largest probability are selected for transmission.

[0036] Furthermore, the output of the supernet is expressed as:

[0037] α i =HN i (v i , ψ i )

[0038] Among them, α i is the output of the ith computer node; HN i () is the supernet of the i-th computer node, v i and ψ i are the embedding vector of the i-th computer node and the network weight of the supernet respectively; α i is the output of the supernet of the ith computer node, that is, the probability of each layer of the supernet of the ith computer node being selected for training the neural network;

[0039] The expression for smoothing the difference using exponential moving average is:

[0040]

[0041] Among them, ∈ is a hyperparameter; are the probabilities of each layer of the neural network to be trained generated by the supernet of the i-th computer node being selected at the t-1th and tth iterations; is the difference between the output of the supernet of the i-th computer node at the t-th iteration and the output at the t-1-th iteration.

[0042] The beneficial effects of the present invention are as follows: the present solution designs an independent supernet on each computer node, and is trained to generate the weight of each layer of the neural network to be trained to represent the importance of the layer, that is, the supernet is used to identify the most "important" layer in each iteration, and the gradients of these "important" neural network layers are transmitted, while other layers are discarded and gradients are accumulated and transmitted in subsequent operations. In this way, the present solution improves the transmission efficiency in the distributed training process, avoids the additional computing cost caused by frequent operations on tensor elements by the current mainstream gradient compression method, and thus improves the training efficiency of the neural network.

[0043] This scheme has been comprehensively experimented on Resnet-18 and VGG-16, and the results show that this scheme can reduce communication overhead with little loss of accuracy. In addition, the method provided by this scheme can be overlapped with other compression methods to further reduce the amount of communication, which is especially suitable for the distributed training of the current mainstream graph classification and natural language processing neural networks, which can significantly reduce communication overhead and reduce training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Schematic diagram of distributed training for a layer-sparse neural network distributed training system based on a hypernet.

[0045] Figure 2 Flowchart of the distributed training method of layer-sparse neural networks based on hypernet. DETAILED DESCRIPTION

[0046] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0047] This scheme provides a layer-sparse neural network distributed training system based on a supernet, which includes a global node and multiple computer nodes. A supernet is set on each computer node. The supernet includes an input layer, two fully connected layers and an output layer. Each fully connected layer includes multiple neurons.

[0048] The computer node is used to obtain a copy of the neural network to be trained, and train the neural network to be trained and the supernet based on the local training data set to obtain the complete gradient of the neural network to be trained during the training; the layer sparsification operation is performed according to the output of the supernet, and the gradients of multiple layers in the neural network to be trained are selected for transmission.

[0049] The global node is used to receive and aggregate the gradients uploaded by all computer nodes, update the global neural network weights, and determine whether the global neural network has converged or whether the number of training times has reached a preset number. If so, the trained neural network model is determined based on the network models trained by multiple computer nodes, otherwise the global neural network weights are distributed to all computer nodes for re-training.

[0050] The neural network to be trained is an image classification ResNet or a natural language processing neural network.

[0051] Hypernetworks on computer nodes is a special neural network architecture, the core idea of ​​which is to train the weights of another larger network in a smaller network. Figure 1 As shown in the figure, the supernet maps the model parameters associated with the target task to the corresponding model parameters by mapping the embedding vector. In this scheme, the supernet is used on each computer node and is trained to generate the weights of each layer of the network to represent the importance of the layer. Then, after calculating the gradient of each layer, the target network is pruned according to the weights generated by the supernet. Unlike manual pruning, this scheme allows different node network pruning structures to be different. This means that our method is more in line with the dynamic characteristics of the training process.

[0052] In one embodiment of the present invention, a method for performing a layer sparsification operation according to an output of a supernet includes:

[0053] Calculate the difference between the outputs of two consecutive supernet trainings. The output expression of the supernet is:

[0054] α i =HN i (v i , ψ i )

[0055] Among them, NH i () is the supernet of the ith computer node, υ i and ψ i are the embedding vector of the i-th computer node and the network weight of the supernet respectively; α i is the output of the supernet of the ith computer node, that is, the probability of each layer of the supernet of the ith computer node being selected for training the neural network;

[0056] The difference is smoothed by using the exponential moving average to obtain the probability of each layer of the neural network to be trained generated by the supernet being selected; the expression for smoothing the difference by using the exponential moving average is:

[0057]

[0058] Among them, ∈ is a hyperparameter; are the probabilities of each layer of the neural network to be trained generated by the supernet of the i-th computer node being selected at the t-1th and tth iterations; is the difference between the output of the supernet of the i-th computer node at the t-th iteration and the output at the t-1-th iteration.

[0059] According to the probability of each layer of the neural network to be trained generated by the supernet being selected, the gradients of the k layers with the largest probability are selected for transmission.

[0060] In actual use, α i Generated directly by the supernet, this scheme uses α i The change between the two times Δα i Indicates the importance of the layer. However, in some extreme cases, due to the problem of training samples, the value changes greatly and is not suitable for directly indicating the importance of the layer. This solution can solve the problem of training samples by smoothing the difference using an exponential moving average. i A big problem of change.

[0061] In order to make the weights generated for the layer approximately reflect its importance in each training iteration, in implementation, the present solution preferably adopts the expression of the objective function of the supernet as:

[0062]

[0063] Among them, f i (θ i ; α i ) is the objective function value of the supernet of the i-th computer node; θ i is the weight of the neural network to be trained on the i-th computer node; and They are θ i The weights of the l1 and ln layers in L i (·) is the loss function; and Respectively α i where is the probability of selecting the l1th and lnth layers of the neural network to be trained; ⊙ is the Hadamard product.

[0064] The purpose of the loss function is to convert α iThe range of is limited between 0 and 1. The objective function essentially represents the loss of the original network, with each layer multiplied by the contraction factor. When all α = 1, the function value will reach a minimum. Obviously, the network without any modification will produce the minimum loss. In order to minimize the function value, the expression for updating the gradient of the embedding vector and the supernet network weight at each iteration is:

[0065]

[0066] in, For i The gradient of is i The gradient of θ i is the weight of the neural network to be trained on the i-th computer node; is the intermediate parameter; and They are θ i The weights of the l1 and ln layers in L i (·) is the loss function; and Respectively α i where is the probability of selecting the l1th and lnth layers of the neural network to be trained; ⊙ is the Hadamard product.

[0067] like Figure 2 As shown, the present solution also provides a distributed training method for a layer-sparse neural network based on a supernet, the method S comprising steps S1 to S6:

[0068] In step S1, a supernet is arranged on each computer node, and each computer node obtains a copy of the neural network to be trained;

[0069] In step S2, the neural network to be trained and the supernet are trained based on the local training data set to obtain the complete gradient of the neural network to be trained during the training;

[0070] In step S3, a layer sparsification operation is performed according to the output of the supernet, and gradients of multiple layers in the neural network to be trained are selected for transmission; the method of the sparsification operation includes:

[0071] Calculate the difference between the outputs of two consecutive supernet trainings and use the exponential moving average to smooth the difference to obtain the probability of each layer of the neural network to be trained generated by the supernet being selected;

[0072] According to the probability of each layer of the neural network to be trained generated by the supernet being selected, the gradients of the k layers with the largest probability are selected for transmission.

[0073] In step S4, the gradients uploaded by all computer nodes are received and aggregated to update the global neural network weights;

[0074] In step S5, it is determined whether the global neural network has converged or whether the number of training times has reached a preset number. If so, it proceeds to step S6. Otherwise, the weights of the global neural network are distributed to all computer nodes, and the process returns to step S2.

[0075] In step S6, a trained neural network model is determined based on the network models trained by multiple computer nodes.

[0076] In order to verify the effectiveness of the distributed training method of this scheme, the effect of this scheme is explained below in combination with the training of two neural networks:

[0077] This example selects two image classification neural networks (ResNet-18 and VGG-16) for training on two different image classification data sets, and compares the time consumed for data compression and the accuracy of the neural network; for the convenience of the experiment, the distributed training process is simulated on an NVIDIA Tesla V100 card. A total of 2 groups of experiments were set up, and the number of nodes involved in the training (the number of workers) was set to 4 and 8 nodes respectively. For the CIFAR10 dataset for image classification, the number of training periods is set to 100 when performing classification training, and for the CIFAR100 dataset, the number of training periods is set to 150.

[0078] All hyperparameters remain consistent throughout the experiment and are compared with four classic methods: Baseline is the original method without compression, the method with the largest K elements (Top-k), the quantization method (QSGD), the bidirectional compression (TernGrad), and this scheme is based on the layer-sparse neural network distributed training method (LSH) based on the hypernet. The transmission compression rate is 1 / 3, that is, the original neural network gradient has a total of 30 layers, and 10 layers are selected for transmission.

[0079] The accuracy comparison of ResNet-18 based on five different gradient compression methods on two datasets (CIFAR10 and CIFAR100) is shown in Table 1. The accuracy comparison of VGG-16 based on five different gradient compression methods on two datasets (CIFAR10 and CIFAR100) is shown in Table 2.

[0080] Table 1

[0081]

[0082]

[0083] ResNet-18 is trained on two datasets (CIFAR10 and CIFAR100). Compared with mainstream methods (data complete transmission, the largest K elements method (Top-k), quantization method (QSGD), bidirectional compression (TernGrad)), the accuracy of this method before and after LSH is shown in Table 3.

[0084] Table 3

[0085]

[0086]

[0087] VGG-16 is trained on two datasets (CIFAR10 and CIFAR100), and the accuracy of the other four methods before and after using the hypernet-based layer sparsification method (LSH) is shown in Table 4.

[0088] Table 4

[0089]

[0090]

[0091] As shown in Tables 1 and 2, ResNet-18 and VGG-16 networks were tested on two data sets. Under different cluster sizes, the accuracy of this solution method LSH is basically consistent with that of the baseline (uncompressed). In addition, compared with other classic methods in the industry (data transmission, the method with the largest K elements (Top-k), the quantization method (QSGD), and the bidirectional compression (TernGrad)), this solution method LSH has a higher accuracy rate, as shown in Tables 3 and 4. Compared with the compression rate, this solution method LSH also has obvious advantages.

[0092] In summary, in the distributed training environment of neural networks, this scheme uses supernet to help screen the optimal layer for gradient transmission in each iteration, which can reduce communication overhead and improve training efficiency. It is especially suitable for distributed training of image classification network ResNet and natural language neural network models.

Claims

1. A layer-sparse neural network distributed training system based on a supernet, characterized in that: It includes a global node and multiple computer nodes, and each computer node is provided with a supernet; The computer node is used to obtain a copy of the neural network to be trained, and train the neural network to be trained and the supernet based on the local training data set to obtain the complete gradient of the neural network to be trained at that time; perform a layer sparsification operation according to the output of the supernet, and select the gradients of multiple layers in the neural network to be trained for transmission; The global node is used to receive and aggregate the gradients uploaded by all computer nodes, update the global neural network weights, and determine whether the global neural network has converged or whether the number of training times has reached a preset number. If so, the trained neural network model is determined based on the network models trained by multiple computer nodes, otherwise the global neural network weights are distributed to all computer nodes for retraining; The neural network to be trained is an image classification ResNet or a natural language processing neural network.

2. The layer-sparse neural network distributed training system based on supernet according to claim 1, characterized in that: Methods for performing layer sparsification operations based on the output of the supernet include: Calculate the difference between the outputs of two consecutive supernet trainings and use the exponential moving average to smooth the difference to obtain the probability of each layer of the neural network to be trained generated by the supernet being selected; According to the probability of each layer of the neural network to be trained generated by the supernet being selected, the gradients of the k layers with the largest probability are selected for transmission.

3. The layer-sparse neural network distributed training system based on supernet according to claim 2, characterized in that: The output expression of the supernet is: a i =HN i (u i ,ψ i ) Among them, HN i () is the supernet of the ith computer node; i and ψ o are the embedding vector of the i-th computer node and the network weight of the supernet respectively; α i is the output of the supernet of the ith computer node, that is, the probability of each layer of the supernet of the ith computer node being selected for training the neural network; The expression for smoothing the difference using exponential moving average is: Among them, ∈ is a hyperparameter; are the probabilities of each layer of the neural network to be trained generated by the supernet of the i-th computer node being selected at the t-1th and tth iterations; is the difference between the output of the supernet of the i-th computer node at the t-th iteration and the output at the t-1-th iteration.

4. The layer-sparse neural network distributed training system based on supernet according to claim 3, characterized in that: The objective function of the supernet is expressed as: Among them, f i (θ i ; α i ) is the objective function value of the supernet of the i-th computer node; θ i is the weight of the neural network to be trained on the i-th computer node; and They are θ i The weights of the l1 and ln layers in L i (·) is the loss function; and Respectively α i where is the probability of selecting the l1th and lnth layers of the neural network to be trained; ⊙ is the Hadamard product.

5. The layer-sparse neural network distributed training system based on supernet according to claim 3, characterized in that: The expressions for updating the gradients of the embedding vector and the supernet weights at each iteration are: in, v i The gradient of is i The gradient of θ i is the weight of the neural network to be trained on the i-th computer node; is the intermediate parameter; and They are θ i The weights of the l1 and ln layers in L i (·) is the loss function; and Respectively α i where is the probability of selecting the l1th and lnth layers of the neural network to be trained; ⊙ is the Hadamard product.

6. The layer-sparse neural network distributed training system based on supernet according to claim 3, characterized in that: The supernet of each computer node includes an input layer, two fully connected layers, and an output layer, and each fully connected layer includes multiple neurons.

7. A distributed training method for layer-sparse neural networks based on supernets, characterized in that: Includes steps: S1. Arrange a supernet on each computer node, and each computer node obtains a copy of the neural network to be trained; S2. Train the neural network to be trained and the supernet based on the local training data set to obtain the complete gradient of the neural network to be trained during the training; S3, perform layer sparsification operation according to the output of the supernet, and select the gradients of multiple layers in the neural network to be trained for transmission; S4, receiving and aggregating the gradients uploaded by all computer nodes, and updating the global neural network weights; S5, determine whether the global neural network converges or whether the number of training times reaches a preset number, if so, proceed to step S6, otherwise distribute the global neural network weights to all computer nodes and return to step S2; S6. Determine a trained neural network model based on the network models trained by multiple computer nodes; The neural network to be trained is an image classification ResNet or a natural language processing neural network.

8. The distributed training method of a layer-sparse neural network based on a supernet according to claim 7, characterized in that: Methods for performing layer sparsification operations based on the output of the supernet include: Calculate the difference between the outputs of two consecutive supernet trainings and use the exponential moving average to smooth the difference to obtain the probability of each layer of the neural network to be trained generated by the supernet being selected; According to the probability of each layer of the neural network to be trained generated by the supernet being selected, the gradients of the k layers with the largest probability are selected for transmission.

9. The method for distributed training of a layer-sparse neural network based on a supernet according to claim 8, characterized in that: The output expression of the supernet is: a i =HN i (u i ,ψ i ) Among them, α i is the output of the i-th computer node; HN i () is the supernet of the ith computer node, v i and ψ i are the embedding vector of the i-th computer node and the network weight of the supernet respectively; α i is the output of the supernet of the ith computer node, that is, the probability of each layer of the supernet of the ith computer node being selected for training the neural network; The expression for smoothing the difference using exponential moving average is: Among them, ∈ is a hyperparameter; are the probabilities of each layer of the neural network to be trained generated by the supernet of the i-th computer node being selected at the t-1th and tth iterations; is the difference between the output of the supernet of the i-th computer node at the t-th iteration and the output at the t-1-th iteration.

Images