Distributed deep learning acceleration method, device, equipment and readable storage medium

Abstract

The present application relates to a distributed deep learning acceleration method, apparatus, device and readable storage medium, and relates to the field of distributed deep learning technology, including performing branch packaging processing on multiple parallel network branches in a target nonlinear network to obtain two branch sets and store them in a parameter server, each branch set including the parameters contained in the corresponding network branch; grouping multiple computing nodes in a computing cluster to obtain multiple computing node groups; for each computing node group, each computing node pulls all parameters in the corresponding branch set from the parameter server for gradient calculation, and at the same time pulls all parameters in other branch sets from other computing nodes belonging to the same computing node group for gradient calculation; the gradient calculation results are transmitted back to the parameter server, and the parameter server updates the training target nonlinear network based on the gradient calculation results, thereby effectively alleviating network congestion and accelerating the training speed of the nonlinear network in the cluster.

Description

Technical Field

[0001] The present application relates to the field of distributed deep learning technology, and in particular to a distributed deep learning acceleration method, apparatus, device, and readable storage medium. Background Art

[0002] Deep learning is finding increasingly widespread applications in fields such as computer vision and natural language processing, and more companies are attempting to integrate deep learning techniques (such as neural network models) into their products to enhance their competitiveness. Large models and massive amounts of data have become a trend in deep learning models. For example, ChatGPT (a chatbot program) developed by OpenAI is based on the 175 billion-parameter GPT-3 model and trained on 45TB of training data. Therefore, how to train large models on such massive datasets has become one of the main challenges in training.

[0003] In related technologies, parameter servers generally accelerate training through distributed training. However, due to the high communication overhead, the parameter server architecture has high requirements for network bandwidth and latency. However, during distributed training, after each node completes the calculation, the parameter server needs to perform a reduce operation (a node communication method) on the gradients calculated by all nodes and update the model parameters based on all collected gradients. Since outdated gradients have a significant impact on model updates, it is necessary to synchronize the gradients on all nodes at the same time. This process is often accompanied by network congestion. Especially when network conditions are poor, network synchronization congestion can become a major performance bottleneck for the cluster, thereby affecting the training speed and efficiency of the parameter server. Summary of the Invention

[0004] This application provides a distributed deep learning acceleration method, apparatus, device, and readable storage medium to solve problems existing in related technologies.

[0005] In a first aspect, a distributed deep learning acceleration method is provided, comprising the following steps:

[0006] Perform branch packaging processing on multiple parallel network branches in the target nonlinear network to obtain two branch sets and store them in the parameter server, where each branch set includes the parameters contained in the corresponding network branch;

[0007] Grouping multiple computing nodes in a computing cluster to obtain multiple computing node groups;

[0008] For each computing node group, each computing node pulls all parameters in the corresponding branch set from the parameter server for gradient calculation, and at the same time pulls all parameters in other branch sets from other computing nodes in the same computing node group for gradient calculation;

[0009] The gradient calculation result is transmitted back to the parameter server, so that the parameter server can update the training target nonlinear network based on the gradient calculation result.

[0010] In some embodiments, grouping multiple computing nodes in a computing cluster to obtain multiple computing node groups includes:

[0011] Group multiple computing nodes in the computing cluster into pairs to obtain multiple computing node groups;

[0012] When the number of computing nodes is an odd number, one computing node group has three computing nodes.

[0013] In some embodiments, the two branch sets include a first branch set and a second branch set. When the computing node group consists of the first computing node and the second computing node, each computing node pulls all parameters in the corresponding branch set from the parameter server to perform gradient calculation, and simultaneously pulls parameters corresponding to other branch sets from other computing nodes in the same computing node group to perform gradient calculation, including:

[0014] The first computing node pulls the first parameters corresponding to all intermediate network layers in the first branch set from the parameter server, and the second computing node pulls the second parameters corresponding to all intermediate network layers in the second branch set from the parameter server;

[0015] The first computing node performs gradient calculation based on the pulled first parameter, and simultaneously pulls the second parameter from the second computing node and performs gradient calculation based on the pulled second parameter;

[0016] The second computing node performs gradient calculation based on the pulled second parameter, and simultaneously pulls the first parameter from the first computing node and performs gradient calculation based on the pulled first parameter.

[0017] In some embodiments, the two branch sets include a first branch set and a second branch set. When the computing node group consists of a first computing node, a second computing node, and a third computing node, each computing node pulls all parameters in the corresponding branch set from the parameter server to perform gradient calculation, and simultaneously pulls parameters corresponding to other branch sets from other computing nodes in the same computing node group to perform gradient calculation, including:

[0018] The first computing node pulls the first parameters corresponding to all intermediate network layers in the first branch set from the parameter server, and the second computing node and the third computing node respectively pull the second parameters corresponding to all intermediate network layers in the second branch set from the parameter server;

[0019] The first computing node performs gradient calculation based on the pulled first parameter, and simultaneously pulls the second parameter from the second computing node or the third computing node and performs gradient calculation based on the pulled second parameter;

[0020] The second computing node and the third computing node respectively perform gradient calculation based on the pulled second parameter, and at the same time respectively pull the first parameter from the first computing node and perform gradient calculation based on the pulled first parameter.

[0021] In some embodiments, performing branch packing processing on multiple parallel network branches in the target nonlinear network to obtain two branch sets includes:

[0022] Multiple parallel network branches are packaged according to the parameter sizes of the network branches to obtain two branch sets.

[0023] In some embodiments, before the step of performing branch packaging processing on the multiple parallel network branches in the target nonlinear network, the method further includes:

[0024] The target nonlinear network is divided into multiple parallel network branches.

[0025] In a second aspect, a distributed deep learning acceleration device is provided, comprising:

[0026] A branch packaging unit is used to: perform branch packaging processing on multiple parallel network branches in the target nonlinear network to obtain two branch sets and store them in a parameter server, where each branch set includes parameters contained in the corresponding network branch;

[0027] A node grouping unit, which is used to group multiple computing nodes in the computing cluster to obtain multiple computing node groups;

[0028] The transfer calculation unit is used to pull all parameters in the corresponding branch set from the parameter server for each computing node group to perform gradient calculation, and at the same time pull all parameters in other branch sets from other computing nodes in the same computing node group to perform gradient calculation;

[0029] The updating training unit is used to transmit the gradient calculation result back to the parameter server so that the parameter server can update the training target nonlinear network based on the gradient calculation result.

[0030] In some embodiments, the two branch sets include a first branch set and a second branch set. When the computing node group consists of the first computing node and the second computing node, the transfer computing unit is specifically configured to:

[0031] The first computing node pulls first parameters corresponding to all intermediate network layers in the first branch set from the parameter server, and the second computing node pulls second parameters corresponding to all intermediate network layers in the second branch set from the parameter server;

[0032] The first computing node performs gradient calculation based on the first parameter pulled, and simultaneously pulls the second parameter from the second computing node and performs gradient calculation based on the second parameter pulled;

[0033] The second computing node is caused to perform gradient calculation based on the pulled second parameter, and the first parameter is pulled from the first computing node and gradient calculation is performed based on the pulled first parameter.

[0034] In a third aspect, a distributed deep learning acceleration device is provided, comprising: a memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the aforementioned distributed deep learning acceleration method.

[0035] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the aforementioned distributed deep learning acceleration method is implemented.

[0036] The present application provides a distributed deep learning acceleration method, apparatus, device, and readable storage medium, including performing branch packaging processing on multiple parallel network branches in a target nonlinear network to obtain two branch sets and store them in a parameter server, each branch set including parameters contained in its corresponding network branch; grouping multiple computing nodes in a computing cluster to obtain multiple computing node groups; for each computing node group, each computing node pulls all parameters in the corresponding branch set from the parameter server to perform gradient calculation, and simultaneously pulls all parameters in other branch sets from other computing nodes in the same computing node group to perform gradient calculation; and transmits the gradient calculation results back to the parameter server, so that the parameter server can update the training target nonlinear network based on the gradient calculation results. The present application divides the nonlinear network into different computationally independent parts, deploys the different parts in parallel on different computing nodes, and utilizes the idle bandwidth between different computing nodes to transfer parameters to alleviate network congestion. In addition, while transferring parameters, gradient calculation is also performed on the pulled parameters to achieve overlap between parameter transfer and parameter calculation, thereby accelerating the training speed of the nonlinear network in the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0038] Figure 1 A schematic diagram of a distributed deep learning acceleration method provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of a nonlinear network structure provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the specific structure of a nonlinear network provided in an embodiment of the present application;

[0041] Figure 4 Schematic diagram of the communication and calculation method of the traditional parameter server architecture in the prior art;

[0042] Figure 5 Schematic diagram of a communication method in which forward computation and parameter transfer overlap with each other, provided in an embodiment of the present application;

[0043] Figure 6 Schematic diagram of experimental results on the DFN model provided in the embodiments of the present application;

[0044] Figure 7 Schematic diagram of experimental results on the InceptionV4 model provided in the embodiments of this application;

[0045] Figure 8 A schematic diagram of the structure of a distributed deep learning acceleration device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0047] Figure 1 A distributed deep learning acceleration method provided in an embodiment of the present application includes the following steps:

[0048] Step S10: performing branch packaging processing on multiple parallel network branches in the target nonlinear network to obtain two branch sets and store them in a parameter server, where each branch set includes parameters contained in the corresponding network branch;

[0049] For example, it can be understood that distributed training can deploy data or models on different computing nodes (such as GPU (Graphics Processing Unit, graphics processing unit) and TPU (Tensor Processing Unit, tensor processing unit), etc.), and accelerate training through parallel computing. Among them. Distributed training is mainly divided into data parallelism and model parallelism. Data parallelism divides the training data set into different storage nodes, thereby utilizing multiple training nodes for parallel training; while model parallelism divides the model into different parts, and trains different positions of the model on different training units, thereby achieving parallelism. However, since model parallelism requires fine-grained division and scheduling of the model, it is often limited in use. Data parallelism can make full use of the computing resources of each computing node to calculate the gradients of different small batches (i.e., mini-batches) separately, and then update the global model by averaging these gradients.

[0050] The parameter server architecture is a framework for distributed machine learning computing that can effectively manage large-scale model parameters. The parameter server architecture typically includes the following components: 1) Parameter servers: One or more parameter servers are responsible for storing model parameters and receiving and processing gradient information uploaded by compute nodes; 2) Compute nodes: Compute nodes during training are responsible for calculating the gradients of data and uploading them to the parameter servers; 3) Communication components: These components are used for communication between compute nodes and parameter servers, typically using efficient communication protocols to reduce network overhead.

[0051] As can be seen, in this parameter server architecture, one or more parameter servers store all the parameters of the machine learning model, while the training process is performed on multiple compute nodes. Each node is responsible for calculating the gradients of a portion of the data and uploading the gradients to the parameter server. The parameter server then updates the model parameters based on the received gradients. The advantage of the parameter server is that it can effectively handle the training of large-scale models because different compute nodes can independently calculate the gradients of the data, reducing the computation time during training. Furthermore, since the parameter server is responsible for storing the model parameters, the compute nodes only need to process the data and gradients, thus reducing the burden on the compute nodes.

[0052] However, since the parameter server not only needs to process a large amount of gradient information, which can easily become a system bottleneck, but also has high requirements for network bandwidth and latency due to the large communication overhead, it in turn affects the training speed and efficiency of the model.

[0053] In this embodiment, a nonlinear network is divided into several network branches that are independent of each other in terms of computation and can be run in parallel. That is, there are multiple modules with independent computation branches, so that the model structure can be deployed in parallel between these branches. At the same time, multiple parallel network branches are packaged to obtain two branch sets (for example, the first branch set is and the second branch set ), and each branch set contains the model parameters corresponding to the intermediate network layer contained in its corresponding network branch, so as to prepare data for parameter transfer between subsequent computing nodes; then the two branch sets are stored in the parameter server, so that the computing nodes can pull the model parameters from the parameter server for gradient calculation.

[0054] Furthermore, before the step of performing branch packaging processing on the multiple parallel network branches in the target nonlinear network, the method further includes:

[0055] The target nonlinear network is divided into multiple parallel network branches.

[0056] For example, in this embodiment, the target nonlinear network is divided, for example, see Figure 2 As shown in the figure, the target nonlinear network is divided into three different branches, namely A-[BC]-G, A-[D]-G and A-[E,F]-G. The intermediate network layers in “[]” are independent of each other during calculation; that is, the output of network layer A will be used as input by B, D and E at the same time.

[0057] In this embodiment, the represents all network layers, and l1 represents the input layer, l T represents the output layer; (l i ,l j )∈O represents the set of connections between all layers, (l i ,l j ) refers to the connection from layer i to layer j, that is, the output of layer i will be used as the input of layer j. At the same time, you can also use Represents the set of all network branches; For network branch Z j The set of network layers included, for example, Z j is A-[BC]-G, then is B; since each network layer must belong to a certain network branch, then Both of the following requirements will be met:

[0058]

[0059]

[0060] It should be noted that, for a nonlinear network, the number of network branches can be determined according to actual needs and is not limited here.

[0061] Furthermore, the branch packaging process is performed on the multiple parallel network branches in the target nonlinear network to obtain two branch sets, including:

[0062] Multiple parallel network branches are packaged according to the parameter sizes of the network branches to obtain two branch sets.

[0063] For example, in this embodiment, the parameter amount and computation amount of different network branches are calculated based on the preset branch partitioning and packaging algorithm (i.e., Packing algorithm), and the total branch set is Divided into two branch sets: and and The number of parameters and the amount of calculation are approximately the same, where the amount of calculation and parameters of each network branch is equivalent to the sum of the number of parameters and the amount of calculation of all network layers of the network branch.

[0064] Among them, the branch partitioning and packaging algorithm is as follows:

[0065]

[0066]

[0067] Specifically, the principle of the Packing algorithm is: the input of the algorithm is a nonlinear network The total set of branches and its corresponding parameter set The parameter volume refers to the storage space occupied by the model parameters; the output of the algorithm is divided into two branch sets and The algorithm process is as follows: First, according to the size of the parameter, the parameter set Sort in descending order; then, and Initialize to two empty sets; then, define a loop for Each network branch Z in j ,Will Put in and Among the smallest ones, and recalculate and Repeat the above steps until all network branches are placed and Among.

[0068] For example, see Figure 3 As shown in Figure 1, the target nonlinear network includes four network branches, namely branch 1, branch 2, branch 3 and branch 4; wherein, the connection operation layer I9 and the connection operation layer I0 represent the first and last network layers respectively, the intermediate network layers included in branch 1 are convolution layer I2 and average pooling layer I1, the intermediate network layers included in branch 2 are convolution layer I3, the intermediate network layers included in branch 3 are convolution layer I5 and convolution layer I4, and the intermediate network layers included in branch 4 are convolution layer I8, convolution layer I7 and convolution layer I6; after calculation by the Packing algorithm, branches 1, 2 and 3 are put into And branch 4 is put into Therefore, the total set of branches The total is divided into and It is understandable that when the and After that, you can deploy the parameter server architecture to the computing cluster.

[0069] Step S20: Grouping multiple computing nodes in the computing cluster to obtain multiple computing node groups;

[0070] Specifically, the plurality of computing nodes in the computing cluster are grouped to obtain a plurality of computing node groups, including:

[0071] Group multiple computing nodes in the computing cluster into pairs to obtain multiple computing node groups;

[0072] When the number of computing nodes is an odd number, one computing node group has three computing nodes.

[0073] For example, in this embodiment, multiple compute nodes in a compute cluster are grouped in pairs, enabling parameter transfer between compute nodes through these groups. For example, a compute cluster includes six compute nodes, namely compute nodes 1 through 6. These six compute nodes are divided into three compute node groups. For example, compute node group 1 includes compute nodes 1 and 2, compute node group 2 includes compute nodes 3 and 4, and compute node group 3 includes compute nodes 5 and 6.

[0074] It is understood that if the number of compute nodes is an odd number, there is a compute node group containing three compute nodes. For example, if the computing cluster includes seven compute nodes, namely compute nodes 1 to 7, and the seven compute nodes are divided into three compute node groups, then compute node group 1 includes compute nodes 1 and 2, compute node group 2 includes compute nodes 3 and 4, and compute node group 3 includes compute nodes 5, 6, and 7.

[0075] Step S30: For each computing node group, each computing node pulls all parameters in the corresponding branch set from the parameter server to perform gradient calculation, and at the same time pulls all parameters in other branch sets from other computing nodes in the same computing node group to perform gradient calculation;

[0076] For example, see Figure 4 As shown in the figure, the communication calculation method of the traditional parameter server architecture is to perform forward calculation and parameter transfer in sequence, that is, first pull all the parameters corresponding to layers 1 to 8, and then perform forward propagation after all the parameters are pulled. The time consumed in the whole process is the forward calculation time + parameter transfer time, which is not only slow and inefficient, but also has communication congestion problems in the parameter synchronization process.

[0077] In this embodiment, the model training is divided into different stages so that different training stages of different mini-batches can overlap, thereby reducing the overall training time. Figure 5 As shown, this embodiment reduces computing time by providing an algorithm that overlaps communication and computing to alleviate communication congestion, that is, the forward propagation of the model and the transfer of parameters are performed in parallel, thereby covering the time for training or parameter transfer.

[0078] Specifically, a computing node pulls the After all the parameters in (i.e. the parameters of network layers I2 to I5) are calculated, forward calculations of I1 to I5 will be started directly; at the same time, the forward calculations of I1 to I5 will be pulled from other computing nodes in the same group. All the corresponding parameters (i.e., the parameters of I6 to I8) are transferred, and after the calculation of I1 to I5 is completed, the forward propagation calculation of I6 to I8 is immediately performed. As can be seen, in this embodiment, when the parameters of I6 to I8 are transferred, the forward propagation calculation of I1 to I5 can be performed simultaneously, achieving the overlap of parameter transfer and forward calculation, that is, the forward calculation and parameter transfer cover each other, making the time consumed in the entire process much less than the time consumed by the traditional method of performing forward calculation and parameter transfer in sequence.

[0079] In this embodiment, the deployment of parameter transfer is implemented through a deployment algorithm (i.e., the Deploying algorithm), that is, the model is deployed based on the previously calculated branch set and the number of GPUs, and through small batch data testing. The Deploying algorithm is as follows:

[0080]

[0081] Specifically, the principle of the Deploying algorithm is: the input of the algorithm is a nonlinear network The branch set obtained after Packing and The number of GPUs is n; the output of the algorithm is the deployment decision. If the number of GPUs is odd, for each GPU i from number 1 to number n-1, let the i-th GPU pull The parameters are transferred between the (i / 2, i / 2+1)th GPUs, and the other GPUs in the same group pull the other part of the parameters, and the GPU numbered n will communicate with the GPU numbered n-1; if the number of GPUs is even, for each GPU i, let the i-th GPU pull The parameters are transferred between the (i / 2, i / 2+1)th GPU and the other GPUs in the same group pull other parameters from each other. It should be noted that (i / 2, i / 2+1) represents the result after integer division.

[0082] Furthermore, the two branch sets include a first branch set and a second branch set. When the computing node group consists of the first computing node and the second computing node, each computing node pulls all parameters in the corresponding branch set from the parameter server to perform gradient calculation, and at the same time pulls parameters corresponding to other branch sets from other computing nodes in the same computing node group to perform gradient calculation, including:

[0083] The first computing node pulls the first parameters corresponding to all intermediate network layers in the first branch set from the parameter server, and the second computing node pulls the second parameters corresponding to all intermediate network layers in the second branch set from the parameter server;

[0084] The first computing node performs gradient calculation based on the pulled first parameter, and simultaneously pulls the second parameter from the second computing node and performs gradient calculation based on the pulled second parameter;

[0085] The second computing node performs gradient calculation based on the pulled second parameter, and simultaneously pulls the first parameter from the first computing node and performs gradient calculation based on the pulled first parameter.

[0086] For example, in this embodiment, it is assumed that the number of GPUs is 4 (i.e., GPU1, GPU2, GPU3, and GPU4), and computing node group 1 includes GPU1 and GPU2, and computing node group 2 includes GPU3 and GPU4. Taking the parameter transfer and calculation of computing node group 1 as an example: GPU1 pulls the parameters of I2 to I5 from the parameter server, and GPU2 pulls the parameters of I6 to I8 from the parameter server; when GPU1 finishes pulling the parameters of I2 to I5, it immediately performs forward calculation on I1 to I5, and at the same time pulls the parameters of I6 to I8 from GPU2; similarly, when GPU2 finishes pulling the parameters of I6 to I8, it immediately performs forward calculation on I6 to I8, and at the same time pulls the parameters of I2 to I5 from GPU1.

[0087] It is understandable that in the parameter server architecture, the calculation is divided into two main stages: forward pass and backward pass. Among them, in the forward pass stage, the input data and the model on the parameter server are sent to multiple computing nodes in the computing cluster for processing. Each computing node will process a part of the data according to the model on the server and calculate the forward propagation result corresponding to the data. In the backward pass stage, each computing node performs backward propagation calculation based on the result calculated by the forward propagation to guide the direction of model update, and the gradient calculated by the backward propagation is sent back to the computing node and used to update the local model weights. Therefore, after completing the forward propagation calculation, the GPU in this embodiment will perform backward propagation calculation based on the result calculated by the forward propagation to obtain the final gradient calculation result.

[0088] Furthermore, the two branch sets include a first branch set and a second branch set. When the computing node group consists of a first computing node, a second computing node, and a third computing node, each computing node pulls all parameters in the corresponding branch set from the parameter server to perform gradient calculation, and at the same time pulls parameters corresponding to other branch sets from other computing nodes in the same computing node group to perform gradient calculation, including:

[0089] The first computing node pulls the first parameters corresponding to all intermediate network layers in the first branch set from the parameter server, and the second computing node and the third computing node respectively pull the second parameters corresponding to all intermediate network layers in the second branch set from the parameter server;

[0090] The first computing node performs gradient calculation based on the pulled first parameter, and simultaneously pulls the second parameter from the second computing node or the third computing node and performs gradient calculation based on the pulled second parameter;

[0091] The second computing node and the third computing node respectively perform gradient calculation based on the pulled second parameter, and at the same time respectively pull the first parameter from the first computing node and perform gradient calculation based on the pulled first parameter.

[0092] For example, in this embodiment, it is assumed that the number of GPUs is 5 (i.e., GPU1, GPU2, GPU3, GPU4, and GPU5), and computing node group 1 includes GPU1 and GPU2, and computing node group 2 includes GPU3, GPU4, and GPU5. Taking the parameter transfer and calculation of computing node group 2 as an example: GPU3 pulls the parameters of I2 to I5 from the parameter server, and GPU4 and GPU5 pull the parameters of I6 to I8 from the parameter server; when GPU3 finishes pulling the parameters of I2 to I5, it immediately performs forward calculation on I1 to I5, and at the same time pulls the parameters of I6 to I8 from GPU4 or GPU5; similarly, when GPU4 finishes pulling the parameters of I6 to I8, it immediately performs forward calculation on I6 to I8, and at the same time pulls the parameters of I2 to I5 from GPU3; when GPU5 finishes pulling the parameters of I6 to I8, it immediately performs forward calculation on I6 to I8, and at the same time pulls the parameters of I2 to I5 from GPU3.

[0093] Step S40: Transmitting the gradient calculation result back to the parameter server, so that the parameter server updates the training target nonlinear network based on the gradient calculation result.

[0094] For example, it can be understood that the calculation process of the parameter server architecture can be summarized as follows: the input data and the global model on the parameter server are distributed to different computing nodes for forward and backward transfer calculations to generate calculation results; each computing node calculates the gradient based on the calculation results and transmits it back to the parameter server; after the parameter server calculates the average gradient of all computing nodes, it updates the network model on the parameter server; repeat the above process until the model training is completed.

[0095] Therefore, in this embodiment, the parameter server architecture also includes a parameter synchronization phase and a model output phase. In the parameter synchronization phase, the gradients calculated by each computing node according to the algorithm that overlaps communication and computation are sent back to the parameter server so that they can be averaged with the weights of other computing nodes. Therefore, after the parameter server receives all the gradients, it will average the gradients and update the network model (i.e., the target nonlinear network) on the parameter server based on the average gradients. In the model output phase, the trained model will be used to infer new input data and output the corresponding results.

[0096] In summary, this embodiment not only proposes a model parameter partitioning algorithm for nonlinear networks, that is, modeling the model structure of the nonlinear network and dividing it into different parts that are computationally independent of each other, so as to perform parallel deployment of the model structure between these parts, but also proposes a distributed communication optimization algorithm for nonlinear networks, that is, deploying different parts of the model on different computing nodes based on the model parameter partitioning algorithm, and utilizing the idle bandwidth between different computing nodes and the computing-communication coverage algorithm to optimize the deployment of the model and the communication process between nodes, thereby accelerating the training speed of the nonlinear network in the cluster.

[0097] This embodiment will be further explained below in conjunction with experimental data.

[0098] 1. Experimental conditions and training data

[0099] The experiments were conducted on a private GPU cluster consisting of 11 homogeneous GPU servers. Each server was equipped with an RTX-3080, a 3.70GHz i9-10900X Intel CPU, 64GB of memory, and a 1Gbps Ethernet connection. All servers were connected to a Huawei S1700-24GR switch and ran Ubuntu 16.04.

[0100] Experiments were conducted on two different nonlinear network models: DFN and InceptionV4. The DFN model is a simple model used to test convolutional network ensembles, with a total of 25.41M parameters. InceptionV4 is a deep convolutional neural network model that has performed well in many computer vision tasks, such as image classification, object detection, and semantic segmentation, with a total of 42.68M parameters.

[0101] Experiments were conducted on the CIFAR-10 dataset. CIFAR-10 (Canadian Institute for Advanced Research) is a classic machine learning and computer vision dataset, commonly used to test the performance of image classification algorithms. Due to its relatively small size and simple images, it is often used as a benchmark dataset to verify the performance of new algorithms. The dataset contains 10 different categories, each containing 6,000 32×32 pixel color images, of which 50,000 are used for training and 10,000 are used for testing.

[0102] 2. Results and Analysis

[0103] (1) Overall performance analysis

[0104] See also Figure 6 and Figure 7As shown, in terms of the overall performance of the algorithm, the algorithm in this embodiment can effectively reduce transmission congestion compared to the traditional parameter server architecture (such as the PS architecture), thereby improving the overall training speed. Specifically, in the DFN and InceptionV4 models, the algorithm of this embodiment has an improvement of about 24.6% compared to the traditional parameter server architecture, and the improvement gradually increases with the size of the cluster. This is because as the cluster size increases, the bandwidth upper limit of the cluster is gradually approaching. Therefore, when the cluster size increases, the communication bottleneck in the parameter transfer process will gradually increase, and the effect of the method proposed in this embodiment on congestion control will gradually become apparent.

[0105] Among the two compared models, DFN and InceptionV4, the method of this embodiment has better performance on DFN than InceptionV4. This is because DFN has higher nonlinearity and can better achieve the overlap of computing and communication between GPUs, thereby reducing communication overhead and improving overall training efficiency.

[0106] (2) Additional cost analysis

[0107] The additional overhead of the algorithm should be given extra attention. This embodiment experimentally measured the runtime overhead, where the Packing algorithm and the Deploying algorithm are both linear time complexity algorithms. Therefore, after experimental measurement, it was found that both algorithms took only a few milliseconds during the training process. Compared with the communication time and training time, the overhead of the Packing algorithm and the Deploying algorithm is very low and can be ignored. This shows that the method provided in this embodiment can significantly reduce the training time, thereby achieving better distributed training results.

[0108] The present application also provides a distributed deep learning acceleration device, including:

[0109] A branch packaging unit is used to: perform branch packaging processing on multiple parallel network branches in the target nonlinear network to obtain two branch sets and store them in a parameter server, where each branch set includes parameters contained in the corresponding network branch;

[0110] A node grouping unit, which is used to group multiple computing nodes in the computing cluster to obtain multiple computing node groups;

[0111] The transfer calculation unit is used to pull all parameters in the corresponding branch set from the parameter server for each computing node group to perform gradient calculation, and at the same time pull all parameters in other branch sets from other computing nodes in the same computing node group to perform gradient calculation;

[0112] The updating training unit is used to transmit the gradient calculation result back to the parameter server so that the parameter server can update the training target nonlinear network based on the gradient calculation result.

[0113] Furthermore, the node grouping unit is specifically used to:

[0114] Group multiple computing nodes in the computing cluster into pairs to obtain multiple computing node groups;

[0115] When the number of computing nodes is an odd number, one computing node group has three computing nodes.

[0116] Furthermore, the two branch sets include a first branch set and a second branch set. When the computing node group consists of the first computing node and the second computing node, the transfer computing unit is specifically configured to:

[0117] The first computing node pulls first parameters corresponding to all intermediate network layers in the first branch set from the parameter server, and the second computing node pulls second parameters corresponding to all intermediate network layers in the second branch set from the parameter server;

[0118] The first computing node performs gradient calculation based on the first parameter pulled, and simultaneously pulls the second parameter from the second computing node and performs gradient calculation based on the second parameter pulled;

[0119] The second computing node is caused to perform gradient calculation based on the pulled second parameter, and the first parameter is pulled from the first computing node and gradient calculation is performed based on the pulled first parameter.

[0120] Furthermore, the two branch sets include a first branch set and a second branch set. When the computing node group consists of a first computing node, a second computing node, and a third computing node, the transfer computing unit is further configured to:

[0121] The first computing node pulls the first parameters corresponding to all intermediate network layers in the first branch set from the parameter server, and the second computing node and the third computing node respectively pull the second parameters corresponding to all intermediate network layers in the second branch set from the parameter server;

[0122] The first computing node performs gradient calculation based on the pulled first parameter, and simultaneously pulls the second parameter from the second computing node or the third computing node and performs gradient calculation based on the pulled second parameter;

[0123] The second computing node and the third computing node respectively perform gradient calculation based on the pulled second parameter, and at the same time respectively pull the first parameter from the first computing node and perform gradient calculation based on the pulled first parameter.

[0124] Furthermore, the branch packaging unit is specifically used to:

[0125] Multiple parallel network branches are packaged according to the parameter sizes of the network branches to obtain two branch sets.

[0126] Furthermore, the branch packaging unit is further configured to:

[0127] The target nonlinear network is divided into multiple parallel network branches.

[0128] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and each unit can refer to the corresponding processes in the aforementioned distributed deep learning acceleration method embodiment, and will not be repeated here.

[0129] The apparatus provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in Figure 8 The distributed deep learning acceleration device shown is running on the

[0130] An embodiment of the present application also provides a distributed deep learning acceleration device, comprising: a memory, a processor, and a network interface connected via a system bus, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement all or part of the steps of the aforementioned distributed deep learning acceleration method.

[0131] Among them, the network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0132] The processor may be a CPU, other general-purpose processors, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor, or any conventional processor. The processor is the control center of a computer device, connecting various parts of the entire computer device using various interfaces and lines.

[0133] The memory can be used to store computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function (such as a video playback function, an image playback function, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as video data, image data, etc.). In addition, the memory can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, an SMC (SmartMediaCard, smart memory card), an SD (Secure Digital) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0134] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, all or part of the steps of the aforementioned distributed deep learning acceleration method are implemented.

[0135] The embodiments of the present application implement all or part of the aforementioned processes, and may also be completed by instructing the relevant hardware through a computer program. The computer program may be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above methods may be implemented. The computer program includes computer program code, and the computer program code may be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, ROM (Read-Only memory), RAM (Random Access memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, servers, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0139] The foregoing is merely a list of specific embodiments of the present application, intended to enable those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the broadest scope consistent with the principles and novel features of the present application.

Claims

1. A distributed deep learning acceleration method, characterized in that: The following steps are involved: Perform branch packaging processing on multiple parallel network branches in the target nonlinear network to obtain two branch sets and store them in the parameter server, where each branch set includes the parameters contained in the corresponding network branch; Grouping multiple computing nodes in a computing cluster to obtain multiple computing node groups; For each computing node group, each computing node pulls all parameters in the corresponding branch set from the parameter server for gradient calculation, and at the same time pulls all parameters in other branch sets from other computing nodes in the same computing node group for gradient calculation; The gradient calculation result is transmitted back to the parameter server, so that the parameter server can update the training target nonlinear network based on the gradient calculation result.

2. The distributed deep learning acceleration method according to claim 1, wherein: The step of grouping multiple computing nodes in the computing cluster to obtain multiple computing node groups includes: Group multiple computing nodes in the computing cluster into pairs to obtain multiple computing node groups; When the number of computing nodes is an odd number, one computing node group has three computing nodes.

3. The distributed deep learning acceleration method according to claim 2, wherein: The two branch sets include a first branch set and a second branch set. When the computing node group consists of the first computing node and the second computing node, each computing node pulls all parameters in the corresponding branch set from the parameter server to perform gradient calculation, and at the same time pulls parameters corresponding to other branch sets from other computing nodes in the same computing node group to perform gradient calculation, including: The first computing node pulls the first parameters corresponding to all intermediate network layers in the first branch set from the parameter server, and the second computing node pulls the second parameters corresponding to all intermediate network layers in the second branch set from the parameter server; The first computing node performs gradient calculation based on the pulled first parameter, and simultaneously pulls the second parameter from the second computing node and performs gradient calculation based on the pulled second parameter; The second computing node performs gradient calculation based on the pulled second parameter, and simultaneously pulls the first parameter from the first computing node and performs gradient calculation based on the pulled first parameter.

4. The distributed deep learning acceleration method according to claim 2, wherein: The two branch sets include a first branch set and a second branch set. When the computing node group consists of a first computing node, a second computing node, and a third computing node, each computing node pulls all parameters in the corresponding branch set from the parameter server to perform gradient calculation, and at the same time pulls parameters corresponding to other branch sets from other computing nodes in the same computing node group to perform gradient calculation, including: The first computing node pulls the first parameters corresponding to all intermediate network layers in the first branch set from the parameter server, and the second computing node and the third computing node respectively pull the second parameters corresponding to all intermediate network layers in the second branch set from the parameter server; The first computing node performs gradient calculation based on the pulled first parameter, and simultaneously pulls the second parameter from the second computing node or the third computing node and performs gradient calculation based on the pulled second parameter; The second computing node and the third computing node respectively perform gradient calculation based on the pulled second parameter, and at the same time respectively pull the first parameter from the first computing node and perform gradient calculation based on the pulled first parameter.

5. The distributed deep learning acceleration method according to claim 1, wherein: The branch packaging process is performed on multiple parallel network branches in the target nonlinear network to obtain two branch sets, including: Multiple parallel network branches are packaged according to the parameter sizes of the network branches to obtain two branch sets.

6. The distributed deep learning acceleration method according to claim 1, wherein: Before the step of performing branch packaging processing on the multiple parallel network branches in the target nonlinear network, the method further includes: The target nonlinear network is divided into multiple parallel network branches.

7. A distributed deep learning acceleration device, characterized in that: include: A branch packaging unit is used to: perform branch packaging processing on multiple parallel network branches in the target nonlinear network to obtain two branch sets and store them in a parameter server, where each branch set includes parameters contained in the corresponding network branch; A node grouping unit, which is used to group multiple computing nodes in the computing cluster to obtain multiple computing node groups; The transfer calculation unit is used to pull all parameters in the corresponding branch set from the parameter server for each computing node group to perform gradient calculation, and at the same time pull all parameters in other branch sets from other computing nodes in the same computing node group to perform gradient calculation; The updating training unit is used to transmit the gradient calculation result back to the parameter server so that the parameter server can update the training target nonlinear network based on the gradient calculation result.

8. The distributed deep learning acceleration device according to claim 7, wherein: The two branch sets include a first branch set and a second branch set. When the computing node group consists of the first computing node and the second computing node, the transfer computing unit is specifically configured to: The first computing node pulls first parameters corresponding to all intermediate network layers in the first branch set from the parameter server, and the second computing node pulls second parameters corresponding to all intermediate network layers in the second branch set from the parameter server; The first computing node performs gradient calculation based on the first parameter pulled, and simultaneously pulls the second parameter from the second computing node and performs gradient calculation based on the second parameter pulled; The second computing node is caused to perform gradient calculation based on the pulled second parameter, and the first parameter is pulled from the first computing node and gradient calculation is performed based on the pulled first parameter.

9. A distributed deep learning acceleration device, characterized in that: include: A memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the distributed deep learning acceleration method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the distributed deep learning acceleration method according to any one of claims 1 to 6 is implemented.

Images