A distributed training method based on hybrid parallelism

Abstract

The present invention discloses a distributed model training method based on hybrid parallelism, which relates to the technical field of deep neural network model training. It adopts a hybrid parallel method of data parallelism and model parallelism, and uses multiple nodes and multiple GPUs to solve the above problems. First, in order to solve the problem of long training time, a distributed cluster method is used to perform parallel calculations on massive data to improve the training speed; secondly, in order to solve the problem that the classification layer model occupies too much video memory during training, a model parallel method is used to divide the classification layer model into several parts, and deploy it on multiple GPUs of multiple nodes in the cluster. At the same time, the number of nodes can be dynamically adjusted according to the size of the classification layer model to meet the classification model training under large ID conditions. The present invention uses a hybrid parallel method based on data parallelism and model parallelism, and uses distributed cluster training. While maintaining the original deep learning training effect, it can greatly improve the model training efficiency and meet the classification model training under large IDs.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology for deep neural network model training, and in particular to a distributed training method based on hybrid parallelism. Background Art

[0002] The concept of deep learning originated from the research on artificial neural networks. It is a systematic science based on the theory of computer neural networks and machine learning. It extracts and represents information through multi-layer neural networks, combines the underlying features to form more abstract high-level features, and learns the underlying rules of data samples.

[0003] With the increasing requirements of industrial applications, large-scale model structure design and massive data model training have become mainstream methods, which has increased the complexity and cost of deep learning. For example, when training a massive ID face recognition model, it takes nearly a day to train hundreds of thousands of IDs on a single GPU, and model tuning may take several weeks, which is time-consuming. In addition, as the scale of IDs continues to increase, the amount of GPU memory occupied by the classification layer will increase exponentially, and there may even be a problem that the number of single-layer network parameters is too large, resulting in the inability of a single card's memory to accommodate the model. Therefore, in the field of massive data and large-scale model structure training, how to reduce training costs and how to solve the problem of model memory occupation have become research hotspots in the field of deep learning.

[0004] At present, the industry usually adopts data parallel solutions to reduce training time, and adopts model parallel solutions to solve the problem that the model is too large to train on a single GPU. Among them, data parallelism is to split the training data, use multiple model instances, and train the split data in parallel; model parallelism refers to slicing different layers of the network model and deploying them to different GPUs. During the data parallel training process, the parameter server is responsible for parameter exchange between models to ensure that all split data can affect the model parameters at the same time; during the model parallel training process, different layers of the network model are sliced ​​and deployed on different GPUs to reduce the number of model parameters on a single GPU, thereby avoiding video memory limitations.

[0005] However, the data parallel approach is to deploy the same network model on each machine, which cannot solve the problem that the single-layer model is too large to fit in the single-card video memory. In addition, the traditional model parallel solution officially integrated by the general neural network training framework PyTorch and TensorFlow can only deploy different layers of the network to different GPUs respectively, and still cannot solve the problem that the number of single-layer parameters exceeds the single-card video memory and the model cannot be trained. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a distributed training method based on hybrid parallelism to address the problems of long training time in massive ID data training and the inability to train the model due to the number of model parameters exceeding the GPU video memory. The method adopts a hybrid parallel method of data parallelism and model parallelism, and uses multiple nodes and multiple GPUs to solve the above problems.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] A distributed training method based on hybrid parallelism specifically includes the following steps:

[0009] Step 1: Build the model network structure, including input layer, convolution layer, pooling layer, fully connected layer and classification layer, and include two parts: feature extraction network and classification network;

[0010] Step 2: Design a distributed data parallel training architecture;

[0011] Step 3: Design a distributed model parallel training architecture;

[0012] Step 4: Design a distributed hybrid parallel training architecture;

[0013] Step 5: Design a distributed neural network back-propagation mechanism to perform back-propagation on the feature extraction network and update the gradient to complete the distributed deep neural network training based on hybrid parallelism.

[0014] As a further preferred solution of the distributed training method based on hybrid parallelism of the present invention, step 2 is specifically as follows: the distributed data parallel training architecture adopts Ring-Allreduce for data communication, and each Worker synchronizes its own gradient to the next Worker in turn, and completes the update of all Worker gradients in turn to avoid the bandwidth and memory bottlenecks of a single master node in ParameterServer. At the same time, the overall communication framework adopts Nvidia's NCCL method for data communication.

[0015] As a further preferred scheme of the distributed training method based on hybrid parallelism of the present invention, the step 3 is specifically as follows: based on the distributed data parallel training architecture of step 2 and the model network structure constructed based on step 1, a node is selected to deploy the feature extraction model, and the classification network layer is changed to a model parallel mode, that is, the single-layer model of the classification layer is sliced ​​into several parts, and deployed on the GPUs of other nodes respectively. At the same time, the feature extraction model uses a broadcast mechanism to send the feature information extracted by the convolutional neural network to each shard model to ensure that each shard model obtains the same feature information. Finally, the GPU where each shard model is located independently performs the forward propagation process and obtains its own loss value.

[0016] As a further preferred scheme of a distributed training method based on hybrid parallelism of the present invention, step 4 is specifically as follows: the distributed data parallel training architecture designed based on step 2 and the distributed model parallel training architecture designed based on step 3 respectively split the feature extraction model and the classification layer model into different nodes and different GPUs, which can effectively reduce the training time and thus accelerate the training; the data parallel architecture established based on step 2 and the model parallel architecture established based on step 3 are integrated to realize the hybrid parallel model training architecture.

[0017] As a further preferred embodiment of the hybrid parallel distributed training method of the present invention, step 5 is specifically as follows: based on the distributed hybrid parallel training architecture designed in step 4, the loss values ​​of the GPUs of all classification layers in the cluster are calculated respectively, and the All-Reduce algorithm is executed on the loss values ​​between all GPUs to calculate the total loss value; when performing back propagation, the gradient error of the classification layer is reversely calculated on different GPUs respectively, and the model parameters of the classification layer are updated simultaneously;

[0018] For the forward propagation process of the face recognition classification layer, let Represents the input features of the classification layer, then the input features need to be mapped to the cosine space, and the calculation formula is as follows:

[0019]

[0020] In the formula, is a hyperparameter, represents the input features of the classification layer, Indicates classification as The weight vector of Respectively The unit vector of express Classified as The classification score of Representation characteristics With weight The angle of

[0021] The cross entropy loss is calculated as follows:

[0022]

[0023] In the formula, express Classified as The classification probability, C represents the total number of classifications; represents the feature vector of the output layer, Indicates the category number. Indicates real categories;

[0024] Therefore, in back propagation, it is necessary to calculate and The gradient of is calculated as follows:

[0025]

[0026]

[0027] In the formula, when The value is 1 when , otherwise it is 0. and The calculation formula is as follows:

[0028]

[0029]

[0030] At the same time, ArcFace Loss is used, so the above formula The calculation formula is as follows:

[0031]

[0032] From this, we can get the input for the classification layer respectively and weight The gradient of each shard model is calculated, and the classification layer model parameters on different GPUs in different nodes are updated; at the same time, the classification layer errors on different GPUs are accumulated using the All-Reduce algorithm, and transmitted to the last layer of the feature extraction network using the NCCL communication mechanism, and finally the feature extraction network is back-propagated based on the back-propagation mechanism mentioned in step 2 and the gradient is updated to complete the distributed deep neural network training based on hybrid parallelism. Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0033] 1. The present invention aims to solve the problems of long training time and inability to train models due to the number of model parameters exceeding the GPU memory in the training of massive ID data. The present invention adopts a hybrid parallel method of data parallelism and model parallelism, and uses multi-nodes and multi-GPUs to solve the above problems.

[0034] 2. To address the problem of long training time, the present invention adopts a distributed cluster method to perform parallel calculations on massive data to improve the training speed; secondly, to address the problem of excessive video memory occupied by the classification layer model during training, a model parallel method is adopted to divide the classification layer model into several parts and deploy them on multiple GPUs of multiple nodes in the cluster. At the same time, the number of nodes can be dynamically adjusted according to the size of the classification layer model to meet the classification model training requirements under large ID conditions;

[0035] 3. The present invention adopts a hybrid parallel method based on data parallelism and model parallelism, and uses distributed cluster training. While maintaining the original deep learning training effect, it can greatly improve the model training efficiency and meet the classification model training under large IDs. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is the overall structure diagram of the face recognition network model of the present invention;

[0037] Figure 2 It is a schematic diagram of the data parallel structure of the present invention;

[0038] Figure 3 It is a schematic diagram of the parallel structure of the model of the present invention;

[0039] Figure 4 It is a schematic diagram of obtaining respective loss values ​​in the present invention;

[0040] Figure 5 It is a schematic diagram of the hybrid parallel architecture of the present invention. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings:

[0042] The invention is further described in detail in conjunction with the accompanying drawings, at least one optimal embodiment is specifically described. The degree of specificity of this description should be such that technicians in the relevant technical field can reproduce the invention or utility model according to the described content without having to expend creative labor, such as conducting exploratory research and experiments.

[0043] A distributed model training method based on hybrid parallelism is described in detail here with reference to an example of a face recognition algorithm for large-scale IDs. The method specifically includes the following steps:

[0044] Step 1. Build a face recognition network model, where the feature extraction network (backbone) can be selected from the commonly used

[0045] Resnet50 model, the loss function uses ArcFace Loss. The overall structure is shown in the figure Figure 1 As shown:

[0046] Step 2. Design a distributed data parallel training architecture. The distributed data parallel architecture of the present invention adopts the Ring-Allreduce architecture (based on the protocol), and each node serves as a Worker. In the forward reasoning stage, each Worker is responsible for extracting the image features of part of the data, and then uses the All-gather mechanism to gather all image features, and uses the broadcast mechanism to send the image features to all classification layer nodes. In the back propagation stage, the classification layer propagates the error to the last layer of the feature extraction network, and updates the parameters of the feature extraction network according to the customized back propagation algorithm. Each Worker synchronizes its own gradient to the next Worker in turn, and completes the update of all Worker gradients in order to avoid the bandwidth and memory bottlenecks of the single master node in the Parameter Server. At the same time, the overall communication framework uses Nvidia's NCCL method for data communication to improve parallel efficiency. The data parallel structure diagram is shown in the figure. Figure 2 shown.

[0047] Step 3. Design a distributed model parallel training architecture. Based on the data parallel solution in step 2, select a node to deploy the feature extraction model, and change the classification network layer to model parallel mode, that is, slice the single-layer model of the classification layer into several pieces and deploy them on the GPUs of other nodes respectively. At the same time, the node where the feature extraction model is located uses the broadcast mechanism to distribute all feature information to each model slice, so that each model slice is trained with all data. Finally, each GPU independently performs the forward propagation process and obtains its own loss value, as shown in the figure. Figure 3-4 As shown:

[0048] In particular, the traditional hybrid parallel solution uses a model parallel solution on different GPUs of the same node server, and a data parallel solution between nodes. In the model parallel stage of the present invention, the deployment of the classification layer model is not limited to the GPU in a single node, but can also be a GPU across nodes, which greatly increases the processing power of the classification layer, and can dynamically adjust the number of nodes according to the size of the ID scale.

[0049] Step 4. Design a distributed training scheme based on hybrid parallelism. Based on the data parallel scheme designed in step 2, use 2 machines for data parallelism; based on step 3, design a model parallel scheme, use 3 machines for model parallelism, and form a distributed cluster. The data parallel machine is responsible for extracting image features, and the model parallel machine is responsible for classifying the features, thereby realizing the image classification task. First, the two data parallel machines are responsible for extracting the image features of the training data set. Then, the features extracted by the two machines are aggregated using All-gather and sent to the three model parallel machines. After the model parallel machine classifies the image features, it uses the loss function to calculate the corresponding errors respectively. Finally, the corresponding parameter gradient of the error is calculated through the hybrid parallel back-propagation mechanism established in step 5, and all network parameters are updated using SGD (stochastic gradient descent) to realize distributed training. Hybrid parallel architecture

[0050] Schematic diagram as Figure 5 shown.

[0051] Step 5. Design a distributed neural network back-propagation mechanism. Based on the hybrid parallel solution in step 4, calculate the loss values ​​for all GPUs of the classification layer in the cluster, and then perform the All-Reduce algorithm on the loss values ​​between all GPUs to calculate the total loss value. When performing back-propagation, first reversely calculate the gradient error of the classification layer on different GPUs, and update the model parameters of the classification layer at the same time.

[0052] For the forward propagation process of the face recognition classification layer, let Represents the input features of the classification layer, then the input features need to be mapped to the cosine space, and the calculation formula is as follows:

[0053]

[0054] In the formula, is a hyperparameter, Indicates classification as The weight vector of Respectively The unit vector of express Classified as The classification score of Representation characteristics With weight Angle.

[0055] Then calculate its cross entropy loss as follows:

[0056]

[0057] In the formula, express Classified as The classification probability of , C represents the total number of classifications.

[0058] Therefore, in back propagation, we need to calculate and The gradient of is calculated as follows:

[0059]

[0060]

[0061] In the formula, when The value is 1 when , otherwise it is 0. and The calculation formula is as follows:

[0062]

[0063]

[0064] At the same time, this example uses ArcFace Loss, so the above formula The calculation formula is as follows:

[0065]

[0066] From this, we can get the input for the classification layer respectively and weight The gradient of each shard model is calculated separately, and the classification layer model parameters on different GPUs in different nodes are updated. At the same time, the classification layer errors on different GPUs are accumulated using the All-Reduce algorithm and transmitted to the last layer of the feature extraction network using the NCCL communication mechanism. Finally, the feature extraction network is back-propagated based on the back-propagation mechanism mentioned in step 2 and the gradient is updated to complete the distributed deep neural network training based on hybrid parallelism.

[0067] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0068] The above embodiments are only for explaining the technical idea of ​​the present invention, and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution according to the technical idea proposed by the present invention shall fall within the protection scope of the present invention. The above embodiments of the present invention are described in detail, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A distributed training method based on hybrid parallelism, characterized in that: The specific steps include: Step 1: Build a face recognition network model, where the feature extraction network backbone uses the commonly used Resnet50 model and the loss function uses ArcFace Loss; Construct the model network structure, including input layer, convolution layer, pooling layer, fully connected layer and classification layer, and include two parts: feature extraction network and classification network; Step 2: Design a distributed data parallel training architecture. The distributed data parallel architecture adopts the RingAllreduce architecture, with each node acting as a Worker. In the forward inference phase, each Worker is responsible for extracting image features from part of the data, and then uses the All-gather mechanism to aggregate all image features, and uses the broadcast mechanism to send the image features to all classification layer nodes. In the back-propagation phase, the classification layer propagates the error to the last layer of the feature extraction network, and updates the parameters of the feature extraction network according to the custom back-propagation algorithm. Each Worker synchronizes its own gradient to the next Worker in turn, and completes the update of all Worker gradients in this way to avoid the bandwidth and memory bottlenecks of a single master node in the Parameter Server. At the same time, the overall communication framework uses Nvidia's NCCL method for data communication to improve parallel efficiency. Step 3, design a distributed model parallel training architecture; based on the data parallel solution in step 2, select a node to deploy the feature extraction model, and change the classification network layer to model parallel mode, that is, slice the single-layer model of the classification layer into several pieces, and deploy them on the GPUs of other nodes respectively. At the same time, the node where the feature extraction model is located uses the broadcast mechanism to distribute all feature information to each model slice, so that each model slice is trained with all data; finally, each GPU independently performs the forward propagation process and obtains its own loss value. The traditional hybrid parallel solution uses the model parallel solution on different GPUs of the same node server, and the data parallel solution between nodes; Step 4, design a distributed hybrid parallel training architecture; based on the data parallel solution designed in step 2, use 2 machines for data parallelism; based on step 3, design a model parallel solution, use 3 machines for model parallelism, and form a distributed cluster; the data parallel machine is responsible for extracting image features, and the model parallel machine is responsible for classifying the features, thereby realizing the image classification task; the 2 data parallel machines are responsible for extracting image features of the training data set; the features extracted by the 2 machines are aggregated using All-gather and sent to the 3 model parallel machines; after the model parallel machines classify the image features, the corresponding errors are calculated using the loss function; finally, the corresponding parameter gradient of the error is calculated through the hybrid parallel back propagation mechanism established in step 5, and all network parameters are updated using the SGD stochastic gradient descent method to realize distributed training; Step 5: Design a distributed neural network back-propagation mechanism to perform back-propagation on the feature extraction network and update the gradient to complete the distributed deep neural network training based on hybrid parallelism; The step 5 is specifically as follows: based on the distributed hybrid parallel training architecture designed in step 4, the loss values ​​of all GPUs of the classification layer in the cluster are calculated respectively, and the All-Reduce algorithm is executed on the loss values ​​between all GPUs to calculate the total loss value; when performing back propagation, the gradient error of the classification layer is reversely calculated on different GPUs respectively, and the model parameters of the classification layer are updated simultaneously; For the forward propagation process of the face recognition classification layer, let Represents the input features of the classification layer, then the input features need to be mapped to the cosine space, and the calculation formula is as follows: In the formula, s is a hyperparameter, represents the input features of the classification layer, represents the weight vector for classification j, Respectively The unit vector, f i,j express The classification score of being classified as j, θ i,j Representation characteristics With weight The angle of The cross entropy loss is calculated as follows: In the formula, express Classified as y i The classification probability, C represents the total number of classifications; Represents the feature vector of the output layer, i represents the category number, y i represents the i-th true category; Therefore, in back propagation, it is necessary to calculate and The gradient of is calculated as follows: In the formula, Γ(y i =j) when y i = j, the value is 1, otherwise it is 0. and The calculation formula is as follows: At the same time, ArcFace Loss is used, then in the above formula The calculation formula is as follows: Get the input for the classification layer respectively and weight The gradient of each shard model is calculated separately, and the classification layer model parameters on different GPUs on different nodes are updated; at the same time, the All-Reduce algorithm is used to accumulate the classification layer errors on different GPUs, and the NCCL communication mechanism is used to transmit them to the last layer of the feature extraction network. Finally, the feature extraction network is back-propagated based on the back-propagation mechanism mentioned in step 2 and the gradient is updated to complete the distributed deep neural network training based on hybrid parallelism.

Images