Production line parallel training node weight distribution method based on version difference

Abstract

The invention discloses a version difference-based assembly line parallel training node weight distribution method. The method comprises the following steps of: loading an initialized deep learning model; constructing a pipeline parallel training scheme; by using an asynchronous parameter updating method, executing the training of different batches concurrently, and recording the training batches to complete forward and backward transmission processes within unit pipeline execution time; and predicting a node weight in a future training process by using the latest training node weight version, and performing version difference calculation of the batch according to node weight prediction; and after the node completes version difference calculation of all batches, completing prediction weight updating of the node. The version difference of all the nodes is calculated, that is, assembly line parallel training node weight distribution is completed; besides, the data is deployed to heterogeneous computing nodes to obtain an assembly line parallel training node weight allocation scheme for the to-be-trained target network. According to the method, the node weight prediction is more accurate.

Description

technical field [0001] The invention relates to a weight distribution method for pipeline parallel training nodes based on version differences, and belongs to the technical field of computer pipeline system optimization. Background technique [0002] Deep neural networks are widely used in various fields, and have achieved predictive effects that surpass humans. As the requirements for the accuracy of the model become higher and higher, the scale of model parameters and computing requirements become larger and larger, and training the model becomes a very computationally intensive and time-consuming task. Researchers often use distributed computer clusters to speed up the model training process. Distributed deep learning parallel training is dedicated to accelerating the DNN model training process, and has been studied by many scholars. Among them, the research on pipeline parallel training is getting deeper and deeper. Pipeline parallel training can solve the problem of ...

AI Technical Summary

Problems solved by technology

[0003] Purpose of the invention: In order to solve the problem of outdated node weights in pipeline parallel training and the low accuracy of existing node weight prediction methods, the present invention provides a method for assigning node weights for pipeline parallel training based on version differences, based on asynchronous pipeline parallel training , making the node weight prediction more accurate, further promoting the improvement of the accuracy of the deep learning model, and ensuring the effectiveness of the model training

Images