The invention discloses an asynchronous multidirectional
assembly line parallelization method for accelerating
large model training, which comprises the following steps: creating a plurality of
assembly line parallelization groups, and arranging equipment among the groups in different sequences; distributing the segmented model slices of the to-be-trained
large model to each device in each group in sequence, wherein each group comprises the complete to-be-trained
large model; any group is selected to serve as an optimizer registration parallel group, and optimizers are registered for the model slices of all the devices in the optimizer registration parallel group; training the to-be-trained large model according to an asynchronous multidirectional
pipeline scheduling mode, accumulating the gradients of the same model slice on each micro data batch by each device, and entering the next step immediately or at the position of a pipeline
cavitation bubble when the accumulated micro data batches form a data batch; and aggregating the accumulated gradients to the equipment of the optimizer registration parallel group by each equipment, and performing parameter updating by utilizing the aggregated gradients. According to the method, the convergence of training can be considered while high
throughput is ensured, and the storage overhead is reduced on the premise of not increasing extra communication overhead.