The invention discloses a two-way staggered
assembly line parallel method for accelerating
large model training, and compared with an existing method, the method introduces V-shaped staggered scheduling into a two-way
assembly line to form a two-way staggered
assembly line parallel mode, and improves the training efficiency of a
large model and the use balance of a
video memory. A V-shaped staggered
assembly line and an inverted V-shaped staggered
assembly line are fused to form a bidirectional staggered
assembly line, the number of parallel devices is increased, meanwhile, the calculation time of a single micro-batch is shortened, the
cavitation rate of the assembly line can be effectively reduced, and more balanced
video memory occupation is achieved; in addition, a communication optimization strategy is introduced for overlapping or reducing part of communication overhead, and global gradient synchronization is performed by optimizing pipeline equipment mapping and utilizing a high-speed network in a
server, so that the parallel training efficiency of the
large model is further improved. The invention further provides a two-way
interleaving assembly line parallel
system and equipment which are used for achieving the two-way
interleaving assembly line parallel method.