一种深度学习模型的流水线并行训练方法及系统
By employing a weight buffering strategy and a prefix sum partitioning algorithm, the problems of weight version inconsistency and obsolescence in pipelined parallel training are solved, thereby improving model training accuracy and GPU utilization, and optimizing training speed.
CN116185604BActive Publication Date: 2026-07-17SHANDONG COMP SCI CENTNAT SUPERCOMP CENT IN JINAN +1
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG COMP SCI CENTNAT SUPERCOMP CENT IN JINAN
- Filing Date
- 2022-12-13
- Publication Date
- 2026-07-17
AI Technical Summary
Technical Problem
Existing pipelined parallel training methods have shortcomings in terms of inconsistent weight versions and obsolescence, leading to reduced model training accuracy and wasted resources.
Method used
We employ a weight buffering strategy and a prefix sum partitioning algorithm to ensure that the same version of parameters is used for the same mini-batch of data during forward and backward propagation. We also update weights through a hybrid synchronous and asynchronous approach, combined with memory estimation and partitioning optimization, to improve GPU utilization and training efficiency.
Benefits of technology
It improves model training accuracy, saves computing resources (memory), optimizes time complexity, maximizes GPU utilization efficiency, and shortens training time.
✦ Generated by Eureka AI based on patent content.
Smart Images
Figure CN116185604B_ABST
Abstract
本发明提出了一种深度学习模型的流水线并行训练方法及系统，涉及机器学习技术领域，具体方案包括：获取要训练的模型，对模型中每个网络层所占用的内存量进行预估，得到内存预估序列；利用前缀和分区算法对内存预估序列进行分区，将分区均衡分配到流水线上的GPU中；将训练数据集分批连续传入流水线中，进行流水线并行训练；其中，并行训练过程中，采用同步加异步混合的权重缓冲方式，对网络层的权重进行更新；本发明采用一种权重缓冲策略，保证同一小批数据在执行前向传播和反向传播时使用的是同一个版本的参数，从而提高模型训练精度，节省计算资源内存。
Need to check novelty before this filing date? Find Prior Art