Heterogeneous cluster-oriented large model module-level training strategy optimization method and device and computer equipment

By acquiring large model module-level computation and communication information from heterogeneous clusters, the computation and memory usage in the pipeline stage are optimized, solving the problem of low resource utilization in heterogeneous clusters. This achieves coordinated optimization of computational load and memory resources, improving resource utilization and training stability.

CN120874971AActive Publication Date: 2025-10-31ZHEJIANG LAB
7 Cites 3 Cited by

Patent Information

Application Number
CN202511367686.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-24
Publication Date
2025-10-31
Estimated Expiration
2045-09-24

AI Technical Summary

Technical Problem

Current mainstream distributed training frameworks cannot effectively coordinate the differences in computing power, memory bandwidth, and communication among different accelerators in domestic heterogeneous clusters, resulting in low resource utilization.

Method used

By acquiring the large model module-level computational information and operator-level communication information of heterogeneous clusters, the initial distributed training strategy is determined, and the computation time and memory usage in the pipeline stage are optimized based on computation time and memory thresholds, thereby achieving a coordinated balance between computational load, memory usage and communication overhead.

Benefits of technology

It significantly improves the utilization rate of large model resources in heterogeneous clusters, enhances the utilization efficiency and training stability of computing devices, and optimizes the overall utilization rate of resources.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention relates to a large model module level training strategy optimization method and device for a heterogeneous cluster and computer equipment, and the method comprises the steps: obtaining the calculation information of a large model module level and the communication information of an operator level for the heterogeneous cluster; the calculation information comprises calculation time information and module video memory information of modules under different data scales under different distributed strategies and on different chips; the communication information is communication time delays of communication operators under different data scales; determining an initial distributed training strategy of each assembly line based on the module video memory information; based on the calculation time information and the communication information, calculation time of different stages of the assembly line is determined; and optimizing each initial distributed training strategy according to the calculation time and a video memory threshold value for bearing the large model equipment to obtain a target distributed training strategy. Through the method and the device, the problem of relatively low resource utilization rate of a large model oriented to heterogeneous clusters is solved.
Need to check novelty before this filing date? Find Prior Art