Heterogeneous cluster-oriented large model module-level training strategy optimization method and device and computer equipment

Abstract

The invention relates to a large model module level training strategy optimization method and device for a heterogeneous cluster and computer equipment, and the method comprises the steps: obtaining the calculation information of a large model module level and the communication information of an operator level for the heterogeneous cluster; the calculation information comprises calculation time information and module video memory information of modules under different data scales under different distributed strategies and on different chips; the communication information is communication time delays of communication operators under different data scales; determining an initial distributed training strategy of each assembly line based on the module video memory information; based on the calculation time information and the communication information, calculation time of different stages of the assembly line is determined; and optimizing each initial distributed training strategy according to the calculation time and a video memory threshold value for bearing the large model equipment to obtain a target distributed training strategy. Through the method and the device, the problem of relatively low resource utilization rate of a large model oriented to heterogeneous clusters is solved.

Description

Technical Field

[0001] This application relates to the field of large models, and in particular to methods, apparatus and computer equipment for optimizing module-level training strategies for large models in heterogeneous clusters. Background Technology

[0002] In recent years, large-scale models have rapidly evolved to the scale of hundreds of billions and trillions of parameters, placing extreme demands on computing power during their training process. Current international computing power blockades have exacerbated this computing power gap, making domestically developed AI accelerators a core support for building large-scale computing clusters and bridging this gap. Against this backdrop, a general distributed training technology for large-scale models on heterogeneous clusters will directly impact the resource utilization of these clusters.

[0003] Current mainstream automated distributed training frameworks, such as Alpa and Aceso, are primarily designed for homogeneous clusters, with their core architecture assuming that devices possess uniform computing power and communication capabilities. The drawback of these frameworks is that when deployed in heterogeneous clusters composed of different domestically produced AI accelerators, the significant differences in computing power, memory bandwidth, and communication protocols among the accelerators make it difficult to effectively coordinate the differences in computing power, memory, and communication bottlenecks between devices, resulting in low overall cluster resource utilization.

[0004] There is currently no effective solution to the problem of resource utilization needing improvement when dealing with large models of heterogeneous clusters in related technologies. Summary of the Invention

[0005] This embodiment provides a method, apparatus, and computer device for optimizing large model module-level training strategies for heterogeneous clusters, in order to solve the problem of resource utilization needing to be improved when dealing with large models for heterogeneous clusters in related technologies.

[0006] Firstly, this embodiment provides a method for optimizing large model module-level training strategies for heterogeneous clusters, including:

[0007] Acquire computational information at the module level and operator level at the operator level for large model clusters; the computational information includes computation time information and module memory information for modules with different data scales under different distributed strategies and on different chips; the communication information is the communication latency of communication operators under different data scales.

[0008] Based on the memory information of the module, determine the initial distributed training strategy for each pipeline.

[0009] Based on the computation time information and the communication information, the computation time of different stages of the pipeline is determined;

[0010] Based on the computation time and the memory threshold of the device supporting the large model, each of the initial distributed training strategies is optimized to obtain the target distributed training strategy.

[0011] In some embodiments, obtaining computational information at the module level and operator-level communication information for large model clusters in heterogeneous clusters includes:

[0012] Obtain computational information of one layer of the module in the large model for heterogeneous clusters;

[0013] The computation time information includes the forward computation time and backward computation time of the module; the module memory information includes the memory information of the module's input, gradient, weights, and activation value recalculation.

[0014] In some embodiments, determining the initial distributed training strategy for each pipeline based on the module's memory information includes:

[0015] Based on the module memory information and the accelerator's maximum memory value, determine the minimum and maximum memory values ​​of the pipeline corresponding to the preset microprocessor batch and the preset parallel mode.

[0016] Based on the minimum and maximum memory values, an initial distributed training strategy is determined for each pipeline.

[0017] In some embodiments, determining the computation time of different stages of the pipeline based on the computation time information and the communication information includes:

[0018] Based on the initial distributed training strategy of the pipeline, the module set corresponding to each stage of the pipeline is determined.

[0019] The computation time for different stages of the pipeline is determined based on the module set, the computation time information, and the communication information.

[0020] In some embodiments, optimizing each initial distributed training strategy based on the computation time and the memory threshold of the device supporting the large model to obtain the target distributed training strategy includes:

[0021] Determine whether the standard deviation of the computation time at different stages of the pipeline is less than a preset threshold;

[0022] If the standard deviation of the computation time is greater than or equal to the preset threshold, then move one level to the adjacent stage of the pipeline until the standard deviation of the computation time is less than the preset threshold.

[0023] Determine whether the running GPU memory at different stages of the pipeline is greater than the GPU memory threshold of the device carrying the large model. If it is greater than the GPU memory threshold, reduce the running GPU memory by decreasing the micro-batch size, enabling activation value recalculation, and adjusting the parallelism parameter until the running GPU memory at different stages of the pipeline is less than or equal to the GPU memory threshold of the device carrying the large model, and obtain the target distributed training strategy.

[0024] In some embodiments, the method further includes:

[0025] After obtaining the target distributed training strategy, the distributed training strategy with the shortest iteration time is selected as the optimal strategy based on a preset performance function.

[0026] In some embodiments, the module further includes a load balancing module with a hybrid expert model;

[0027] The load balancing module of the hybrid expert model is used to optimize the balanced distribution of words during the training of the large model.

[0028] In some embodiments, the method further includes:

[0029] Obtain bandwidth difference information between and within different nodes in the heterogeneous cluster;

[0030] Based on the bandwidth difference information, the communication links of the heterogeneous cluster are configured in a hierarchical manner for optimization.

[0031] Secondly, this embodiment provides a large model module-level training strategy optimization device for heterogeneous clusters, including: an acquisition module, an initialization module, and an optimization module;

[0032] The acquisition module is used to acquire computational information at the module level and operator-level communication information for large model clusters in heterogeneous clusters; the computational information includes computational time information and module memory information of modules under different data scales, different distributed strategies, and different chips; the communication information is the communication latency of communication operators under different data scales.

[0033] The initialization module is used to determine the initial distributed training strategy for each pipeline based on the module's memory information.

[0034] The optimization module is used to determine the computation time of different stages of the pipeline based on the computation time information and the communication information; and to optimize each initial distributed training strategy according to the computation time and the memory threshold of the large model device to obtain the target distributed training strategy.

[0035] Thirdly, this embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the large model module-level training strategy optimization method for heterogeneous clusters described in the first aspect above.

[0036] Compared with related technologies, the method, apparatus, and computer equipment for optimizing large model module-level training strategies for heterogeneous clusters provided in this embodiment acquire computational information at the module level and operator-level communication information for large model modules in heterogeneous clusters. The computational information includes computation time information and module memory information for modules with different data scales under different distributed strategies and on different chips. The communication information is the communication latency of communication operators under different data scales. Based on the module memory information, the initial distributed training strategy for each pipeline is determined. Based on the computation time information and communication information, the computation time for different stages of the pipeline is determined. Based on the computation time and the memory threshold of the large model device, each initial distributed training strategy is optimized to obtain the target distributed training strategy. This results in higher utilization efficiency of large model resources in heterogeneous clusters. By using a module-level unified abstraction layer for bidirectional decoupling between the model and hardware, a coordinated balance of computational load, memory usage, and communication overhead among heterogeneous devices is achieved, significantly improving resource utilization.

[0037] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0038] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0039] Figure 1 This is a hardware structure block diagram of a terminal device for a method to optimize large model module-level training strategies for heterogeneous clusters, provided in an embodiment of this application.

[0040] Figure 2 This is a flowchart of a method for optimizing large model module-level training strategies for heterogeneous clusters, provided in an embodiment of this application.

[0041] Figure 3 This is a schematic flowchart of a memory-aware heuristic search method provided in an embodiment of this application;

[0042] Figure 4 This is a schematic diagram of a distributed training strategy optimization method provided in an embodiment of this application;

[0043] Figure 5This is a structural block diagram of a large model module-level training strategy optimization device for heterogeneous clusters provided in an embodiment of this application.

[0044] In the diagram: 102, processor; 104, memory; 106, transmission device; 108, input / output device; 210, acquisition module; 220, initialization module; 230, optimization module. Detailed Implementation

[0045] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0046] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0047] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the large model module-level training strategy optimization method for heterogeneous clusters in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0048] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the large model module-level training strategy optimization method for heterogeneous clusters in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0049] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0050] This embodiment provides a method for optimizing large model module-level training strategies for heterogeneous clusters. Figure 2 This is a flowchart of the large model module-level training strategy optimization method for heterogeneous clusters in this embodiment, such as... Figure 2 As shown, the process includes the following steps:

[0051] Step S210: Obtain computational information at the module level and operator-level communication information for large model modules in heterogeneous clusters; computational information includes computational time information and module memory information for modules with different data scales under different distributed strategies and on different chips; communication information is the communication latency of communication operators under different data scales.

[0052] Specifically, the processor systematically collects computational and communication information of large models from both module-level and operator-level perspectives, leveraging the hardware characteristics of various types of accelerators in a heterogeneous cluster. Module-level computational information includes the forward and backward computation times of modules with different data scales under different distributed strategies, such as data parallelism, tensor parallelism, and pipelined parallelism, as well as the GPU memory occupied by inputs, gradients, weights, optimizer states, and activation values. Operator-level communication information is obtained by acquiring communication latency for data ranging from 4K to 3.6GB under different operators, with data spans being multiples of 2. Furthermore, to ensure the statistical reliability of subsequent searches, information from the first few rounds that is too large is discarded based on the information obtained from multiple rounds.

[0053] Step S220: Determine the initial distributed training strategy for each pipeline based on the module memory information.

[0054] Specifically, firstly, based on the module's memory information and the accelerator's maximum memory value, the effective range of values ​​for pipeline parallelism under a preset micro-batch size and a preset distributed strategy is calculated, obtaining the initial distributed training strategy within the effective range. Secondly, based on the actual memory capacity of each device in the heterogeneous cluster, the model's memory requirements are non-uniformly allocated. Finally, based on the activation value, different tensor parallelism, context parallelism, expert parallelism strategies, and micro-batch size, pipeline parallelism and data parallelism strategies are determined, thereby effectively reducing the search space of the distributed training strategy and improving the efficiency of strategy configuration and optimization. In other embodiments, a step-by-step optimization strategy combining dynamic programming and greedy algorithms can also be used, without limitation.

[0055] Step S230: Based on computation time information and communication information, determine the computation time of different stages of the pipeline; optimize each initial distributed training strategy according to the computation time and the memory threshold of the device supporting the large model, and obtain the target distributed training strategy.

[0056] Specifically, the system first calculates the standard deviation of the computation time for each pipeline stage and determines whether the standard deviation is within a preset threshold range. If it is, the system proceeds to the next step of judging memory constraints. If the standard deviation is not within the preset threshold range, it indicates an imbalance in computational load between stages. Based on the computation time information of each module, the system identifies the pipeline stage with the heaviest computational load and migrates one level at its boundary to a neighboring stage with a lighter computational load. The standard deviation is recalculated after each migration, iterating until the standard deviation is within the preset threshold range, achieving initial load balancing. Next, based on the module memory information and the current initial load balancing scheme, the system calculates the memory usage of each stage and compares it with the physical memory capacity of the corresponding deployed device. If it exceeds the physical memory capacity, the system reduces the running memory by decreasing the micro-batch size, enabling activation value recalculation, and adjusting parallelism parameters until the running memory of different pipeline stages is less than or equal to the memory threshold for devices supporting large models, thus obtaining the target distributed training strategy. In other embodiments, reinforcement learning-based adaptive optimization algorithms and multi-objective evolutionary search algorithms can also be used, without limitation.

[0057] The above steps address the issue of insufficient resource utilization for large-scale models on heterogeneous clusters. By employing a three-pronged collaborative load balancing mechanism involving computing power, memory, and communication across heterogeneous clusters, the computational load and GPU memory resources are optimized collaboratively across heterogeneous devices, effectively improving device computing utilization and training stability. Simultaneously, the use of a standard deviation-based load balancing metric and a multi-strategy combination of GPU memory optimization methods accurately identifies and eliminates performance bottlenecks in the pipeline, ensuring the feasibility of training tasks within device GPU memory capacity limitations and providing reliable technical support for efficient training of large-scale models on heterogeneous clusters.

[0058] The above steps are explained in detail below:

[0059] In some embodiments, the method for optimizing large model module-level training strategies for heterogeneous clusters further includes the following steps:

[0060] Obtain computational information for one layer of a module in a large model for heterogeneous clusters;

[0061] The computation time information includes the forward computation time and backward computation time of the module; the module memory information includes the memory information of the module's input, gradient, weights, and activation value recalculation.

[0062] Specifically, profiling tools are used to obtain computational information for a representative layer of the large model. Since the current large model is composed of stacked transformer layers with the same structure, only one layer's computational information needs to be obtained. Computation time information is collected by systematically measuring the execution time under different data scales and parallelism configurations by performing the forward and backward propagation processes of the module on actual hardware. Module memory information is obtained by real-time monitoring of memory allocation during module operation, accurately recording the memory usage for input, gradients, weights, and activation value recalculation.

[0063] For example, the modules include: embedding, dense_input_Norm, dense_MLA_down_proj, dense_MLA_up_proj, dense_MLA_core_attn, dense_MLA_linear_proj, dense_attn_dropout, dense_pre_mlp_Norm, dense_MLP, dense_MLP_dropout, MoE_input_Norm, MoE_MLA_down_proj, MoE_MLA_up_proj, MoE_MLA_core_attn, MoE_MLA_linear_proj, MoE_attn_dropout, MoE_pre_mlp_Norm, MoE_MLP, MoE_MLP_dropout, and post_process, etc. Parallelism strategies include: Data Parallelism (DP), Tensor Parallelism (TP), Pipeline Parallelism (PP), and Expert Parallelism (EPP). Parallelism (EP) and Context Parallelism (CP), etc., are used, and there are no restrictions on modules and parallel strategies here. Refer to Table 1, which records the computation time and memory information of the modules when the microprocessor batch size is 1 and tensor parallelism, expert parallelism, and context parallelism are all 1.

[0064] Table 1. Module computation time and memory information

[0065]

[0066] This embodiment establishes a complete and efficient module-level performance data acquisition scheme. This scheme can infer the performance characteristics of the entire model through performance profiling of representative modules, significantly reducing the time cost and computational resource consumption of data acquisition. The obtained refined performance data provides a reliable data foundation for subsequent parallel policy search, enabling the system to accurately predict the performance and resource requirements of each policy during the policy generation stage, thereby significantly improving the efficiency and accuracy of automatic parallel policy optimization.

[0067] In some embodiments, the method for optimizing large model module-level training strategies for heterogeneous clusters further includes the following steps:

[0068] Based on the module memory information and the accelerator's maximum memory value, determine the minimum and maximum memory values ​​corresponding to the preset microprocessor batch and the pipeline under the preset parallel mode.

[0069] The initial distributed training strategy for each pipeline is determined based on the minimum and maximum memory values.

[0070] Specifically, after obtaining the memory information of modules with different data scales under different distributed strategies, the minimum and maximum memory values ​​corresponding to the current pipeline are calculated based on the module memory information and the maximum memory value of the accelerator.

[0071] For example, firstly, based on the obtained module memory information of the large model and the maximum memory value of the accelerator, the minimum and maximum memory values ​​corresponding to the pipeline under the preset microprocessor batch and preset parallel mode are determined. The formulas for calculating the minimum and maximum memory values ​​corresponding to the pipeline are as follows:

[0072]

[0073]

[0074] in, The maximum video memory of the accelerator, It is the size of the video memory occupied by the module's activation value. This refers to the module's video memory size. The specific calculation method is as follows:

[0075]

[0076] in, This is the input video memory information for the module. The module's weight memory information is used as the basis for determining the module's gradient memory information and optimizer state memory information. The ratio of the module's weight memory information to its gradient memory information is 1:2, and the ratio of the module's weight memory information to its accelerator state memory information is 1:4. This refers to the gradient memory information of the module. This refers to the memory information for the optimizer state.

[0077] Secondly, the memory requirements of the model are non-uniformly allocated based on the actual memory capacity of each device in the heterogeneous cluster. Finally, pipeline parallelism and data parallelism strategies are determined based on activation values, different tensor parallelism, context parallelism, expert parallelism strategies, and micro-batch size, thereby initializing the parallelism strategy under the heterogeneous cluster and obtaining the initial distributed training strategy under the heterogeneous cluster.

[0078] This embodiment introduces strict memory constraints at the beginning of the policy search phase, ensuring that all generated initial distributed training policies theoretically will not lead to memory overflow. This significantly reduces the policy search space, avoids wasting computational resources, and thus significantly improves the overall efficiency of automatic parallel policy search.

[0079] In some embodiments, the method for optimizing large model module-level training strategies for heterogeneous clusters further includes the following steps:

[0080] Based on the initial distributed training strategy of the pipeline, the set of modules corresponding to each stage of the pipeline is determined.

[0081] Based on the module set, computation time information, and communication information, the computation time for different stages of the pipeline is determined.

[0082] Specifically, firstly, based on the initial distributed training strategy of the pipeline, a set of modules responsible for computation in each pipeline stage is formed. Secondly, based on the module set of each pipeline stage, the computation time information of the corresponding modules, as well as the operator communication information involved in inter-module communication and communication between different pipeline stages, are selected from the pre-collected computation time and communication information. Finally, based on the selected information, the computation time of different pipeline stages is determined. The formula for determining the computation time of different pipeline stages is:

[0083]

[0084] in, The computation time for each flow stage is P, in total. Let B be the communication time between the i-th stage and the (i+1)-th stage, and let B be the number of microprocessor batches. This represents the total computation time of P pipelines, i.e., the time for training iterations of a large model.

[0085] In particular, After further expansion at the module level, it looks like this:

[0086]

[0087] in, and For the forward computation time and backward computation time of the module, To obtain operator communication information.

[0088] This embodiment achieves refined prediction of computation time for each pipeline stage under any given initial distributed training strategy. By utilizing pre-collected module-level computational information and operator-level communication information, the prediction results accurately reflect the actual execution characteristics of the model on the target heterogeneous cluster. This provides a reliable quantitative input basis for subsequent load balancing optimization and final strategy performance comparison, and is a key technological foundation for the automated generation of efficient distributed training strategies.

[0089] In some embodiments, the method for optimizing large model module-level training strategies for heterogeneous clusters further includes the following steps:

[0090] Determine whether the standard deviation of computation time at different stages of the pipeline is less than a preset threshold;

[0091] If the calculated time standard deviation is greater than or equal to the preset threshold, move one level to the adjacent stage of the pipeline until the calculated time standard deviation is less than the preset threshold.

[0092] Determine whether the running GPU memory at different stages of the pipeline is greater than the GPU memory threshold for supporting large models. If it is greater than the GPU memory threshold, reduce the running GPU memory by decreasing the micro-batch size, enabling activation value recalculation, and adjusting the parallelism parameters until the running GPU memory at different stages of the pipeline is less than or equal to the GPU memory threshold for supporting large models, thus obtaining the target distributed training strategy.

[0093] Specifically, such as Figure 4 As shown, the load balancing optimization process employs a dual-loop iterative mechanism: First, in the outer loop, the system calculates the standard deviation σ(T) of the computation time for each pipeline stage and compares it with a preset threshold δ. When σ(T) ≥ δ, it indicates a significant imbalance in computational load between stages, and the system initiates a hierarchical migration process: based on the computation time information of each module, the system identifies the pipeline stage with the heaviest computational load and migrates one or more computational modules at its boundary to adjacent stages with lighter computational loads. σ(T) is recalculated after each migration, and this process iterates until σ(T) < δ, achieving initial load balancing.

[0094] Next, the inner loop is entered to verify and optimize memory usage for the currently preliminarily balanced pipeline partitioning scheme. Based on module memory information and the current parallel strategy configuration, the system accurately calculates the peak memory usage during training at each stage and compares it with the physical memory capacity of the corresponding deployed devices. When the estimated memory usage at any stage exceeds the device's memory threshold, the system triggers a series of memory optimization operations according to a preset priority order:

[0095] The first step is to try reducing the micro-batch size to directly reduce the memory usage of intermediate results such as activation values ​​and gradients;

[0096] The second step is to selectively enable recalculation of the activation value during the stage of memory overrun, in order to trade computation time for memory space.

[0097] Third, if the above measures are still insufficient, the tensor parallelism or expert parallelism of the model involved in this stage is adjusted upwards to distribute the memory pressure of a single device by increasing the number of parallel devices.

[0098] After each optimization operation, memory usage is reassessed, and the outer loop may need to be re-entered for load balancing adjustments due to policy changes. This inner and outer loop iterates until the load balancing metric is met, ultimately outputting a load-balanced and memory-safe target distributed training strategy. The load balancing metric can be implemented in several ways. One method uses a mathematical formula to determine whether σ(T) < δ and memory usage is below the device threshold at all stages. The load balancing metric formula is as follows:

[0099]

[0100] Where σ(T) is the standard deviation of the calculation time for each pipeline stage, and δ is a preset threshold. The amount of video memory pre-allocated for each stage of the pipeline. This is the video memory threshold for physical devices.

[0101] This embodiment achieves collaborative optimization of computational load and memory resources across heterogeneous devices. Through dynamic hierarchical migration based on standard deviation and a multi-strategy memory optimization mechanism, it effectively solves the load imbalance problem caused by heterogeneous device computing power and model structural complexity, significantly improving the overall training throughput and stability of the heterogeneous cluster.

[0102] In some embodiments, the method for optimizing large model module-level training strategies for heterogeneous clusters further includes the following steps:

[0103] After obtaining the target distributed training strategy, the distributed training strategy with the shortest iteration time is selected as the best strategy based on the preset performance function.

[0104] Specifically, such as Figure 3 As shown, there are multiple ways to obtain the optimal policy based on the target distributed training strategy. One method is to calculate the iteration time of the target distributed training strategy using a mathematical formula and select the strategy with the shortest iteration time as the optimal policy. If the formula implementation method is adopted, the performance function is calculated as follows:

[0105]

[0106] Where S(M,C) represents the target distributed training strategy set (M is the model module configuration, C is the heterogeneous cluster hardware parameters), T is the total time of a single training iteration, pp is the number of stages in the pipeline parallelism, Ti is the total computation and communication time of the i-th stage, B is the number of micro-batches, and C... DP For data-parallel gradient aggregation communication time.

[0107] In other embodiments, methods such as multi-objective weighted evaluation methods and reinforcement learning methods can be used to select the best policy, and there is no limitation thereto.

[0108] This embodiment quantitatively selects the optimal strategy from multiple theoretically feasible distributed training strategies. It fully considers the real costs of computation, communication, and memory access in heterogeneous clusters, avoiding the subjectivity and uncertainty of human experience-based decisions. This ensures that distributed training tasks can be executed with maximum efficiency, significantly improving the overall resource utilization and training throughput of heterogeneous clusters.

[0109] In some embodiments, the large model module for heterogeneous clusters also includes a load balancing module for hybrid expert models; the load balancing module for hybrid expert models is used to optimize the balanced distribution of lexical units during the training of the large model.

[0110] Specifically, the load balancing module of the hybrid expert model is based on the grouped hybrid expert model technology. It combines the ternary collaborative constraints of heterogeneous cluster computing power, memory, and communication bandwidth to construct a closed-loop adaptive mechanism of "cluster state awareness - routing strategy optimization - dynamic load balancing". First, the system monitors the performance status and network link status of each computing device in the heterogeneous cluster in real time, providing a basis for dynamic balancing decisions. Second, through its built-in gating network and auxiliary loss function, it automatically learns and optimizes the mapping relationship between semantic features and expert groups during training. Thus, during inference, based on the real-time system state and the preset load balancing target, it intelligently distributes input tokens to the most suitable computing nodes, ultimately achieving balanced token distribution during training.

[0111] This embodiment effectively fills the technical gap in current technologies, which only support general dense models and lack dedicated optimization for hybrid expert models. It enables heterogeneous clusters to efficiently support the training of sparse expert hybrid models, expanding the coverage of large model types. Simultaneously, by leveraging dynamic lexical distribution driven by grouped hybrid expert model technology, it effectively solves the problem of uneven lexical distribution among experts during hybrid expert model training, leading to unbalanced device load. This provides key technical support for the efficiency, stability, and practicality of large model training in heterogeneous clusters.

[0112] In some embodiments, the method for optimizing large model module-level training strategies for heterogeneous clusters further includes the following steps:

[0113] Acquire bandwidth difference information between and within different nodes in a heterogeneous cluster;

[0114] Based on bandwidth difference information, the communication links of heterogeneous clusters are configured in a hierarchical manner for optimization.

[0115] Specifically, firstly, cluster hardware topology scanning and performance profiling tools are used to collect full-link bandwidth data within heterogeneous clusters. The data information includes high-speed interconnect bandwidth between computing units within a node, link bandwidth between nodes within a homogeneous device group, and cross-device connection bandwidth across heterogeneous groups. A bandwidth difference information database containing "link type - bandwidth limit - transmission delay - supported protocols" is established to clarify the resource benchmarks for different communication scenarios. Subsequently, based on the communication requirements of different parallel strategies in the distributed training of large models, a hierarchical link binding is implemented: For intra-node scenarios, since tensor parallelism requires high-frequency, high-throughput data interaction, it is bound to a high-speed bandwidth link within the node and a collection communication library is used. The collection communication library can be NCCL, Gloo, or OpenMPI, etc., without restriction. For inter-node scenarios within homogeneous groups, data-parallel gradient aggregation requires large-scale data summarization, so a 400Gb / s communication link is used for data-parallel gradient aggregation. The communication link can be InfiniBand, Ethernet, or RoCEv2, etc., without restriction. For cross-heterogeneous group scenarios, pipelined parallel activation value transmission has higher requirements for bandwidth stability, so it is carried by a 400Gb / s cross-device dedicated link to avoid occupying high-priority communication resources within the homogeneous group.

[0116] This embodiment demonstrates how layered optimized communication links achieve precise matching between communication needs and bandwidth resources. On one hand, it restricts high-bandwidth communication to nodes with sufficient bandwidth or homogeneous groups. On the other hand, through customized configuration of protocols and links, it effectively reduces cross-cluster traffic, thereby significantly optimizing system resource utilization and reducing communication overhead. This provides key communication optimization capabilities for domestically developed heterogeneous clusters to support large-scale, large-model training.

[0117] This embodiment provides a large model module-level training strategy optimization device for heterogeneous clusters. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below can refer to combinations of software and / or hardware that perform a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0118] Figure 5 This is a structural block diagram of the large model module-level training strategy optimization device for heterogeneous clusters in this embodiment, as shown below. Figure 5 As shown, the device includes: an acquisition module 210, an initialization module 220, and an optimization module 230;

[0119] The acquisition module 210 is used to acquire computational information at the module level and operator-level communication information for large models targeting heterogeneous clusters. The computational information includes computational time information of modules under different data scales, under different distributed strategies, and on different chips, as well as module memory information. The communication information is the communication latency of communication operators under different data scales.

[0120] Initialization module 220 is used to determine the initial distributed training strategy for each pipeline based on the module's video memory information;

[0121] The optimization module 230 is used to determine the computation time of different stages of the pipeline based on computation time information and communication information; and to optimize each initial distributed training strategy according to the computation time and the memory threshold of the large model device to obtain the target distributed training strategy.

[0122] The aforementioned device solves the problem of resource utilization needing improvement when dealing with large models of heterogeneous clusters in related technologies, and achieves a coordinated balance of computing load, memory usage and communication overhead among heterogeneous devices, resulting in a significant improvement in resource utilization.

[0123] In some embodiments, the acquisition module 210 is also used to acquire computational information of one layer module of a large model for heterogeneous clusters;

[0124] The computation time information includes the forward computation time and backward computation time of the module; the module memory information includes the memory information of the module's input, gradient, weights, and activation value recalculation.

[0125] In some embodiments, the initialization module 220 is further configured to determine the minimum and maximum memory values ​​corresponding to the pipeline under the preset microprocessor batch and the preset parallel mode based on the module memory information and the maximum memory value of the accelerator.

[0126] The initial distributed training strategy for each pipeline is determined based on the minimum and maximum memory values.

[0127] In some embodiments, the optimization module 230 is also used to determine the set of modules corresponding to each stage of the pipeline based on the initial distributed training strategy of the pipeline.

[0128] Based on the module set, computation time information, and communication information, the computation time for different stages of the pipeline is determined.

[0129] In some embodiments, the optimization module 230 is also used to determine whether the standard deviation of the computation time at different stages of the pipeline is less than a preset threshold.

[0130] If the calculated time standard deviation is greater than or equal to the preset threshold, move one level to the adjacent stage of the pipeline until the calculated time standard deviation is less than the preset threshold.

[0131] Determine whether the running GPU memory at different stages of the pipeline is greater than the GPU memory threshold for supporting large models. If it is greater than the GPU memory threshold, reduce the running GPU memory by decreasing the micro-batch size, enabling activation value recalculation, and adjusting the parallelism parameters until the running GPU memory at different stages of the pipeline is less than or equal to the GPU memory threshold for supporting large models, thus obtaining the target distributed training strategy.

[0132] In some embodiments, the optimization module 230 is further configured to, after obtaining the target distributed training strategy, select the distributed training strategy with the shortest iteration time from the target distributed training strategies as the best strategy based on a preset performance function.

[0133] In some embodiments, a load balancing module for a hybrid expert model is used to optimize the balanced distribution of tokens during the training of a large model.

[0134] In some embodiments, the optimization module 230 is also used to obtain bandwidth difference information between different nodes and within nodes in a heterogeneous cluster.

[0135] Based on bandwidth difference information, the communication links of heterogeneous clusters are configured in a hierarchical manner for optimization.

[0136] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0137] This embodiment provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0138] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0139] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0140] S1, acquires computational information at the module level and operator-level communication information for large model modules in heterogeneous clusters; computational information includes computation time information and module memory information for modules with different data scales under different distributed strategies and on different chips; communication information is the communication latency of communication operators under different data scales.

[0141] S2, based on the module's memory information, determines the initial distributed training strategy for each pipeline;

[0142] S3, based on computation time information and communication information, determines the computation time of different stages of the pipeline; according to the computation time and the memory threshold of the large model device, optimizes each initial distributed training strategy to obtain the target distributed training strategy.

[0143] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0144] Furthermore, in conjunction with the large model module-level training strategy optimization method for heterogeneous clusters provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the large model module-level training strategy optimization methods for heterogeneous clusters in the above embodiments.

[0145] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0146] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0147] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0148] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for optimizing training strategies at the module level of large models for heterogeneous clusters, characterized in that, The method includes: Acquire computational information at the module level and operator level at the operator level for large model clusters; the computational information includes computation time information and module memory information for modules with different data scales under different distributed strategies and on different chips; the communication information is the communication latency of communication operators under different data scales. Based on the memory information of the module, determine the initial distributed training strategy for each pipeline. Based on the computation time information and the communication information, the computation time of different stages of the pipeline is determined; Based on the computation time and the memory threshold of the device supporting the large model, each of the initial distributed training strategies is optimized to obtain the target distributed training strategy.

2. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The acquisition of computational information at the module level and operator-level communication information for large model clusters in heterogeneous clusters includes: Obtain computational information of one layer of the module in the large model for heterogeneous clusters; The computation time information includes the forward computation time and backward computation time of the module; the module memory information includes the memory information of the module's input, gradient, weights, and activation value recalculation.

3. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The step of determining the initial distributed training strategy for each pipeline based on the module's memory information includes: Based on the module memory information and the accelerator's maximum memory value, determine the minimum and maximum memory values ​​of the pipeline corresponding to the preset microprocessor batch and the preset parallel mode. Based on the minimum and maximum memory values, an initial distributed training strategy is determined for each pipeline.

4. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, Determining the computation time of different stages of the pipeline based on the computation time information and the communication information includes: Based on the initial distributed training strategy of the pipeline, the module set corresponding to each stage of the pipeline is determined. The computation time for different stages of the pipeline is determined based on the module set, the computation time information, and the communication information.

5. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The step of optimizing each initial distributed training strategy based on the computation time and the memory threshold of the device supporting the large model to obtain the target distributed training strategy includes: Determine whether the standard deviation of the computation time at different stages of the pipeline is less than a preset threshold; If the standard deviation of the computation time is greater than or equal to the preset threshold, then move one level to the adjacent stage of the pipeline until the standard deviation of the computation time is less than the preset threshold. Determine whether the running GPU memory at different stages of the pipeline is greater than the GPU memory threshold of the device carrying the large model. If it is greater than the GPU memory threshold, reduce the running GPU memory by decreasing the micro-batch size, enabling activation value recalculation, and adjusting the parallelism parameter until the running GPU memory at different stages of the pipeline is less than or equal to the GPU memory threshold of the device carrying the large model, and obtain the target distributed training strategy.

6. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The method further includes: After obtaining the target distributed training strategy, the distributed training strategy with the shortest iteration time is selected as the optimal strategy based on a preset performance function.

7. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The module also includes a load balancing module based on a hybrid expert model; The load balancing module of the hybrid expert model is used to optimize the balanced distribution of words during the training of the large model.

8. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The method further includes: Obtain bandwidth difference information between and within different nodes in the heterogeneous cluster; Based on the bandwidth difference information, the communication links of the heterogeneous cluster are configured in a hierarchical manner for optimization.

9. A device for optimizing training strategies at the module level of large models for heterogeneous clusters, characterized in that, The device includes: an acquisition module, an initialization module, and an optimization module; The acquisition module is used to acquire computational information at the module level and operator-level communication information for large model clusters in heterogeneous clusters; the computational information includes computational time information and module memory information of modules under different data scales, different distributed strategies, and different chips; the communication information is the communication latency of communication operators under different data scales. The initialization module is used to determine the initial distributed training strategy for each pipeline based on the module's memory information. The optimization module is used to determine the computation time of different stages of the pipeline based on the computation time information and the communication information; and to optimize each initial distributed training strategy according to the computation time and the memory threshold of the large model device to obtain the target distributed training strategy.

10. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the method for optimizing large model module-level training strategies for heterogeneous clusters as described in any one of claims 1 to 8.

Images