Model module level training strategy optimization method and device for heterogeneous cluster and computer equipment

Abstract

The application relates to a large model module level training strategy optimization method and device for a heterogeneous cluster and computer equipment, wherein the method comprises the following steps: acquiring computing information of a large model module level and communication information of an operator level for the heterogeneous cluster; the computing information comprises computing time information and module memory information of a module under different distributed strategies and different chips under different data scales; the communication information is communication delay of a communication operator under different data scales; based on the module memory information, an initial distributed training strategy of each flow pipeline is determined; based on the computing time information and the communication information, computing time of different stages of the flow pipeline is determined; according to the computing time and a memory threshold of a large model equipment, each initial distributed training strategy is optimized to obtain a target distributed training strategy. Through the application, the problem of low resource utilization rate of a large model for a heterogeneous cluster is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of large models, in particular to a large model module-level training strategy optimization method and device for a heterogeneous cluster and a computer device. BACKGROUND

[0002] In recent years, large models have rapidly evolved to the scale of hundreds of billions or trillions of parameters, and their training process requires extreme computing power. The current international computing power blockade exacerbates the computing power gap, and domestic AI accelerators have become the core support for building large-scale computing clusters and filling the computing power gap. Under this background, a general large model distributed training technology for a heterogeneous cluster will directly affect the resource utilization of the heterogeneous cluster.

[0003] Current mainstream automated distributed training frameworks include Alpa and Aceso, etc., which are mainly designed for homogeneous clusters, and their architecture core assumes that devices have uniform computing power and communication capabilities. The defects of these frameworks are that when deployed on a heterogeneous cluster composed of different domestic AI accelerators, the computing power, memory bandwidth, and communication protocols of different accelerators in the domestic heterogeneous cluster differ significantly, which cannot effectively coordinate the computing power differences, memory, and communication bottlenecks between devices, resulting in low overall cluster resource utilization.

[0004] There is currently no effective solution to the problem of improving resource utilization for large models facing a heterogeneous cluster in related technologies. SUMMARY

[0005] A large model module-level training strategy optimization method, device, and computer device for a heterogeneous cluster are provided in this embodiment to solve the problem of improving resource utilization for large models facing a heterogeneous cluster in related technologies.

[0006] In a first aspect, a large model module-level training strategy optimization method for a heterogeneous cluster is provided in this embodiment, which includes:

[0007] Obtain computing information at the module level and communication information at the operator level for a large model facing a heterogeneous cluster; the computing information includes computing time information and module memory information of modules at different data scales, different distributed strategies, and different chips; and the communication information is the communication delay of communication operators at different data scales;

[0008] Determine an initial distributed training strategy for each pipeline based on the module memory information;

[0009] Determine the computing time of different stages of the pipeline based on the computing time information and the communication information;

[0010] According to the calculation time and a memory threshold of a device carrying the large model, each of the initial distributed training strategies is optimized to obtain a target distributed training strategy.

[0011] In some embodiments, the obtaining of the calculation information at the module level and the communication information at the operator level of the large model facing the heterogeneous cluster comprises:

[0012] The calculation information of one of the modules of the large model facing the heterogeneous cluster is obtained.

[0013] The calculation time information comprises forward calculation time and backward calculation time of the module, and the module memory information comprises memory information of input, gradient, weight and activation value recalculation of the module.

[0014] In some embodiments, the determination of the initial distributed training strategy of each pipeline based on the module memory information comprises:

[0015] According to the module memory information and a maximum memory value of an accelerator, a preset micro-processing batch and a preset parallel mode are determined, and a minimum memory value and a maximum memory value corresponding to the pipeline are determined.

[0016] Based on the minimum memory value and the maximum memory value, the initial distributed training strategy of each pipeline is determined.

[0017] In some embodiments, the determination of the calculation time of different stages of the pipeline based on the calculation time information and the communication information comprises:

[0018] Based on the initial distributed training strategy of the pipeline, a module set corresponding to each stage of the pipeline is determined.

[0019] According to the module set, the calculation time information and the communication information, the calculation time of different stages of the pipeline is determined.

[0020] In some embodiments, the optimization of each of the initial distributed training strategies according to the calculation time and a memory threshold of a device carrying the large model comprises:

[0021] It is determined whether a standard deviation of the calculation time of different stages of the pipeline is less than a preset threshold value.

[0022] If the standard deviation of the calculation time is greater than or equal to the preset threshold value, a level is moved to an adjacent stage of the pipeline until the standard deviation of the calculation time is less than the preset threshold value.

[0023] determining whether the running memory of different stages of the pipeline is greater than a memory threshold of the large model device, and if the running memory is greater than the memory threshold, reducing the running memory by reducing the micro-batch size, enabling activation value recalculation, and adjusting the parallelism parameter until the running memory of different stages of the pipeline is less than or equal to the memory threshold of the large model device, to obtain the target distributed training strategy.

[0024] In some embodiments, the method further comprises:

[0025] After obtaining the target distributed training strategy, selecting a distributed training strategy with the shortest iteration time from the target distributed training strategies as an optimal strategy based on a preset performance function.

[0026] In some embodiments, the module further comprises a load balancing module for mixed expert models.

[0027] The load balancing module for mixed expert models is configured to optimize the balanced distribution of tokens in the large model training process.

[0028] In some embodiments, the method further comprises:

[0029] Obtaining bandwidth difference information between different nodes in the heterogeneous cluster and within a node;

[0030] Based on the bandwidth difference information, the communication links of the heterogeneous cluster are hierarchically optimized and configured.

[0031] In a second aspect, the present embodiment provides a large model module-level training strategy optimization device for a heterogeneous cluster, comprising an obtaining module, an initialization module, and an optimization module.

[0032] The obtaining module is configured to obtain computing information at the module level and communication information at the operator level for a large model of a heterogeneous cluster. The computing information includes computing time information of a module at different data scales, under different distributed strategies, and on different chips, and module memory information. The communication information is the communication delay of a communication operator at different data scales.

[0033] The initialization module is configured to determine an initial distributed training strategy for each pipeline based on the module memory information.

[0034] The optimization module is configured to determine the computing time of different stages of the pipeline based on the computing time information and the communication information, and to optimize each initial distributed training strategy based on the computing time and a memory threshold of a large model device, to obtain a target distributed training strategy.

[0035] In a third aspect, a computer device is provided in the present embodiment, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the model module level training strategy optimization method for heterogeneous clusters in the first aspect when executing the computer program.

[0036] Compared with the related art, the model module level training strategy optimization method, device and computer device for heterogeneous clusters provided in the present embodiment, by obtaining the computing information of the model module level and the communication information of the operator level for heterogeneous clusters; the computing information includes the computing time information and the module memory information of the module under different distributed strategies and different chips under different data scales; the communication information is the communication delay of the communication operator under different data scales; based on the module memory information, the initial distributed training strategy of each pipeline is determined; based on the computing time information and the communication information, the computing time of different stages of the pipeline is determined; according to the computing time and the memory threshold of the device carrying the large model, each initial distributed training strategy is optimized to obtain the target distributed training strategy. The utilization efficiency of the large model resources of the heterogeneous cluster is higher, the model and the hardware are decoupled through the module level unified abstraction layer, the computing load, the memory occupation and the communication overhead between the heterogeneous devices are balanced and coordinated, and the resource utilization rate is significantly improved.

[0037] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0038] The drawings described herein are intended to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 is a hardware structure block diagram of a terminal device of the model module level training strategy optimization method for heterogeneous clusters provided by an embodiment of the present application;

[0040] Figure 2 is a flowchart of the model module level training strategy optimization method for heterogeneous clusters provided by an embodiment of the present application;

[0041] Figure 3 is a flowchart of the heuristic search method based on memory awareness provided by an embodiment of the present application;

[0042] Figure 4 is a flowchart of the distributed training strategy optimization method provided by an embodiment of the present application;

[0043] Figure 5is a structural block diagram of a device for optimizing a large model module-level training strategy for a heterogeneous cluster provided by an embodiment of the present application.

[0044] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 210, acquisition module; 220, initialization module; 230, optimization module. DETAILED DESCRIPTION

[0045] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and explained below in conjunction with the drawings and embodiments.

[0046] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the general meaning understood by a person with ordinary skill in the art to which the present application belongs. In the present application, "one", "a", "an", "the", "these" and similar words do not represent a quantitative limitation, and they can be singular or plural. In the present application, the terms "include", "contain", "have" and any variants thereof are intended to cover non-exclusive inclusion; for example, a process, method and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. In the present application, the terms "connected", "connected", "coupled" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. In the present application, "multiple" means two or more. The association between the associated objects is described by the term "and / or", which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. In general, the character " / " represents an "or" relationship between the objects before and after it. In the present application, the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0047] The method embodiments provided in the present embodiment can be executed in a terminal, a computer or a similar computing device. For example, the method embodiments are executed on a terminal, Figure 1 is a hardware structure block diagram of a terminal for a large model module-level training strategy optimization method for a heterogeneous cluster of the present embodiment. As shown in Figure 1 the terminal can include one or more (CPU) central processing units, memories, buses, input and output interfaces, and the like. Figure 1Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0048] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the large model module-level training strategy optimization method for heterogeneous clusters in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0049] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0050] This embodiment provides a method for optimizing large model module-level training strategies for heterogeneous clusters. Figure 2 This is a flowchart of the large model module-level training strategy optimization method for heterogeneous clusters in this embodiment, such as... Figure 2 As shown, the process includes the following steps:

[0051] In step S210, the computing information of the large model module level and the communication information of the operator level facing the heterogeneous cluster are obtained; the computing information includes the computing time information and the module memory information of the module under different data scales, different distributed strategies and different chips; and the communication information is the communication delay of the communication operator under different data scales.

[0052] Specifically, the processor starts from the module level and the operator level of the large model, and combines the hardware characteristics of the multiple types of accelerators in the heterogeneous cluster to systematically collect the computing information and the communication information of the large model. The module level computing information includes the forward computing time and the backward computing time of the module under different data scales, different distributed strategies such as data parallelism, tensor parallelism and pipeline parallelism, and the memory occupied by the input, gradient, weight, optimizer state and activation value. The operator level communication information is obtained by acquiring the communication delay of data ranging from 4K to 3.6GB under different operators, wherein the data span is a multiple of 2. At the same time, in order to ensure the statistical reliability of the subsequent search, the information of the previous rounds with excessive data is removed on the basis of obtaining multiple rounds of information.

[0053] In step S220, the initial distributed training strategy of each pipeline is determined based on the module memory information.

[0054] Specifically, first, the effective value range of the pipeline parallelism under the preset micro-batch size and the preset distributed strategy is calculated according to the module memory information and the maximum memory of the accelerator, and the initial distributed training strategy in the effective value range is obtained. Second, the memory demand of the model is non-uniformly distributed according to the actual memory capacity of each device in the heterogeneous cluster. Finally, the pipeline parallelism and the data parallelism strategy are determined according to the activation value, different tensor parallelism, context parallelism, expert parallelism strategy and micro-batch size, thereby effectively pruning the search space of the distributed training strategy and improving the efficiency of the strategy configuration and optimization. In other embodiments, a step-by-step optimization strategy combining dynamic programming and greedy algorithm can also be used, which is not limited.

[0055] In step S230, the computing time of different stages of the pipeline is determined based on the computing time information and the communication information; and each initial distributed training strategy is optimized according to the computing time and the memory threshold of the device carrying the large model to obtain a target distributed training strategy.

[0056] Specifically, the system first calculates the standard deviation of the computation time of each pipeline stage, determines whether the standard deviation is within a preset threshold range, and if it is within the preset threshold range, proceeds to the next step of determining the memory constraint condition; if the standard deviation is not within the preset threshold range, it indicates that there is an imbalance in the computation load between stages, and the pipeline stage with the heaviest computation load needs to be identified based on the computation time information of each module, and one level at the boundary of the pipeline stage is migrated to the adjacent stage with lighter computation load. The standard deviation is recalculated after each migration, and the iteration continues until the standard deviation is within the preset threshold range, achieving preliminary balancing of the computation load. Secondly, the system calculates the memory usage of each stage according to the module memory information and the current preliminary balanced division scheme, and compares it with the physical memory capacity of the corresponding deployment device. If it is greater than the physical memory capacity, the running memory is reduced by reducing the micro-batch size, enabling activation value recalculation, and adjusting the parallelism parameter, until the running memory of different stages of the pipeline is less than or equal to the memory threshold of the device carrying the large model, and the target distributed training strategy is obtained. In other embodiments, adaptive optimization algorithms based on reinforcement learning and multi-objective evolutionary search algorithms can also be used, which are not limited.

[0057] Through the above steps, the problem of improving resource utilization for large models facing heterogeneous clusters is solved. Through the three-way collaborative load balancing mechanism of computing power, memory, and communication of heterogeneous clusters, the collaborative optimization of computation load and memory resources among heterogeneous devices is achieved, effectively improving the device computation utilization and training stability. At the same time, the load balancing measurement method based on standard deviation and the memory optimization scheme with multiple strategy combinations can accurately identify and eliminate performance bottlenecks in the pipeline, ensuring the feasibility of the training task under the limitation of device memory capacity, providing reliable technical support for efficient training of large-scale models on heterogeneous clusters.

[0058] The above steps are described in detail as follows:

[0059] In some embodiments, the large model module-level training strategy optimization method for heterogeneous clusters further includes the following steps:

[0060] Obtain the computation information of one layer of modules of the large model facing the heterogeneous cluster;

[0061] The computation time information includes the forward computation time and the backward computation time of the module; the module memory information includes the memory information of the input, gradient, weight, and activation value recalculation of the module.

[0062] Specifically, the computing information of a representative layer module in the large model is obtained by the Profiling tool. The current large model is stacked by transformer layers with the same structure, so it is only necessary to obtain the computing information of one layer module. The collection of the computing time information is performed by executing the forward propagation and the backward propagation of the module on the actual hardware, and systematically measuring the execution time under different data scales and different parallel strategy configurations. The module memory information is obtained by monitoring the memory allocation during the module running process in real time, and accurately recording the memory size occupied by the input, gradient, weight and activation value recalculation.

[0063] For example, the module includes embedding, dense_input_Norm, dense_MLA_down_proj, dense_MLA_up_proj, dense_MLA_core_attn, dense_MLA_linear_proj, dense_attn_dropout, dense_pre_mlp_Norm, dense_MLP, dense_MLP_dropout, MoE_input_Norm, MoE_MLA_down_proj, MoE_MLA_up_proj, MoE_MLA_core_attn, MoE_MLA_linear_proj, MoE_attn_dropout, MoE_pre_mlp_Norm, MoE_MLP, MoE_MLP_dropout and post_process, and the parallel strategy includes data parallelism (DP), tensor parallelism (TP), pipeline parallelism (PP), expert parallelism (EP) and context parallelism (CP), and the module and the parallel strategy are not limited herein. Referring to Table 1, the computing time information and the memory information of the module are recorded under the condition that the micro-processing batch is 1, the tensor parallelism, the expert parallelism and the context parallelism are all 1.

[0064] Table 1. Computing time information and memory information of the module

[0065]

[0066] Through this embodiment, a complete and efficient module-level performance data collection scheme is established, which can infer the performance characteristics of the whole model through performance profiling of representative modules, greatly reducing the time cost and computational resource consumption of data collection. The obtained refined performance data provides a reliable data foundation for subsequent parallel strategy search, enabling the system to accurately estimate the performance and resource requirements of each strategy in the strategy generation stage, thereby significantly improving the efficiency and accuracy of automatic parallel strategy optimization.

[0067] In some embodiments, the module-level training strategy optimization method for large models facing heterogeneous clusters further comprises the following steps:

[0068] According to the module memory information and the maximum memory of the accelerator, the minimum memory and the maximum memory corresponding to the pipeline under the preset micro-processing batch and the preset parallel mode are determined;

[0069] Based on the minimum memory and the maximum memory, the initial distributed training strategy of each pipeline is determined.

[0070] Specifically, after obtaining the module memory information of the module under different distributed strategies under different data scales, the minimum memory and the maximum memory corresponding to the current pipeline are calculated according to the module memory information and the maximum memory of the accelerator.

[0071] For example, first, the minimum memory and the maximum memory corresponding to the pipeline under the preset micro-processing batch and the preset parallel mode are determined according to the obtained module memory information of the large model and the maximum memory of the accelerator. The calculation formula of the minimum memory and the maximum memory corresponding to the pipeline is:

[0072]

[0073]

[0074] wherein, is the maximum memory of the accelerator, is the size of the occupied memory of the activation value of the module, is the memory size of the module, The specific calculation method is as follows:

[0075]

[0076] wherein, is the input memory information of the module, The module's weight memory information is used as the basis for determining the module's gradient memory information and optimizer state memory information. The ratio of the module's weight memory information to its gradient memory information is 1:2, and the ratio of the module's weight memory information to its accelerator state memory information is 1:4. This refers to the gradient memory information of the module. This refers to the memory information for the optimizer state.

[0077] Secondly, the memory requirements of the model are non-uniformly allocated based on the actual memory capacity of each device in the heterogeneous cluster. Finally, pipeline parallelism and data parallelism strategies are determined based on activation values, different tensor parallelism, context parallelism, expert parallelism strategies, and micro-batch size, thereby initializing the parallelism strategy under the heterogeneous cluster and obtaining the initial distributed training strategy under the heterogeneous cluster.

[0078] This embodiment introduces strict memory constraints at the beginning of the policy search phase, ensuring that all generated initial distributed training policies theoretically will not lead to memory overflow. This significantly reduces the policy search space, avoids wasting computational resources, and thus significantly improves the overall efficiency of automatic parallel policy search.

[0079] In some embodiments, the method for optimizing large model module-level training strategies for heterogeneous clusters further includes the following steps:

[0080] Based on the initial distributed training strategy of the pipeline, the set of modules corresponding to each stage of the pipeline is determined.

[0081] Based on the module set, computation time information, and communication information, the computation time for different stages of the pipeline is determined.

[0082] Specifically, firstly, based on the initial distributed training strategy of the pipeline, a set of modules responsible for computation in each pipeline stage is formed. Secondly, based on the module set of each pipeline stage, the computation time information of the corresponding modules, as well as the operator communication information involved in inter-module communication and communication between different pipeline stages, are selected from the pre-collected computation time and communication information. Finally, based on the selected information, the computation time of different pipeline stages is determined. The formula for determining the computation time of different pipeline stages is:

[0083]

[0084] in, The computation time for each flow stage is P, in total. Let B be the communication time between the i-th stage and the (i+1)-th stage, and let B be the number of microprocessor batches. denotes the total computation time of P pipelines, i.e., the time of a large model training iteration.

[0085] In particular, After further unfolding at the module level, we have:

[0086]

[0087] where, and denote the forward and backward computation time of a module, denotes the obtained operator communication information.

[0088] Through the embodiment, the fine estimation of the computation time of each pipeline stage under any given initial distributed training strategy is realized. By using the pre-collected module-level computation information and operator-level communication information, the estimation result can accurately reflect the actual execution characteristics of the model on the target heterogeneous cluster, providing reliable quantitative input basis for subsequent load balancing optimization and final strategy performance comparison, which is a key technical basis for automatically generating an efficient distributed training strategy.

[0089] In some embodiments thereof, the large model module-level training strategy optimization method for a heterogeneous cluster further includes the following steps:

[0090] determining whether the standard deviation of the computation time of different stages of the pipeline is less than a preset threshold value;

[0091] If the standard deviation of the computation time is greater than or equal to the preset threshold value, moving one level to the adjacent stage of the pipeline until the standard deviation of the computation time is less than the preset threshold value;

[0092] determining whether the running memory of different stages of the pipeline is greater than the memory threshold value of the device carrying the large model, and if it is greater than the memory threshold value, reducing the running memory by reducing the mini-batch size, enabling activation value recalculation, and adjusting the parallelism parameter until the running memory of different stages of the pipeline is less than or equal to the memory threshold value of the device carrying the large model, to obtain the target distributed training strategy.

[0093] Specifically, as shown in Figure 4 The load balancing optimization process adopts a double-loop iteration mechanism: first, in the outer loop, the system first calculates the standard deviation σ(T) of the computation time of each pipeline stage, and compares it with the preset threshold value δ. When σ(T)≥δ, it indicates that there is significant computation load imbalance between stages, and the system will start the level migration process: based on the computation time information of each module, the pipeline stage with the heaviest computation load is identified, and one or more computation modules at the boundary of the stage are migrated to the adjacent stage with lighter computation load. The σ(T) is recalculated after each migration, and the iteration is continued until σ(T)<δ, to achieve the preliminary balancing of the computation load.

[0094] Secondly, enter the inner loop, and perform memory occupation verification and optimization for the current preliminary balanced pipeline division scheme. The system accurately calculates the peak memory occupation of each stage during training according to the module memory information and the current parallel strategy configuration, and compares it with the physical memory capacity of the corresponding deployment device. When the estimated memory occupation of any stage exceeds the device memory threshold, the system triggers a series of memory optimization operations in the preset priority order:

[0095] First, try to reduce the mini-batch size to directly reduce the memory occupation of intermediate results such as activation values and gradients;

[0096] Secondly, if the mini-batch size is adjusted and still does not meet the requirements, then selectively enable activation value recalculation in the memory over-limit stage to exchange memory space for calculation time;

[0097] Thirdly, if the above measures are still insufficient, adjust the tensor parallelism or expert parallelism of the model involved in the stage upwards, and increase the number of parallel devices to share the memory pressure of a single device.

[0098] After each optimization operation is performed, the memory occupation is re-evaluated, and the strategy may need to be re-entered into the outer loop for load balancing adjustment. Through the iteration of the inner and outer loops, the final output is a target distributed training strategy that is load balanced and memory safe. There are various ways to measure the load balancing, one of which is to determine whether σ(T) < δ and the memory occupation of all stages is lower than the device threshold at the same time through a mathematical formula. The formula for measuring the load balancing is:

[0099]

[0100] Where σ(T) is the standard deviation of the calculation time of each pipeline stage, δ is the preset threshold, is the pre-allocated memory size of the pipeline stage, is the memory threshold of the physical device.

[0101] Through this embodiment, the calculation load and memory resources are cooperatively optimized among heterogeneous devices. Through the dynamic hierarchical migration of the standard deviation metric and the memory optimization mechanism of multiple strategy combinations, the load imbalance problem caused by the heterogeneity of device computing power and the complexity of model structure is effectively solved, and the overall training throughput and stability of the heterogeneous cluster are significantly improved.

[0102] In some embodiments, the large model module-level training strategy optimization method for heterogeneous clusters further includes the following steps:

[0103] After obtaining the target distributed training strategy, based on the preset performance function, the distributed training strategy with the shortest iteration time is selected from the target distributed training strategy as the optimal strategy.

[0104] Specifically, as shown in Figure 3 the optimal strategy is obtained based on the target distributed training strategy, and there are various implementation manners. One of the manners is to calculate the iteration time of the target distributed training strategy through a mathematical formula, and select the one with the shortest iteration time as the optimal strategy. If the implementation method of the formula is adopted, the calculation formula of the performance function is as follows:

[0105]

[0106] wherein S(M,C) represents the target distributed training strategy set (M is the model module configuration, and C is the heterogeneous cluster hardware parameter), T is the total time of a single training iteration, pp is the number of stages of pipeline parallelism, Ti is the total time of calculation and communication of the i-th stage, B is the number of micro-batches, and C DP is the gradient aggregation communication time of data parallelism.

[0107] In other embodiments, the method of multi-objective weighted evaluation and the method of reinforcement learning can be adopted to screen the optimal strategy, and no limitation is made.

[0108] Through the embodiment, the optimal strategy is quantitatively screened from the multiple distributed training strategies which are theoretically feasible. The real cost of calculation, communication and memory access in the heterogeneous cluster is fully considered, the subjectivity and uncertainty of the subjective experience decision are avoided, so that the distributed training task can be executed with the highest efficiency, and the overall resource utilization and training throughput of the heterogeneous cluster are significantly improved.

[0109] In some of the embodiments, the large model module for the heterogeneous cluster further includes a load balancing module of the hybrid expert model; the load balancing module of the hybrid expert model is used to optimize the balanced distribution of the word units in the large model training process.

[0110] Specifically, the load balancing module of the mixed expert model takes the group mixed expert model technology as the core, combines the ternary cooperative constraint requirements of heterogeneous cluster computing power, memory, and communication bandwidth, and constructs a closed-loop adaptive mechanism of "cluster state awareness-routing strategy optimization-dynamic load balancing". First, the system monitors the performance state and network link condition of each computing device in the heterogeneous cluster in real time to provide decision basis for dynamic balancing. Second, through its built-in gating network and auxiliary loss function, the mapping relationship between semantic features and expert groups is automatically learned and optimized during the training process, so that the input word units can be intelligently distributed to the most suitable computing node according to the real-time system state and the preset load balancing target during inference, and finally the balanced distribution of word units in the training process is realized.

[0111] Through the present embodiment, the technical gap that the current technology only supports general dense models and has no mixed expert model-specific optimization is effectively filled, so that the heterogeneous cluster can efficiently support the training of sparse expert mixed models, and the coverage range of large model types is expanded. At the same time, with the help of dynamic word unit distribution driven by the group mixed expert model technology, the problem of device load imbalance caused by uneven distribution of word units among experts in the training of mixed expert models is effectively solved, and key technical support is provided for the efficiency, stability and practicality of large model training in heterogeneous clusters.

[0112] In some embodiments thereof, the large model module-level training strategy optimization method for heterogeneous clusters further includes the following steps:

[0113] Obtain bandwidth difference information between different nodes and within nodes in the heterogeneous cluster;

[0114] Based on the bandwidth difference information, the communication links of the heterogeneous cluster are hierarchically optimized and configured.

[0115] Specifically, first, the full-link bandwidth data in the heterogeneous cluster is collected by cluster hardware topology scanning and performance Profiling tools, including the high-speed interconnection bandwidth between each computing unit in the node, the link bandwidth between nodes in the homogeneous device group, and the cross-device connection bandwidth across heterogeneous groups. A bandwidth difference information library containing "link type - bandwidth upper limit - transmission delay - supported protocol" is established to clearly define the resource benchmarks for different communication scenarios. Subsequently, based on the communication needs of different parallel strategies in large model distributed training, hierarchical link binding is performed: for the intra-node scenario, since tensor parallelism requires high-frequency and high-throughput data interaction, it is bound to the intra-node high-speed bandwidth link and uses a collective communication library, which can be NCCL, Gloo, OpenMPI, etc., without limitation; for the inter-node scenario in the homogeneous group, large-scale data aggregation is required for gradient aggregation in data parallelism, and a 400 Gb / s communication link is used for data parallel gradient aggregation, where the communication link can be InfiniBand, Ethernet, and RoCEv2, etc., without limitation; for the cross-heterogeneous group scenario, the activation value transmission of pipeline parallelism requires higher bandwidth stability, which is carried through a 400 Gb / s cross-device dedicated link to avoid occupying high-priority communication resources in the homogeneous group.

[0116] Through the present embodiment, the hierarchical optimization of the communication link realizes the precise matching of the communication demand and the bandwidth resource. On the one hand, the high-bandwidth demand communication is limited in the node or the homogeneous group with sufficient bandwidth, and on the other hand, through the customized configuration of the protocol and the link, the cross-cluster traffic is effectively reduced, thereby significantly optimizing the system resource utilization and reducing the communication overhead, which provides a key communication optimization capability for the large-scale large model training of the domestic heterogeneous cluster.

[0117] In the present embodiment, a large model module-level training strategy optimization device for a heterogeneous cluster is provided, which is used to implement the above-mentioned embodiments and preferred embodiments, and details are not repeated. The terms "module", "unit", "sub-unit" and the like used below can be a combination of software and / or hardware that can realize a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware, or a combination of software and hardware is also possible and is conceived.

[0118] Figure 5 is a structural block diagram of the large model module-level training strategy optimization device for a heterogeneous cluster of the present embodiment, as Figure 5 shown, the device includes an acquisition module 210, an initialization module 220, and an optimization module 230.

[0119] The obtaining module 210 is configured to obtain computation information at a module level and communication information at an operator level of a large model facing a heterogeneous cluster. The computation information includes computation time information of a module under different distributed strategies and on different chips under different data scales, and module memory information. The communication information is communication latency of a communication operator under different data scales.

[0120] The initialization module 220 is configured to determine an initial distributed training strategy of each pipeline based on the module memory information.

[0121] The optimization module 230 is configured to determine computation time of different stages of the pipeline based on the computation time information and the communication information, and optimize each initial distributed training strategy based on the computation time and a memory threshold of a device carrying the large model to obtain a target distributed training strategy.

[0122] The above device solves the problem that resource utilization needs to be improved in related technologies when a large model faces a heterogeneous cluster, and achieves coordination and balance of computation load, memory occupation, and communication overhead among heterogeneous devices, and significant improvement of resource utilization.

[0123] In some embodiments, the obtaining module 210 is further configured to obtain computation information of one layer of modules of the large model facing the heterogeneous cluster.

[0124] The computation time information includes forward computation time and backward computation time of the module. The module memory information includes memory information of input, gradient, weight, and activation value recalculation of the module.

[0125] In some embodiments, the initialization module 220 is further configured to determine, based on the module memory information and a maximum memory value of an accelerator, a minimum memory value and a maximum memory value corresponding to the pipeline under a preset micro-processing batch and a preset parallel mode.

[0126] Based on the minimum memory value and the maximum memory value, the initial distributed training strategy of each pipeline is determined.

[0127] In some embodiments, the optimization module 230 is further configured to determine a module set corresponding to each stage of the pipeline based on the initial distributed training strategy of the pipeline.

[0128] Based on the module set, the computation time information, and the communication information, the computation time of different stages of the pipeline is determined.

[0129] In some embodiments, the optimization module 230 is further configured to determine whether a standard deviation of the computation time of different stages of the pipeline is less than a preset threshold.

[0130] If the standard deviation of the calculation time is greater than or equal to the preset threshold value, moving one level to the adjacent stage of the pipeline until the standard deviation of the calculation time is less than the preset threshold value;

[0131] determining whether the running GPU of the different stages of the pipeline is greater than a GPU threshold value of a device carrying the large model, if the running GPU of the different stages of the pipeline is greater than the GPU threshold value of the device carrying the large model, reducing the running GPU by reducing the micro-batch size, enabling the activation value recalculation, and adjusting the parallelism parameter until the running GPU of the different stages of the pipeline is less than or equal to the GPU threshold value of the device carrying the large model, and obtaining the target distributed training strategy.

[0132] In some embodiments, the optimization module 230 is further configured to, after obtaining the target distributed training strategy, select a distributed training strategy with the shortest iteration time from the target distributed training strategies as the optimal strategy based on a preset performance function.

[0133] In some embodiments, the load balancing module of the mixed expert model is used to optimize the balanced distribution of the word pieces in the large model training process.

[0134] In some embodiments, the optimization module 230 is further configured to obtain bandwidth difference information between different nodes in the heterogeneous cluster and within the nodes.

[0135] Based on the bandwidth difference information, the communication links of the heterogeneous cluster are hierarchically optimized and configured.

[0136] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented by software or hardware. For the modules implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can also be located in different processors in any combination.

[0137] In the embodiment, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any one of the above method embodiments.

[0138] Optionally, the computer device can further include a transmission device and an input / output device, wherein the transmission device is connected with the processor, and the input / output device is connected with the processor.

[0139] Optionally, in the embodiment, the processor can be configured to execute the following steps through the computer program:

[0140] S1, obtaining computing information at a module level and communication information at an operator level of a large model oriented to a heterogeneous cluster; the computing information comprises computing time information and module memory information of the module under different distributed strategies and different chips under different data scales; the communication information is communication delay of the communication operator under different data scales;

[0141] S2, determining an initial distributed training strategy of each pipeline based on the module memory information;

[0142] S3, determining computing time of different stages of the pipeline based on the computing time information and the communication information; and optimizing each initial distributed training strategy according to the computing time and a memory threshold of the device carrying the large model to obtain a target distributed training strategy.

[0143] It should be noted that the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, and will not be described herein again.

[0144] In addition, in combination with the large model module level training strategy optimization method oriented to a heterogeneous cluster provided in the above embodiments, a storage medium can also be provided to implement the method in the embodiment. The storage medium has a computer program stored thereon; the computer program is executed by a processor to implement any one of the large model module level training strategy optimization methods oriented to a heterogeneous cluster in the above embodiments.

[0145] It should be understood that the specific embodiments described herein are only used to explain this application, but not to limit it. According to the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the present application.

[0146] Obviously, the drawings are only some examples or embodiments of the present application, and can be applied to other similar situations without creative labor for those of ordinary skill in the art. In addition, it can be understood that although the work done in the development process may be complex and long, some design, manufacture or production changes according to the technical content disclosed in the present application are only conventional technical means for those of ordinary skill in the art, and should not be regarded as insufficient disclosure of the present application.

[0147] The word "embodiment" in the present application means that the specific features, structures or characteristics described in combination with the embodiments can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean independence or alternative to other embodiments. It can be clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0148] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of patent protection. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for optimizing training strategies at the module level of large models for heterogeneous clusters, characterized in that, The method includes: Acquire computational information at the module level and operator level at the operator level for large models targeting heterogeneous clusters; the computational information includes computation time information and module memory information for modules with different data scales under different distributed strategies and on different chips; the communication information is the communication latency of communication operators under different data scales; the computation time information includes the forward computation time and backward computation time of the module; the module memory information includes the memory information of the module's input, gradient, weights, and activation value recalculation; The step of obtaining computational information at the module level of a large model for heterogeneous clusters includes: obtaining computational information of a module at one layer of the large model for heterogeneous clusters; Based on the memory information of the module, determine the initial distributed training strategy for each pipeline. Based on the computation time information and the communication information, the computation time of different stages of the pipeline is determined; Based on the computation time and the memory threshold of the device supporting the large model, each of the initial distributed training strategies is optimized to obtain the target distributed training strategy.

2. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The step of determining the initial distributed training strategy for each pipeline based on the module's memory information includes: Based on the module memory information and the accelerator's maximum memory value, determine the minimum and maximum memory values ​​of the pipeline corresponding to the preset microprocessor batch and the preset parallel mode. Based on the minimum and maximum memory values, an initial distributed training strategy is determined for each pipeline.

3. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, Determining the computation time of different stages of the pipeline based on the computation time information and the communication information includes: Based on the initial distributed training strategy of the pipeline, the module set corresponding to each stage of the pipeline is determined. The computation time for different stages of the pipeline is determined based on the module set, the computation time information, and the communication information.

4. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The step of optimizing each initial distributed training strategy based on the computation time and the memory threshold of the device supporting the large model to obtain the target distributed training strategy includes: Determine whether the standard deviation of the computation time at different stages of the pipeline is less than a preset threshold; If the standard deviation of the computation time is greater than or equal to the preset threshold, then move one level to the adjacent stage of the pipeline until the standard deviation of the computation time is less than the preset threshold. Determine whether the running GPU memory at different stages of the pipeline is greater than the GPU memory threshold of the device carrying the large model. If it is greater than the GPU memory threshold, reduce the running GPU memory by decreasing the micro-batch size, enabling activation value recalculation, and adjusting the parallelism parameter until the running GPU memory at different stages of the pipeline is less than or equal to the GPU memory threshold of the device carrying the large model, and obtain the target distributed training strategy.

5. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The method further includes: After obtaining the target distributed training strategy, the distributed training strategy with the shortest iteration time is selected as the optimal strategy based on a preset performance function.

6. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The module also includes a load balancing module based on a hybrid expert model; The load balancing module of the hybrid expert model is used to optimize the balanced distribution of words during the training of the large model.

7. The method for optimizing large model module-level training strategies for heterogeneous clusters according to claim 1, characterized in that, The method further includes: Obtain bandwidth difference information between and within different nodes in the heterogeneous cluster; Based on the bandwidth difference information, the communication links of the heterogeneous cluster are configured in a hierarchical manner for optimization.

8. A device for optimizing training strategies at the module level of large models for heterogeneous clusters, characterized in that, The device includes: an acquisition module, an initialization module, and an optimization module; The acquisition module is used to acquire computational information at the module level and operator-level communication information for large models oriented towards heterogeneous clusters. The computational information includes computation time information and module memory information for modules with different data scales under different distributed strategies and on different chips. The communication information is the communication latency of communication operators under different data scales. The computation time information includes the forward computation time and backward computation time of the module. The module memory information includes the memory information of the module's input, gradient, weights, and activation value recalculation. The step of obtaining computational information at the module level of a large model for heterogeneous clusters includes: obtaining computational information of a module at one layer of the large model for heterogeneous clusters; The initialization module is used to determine the initial distributed training strategy for each pipeline based on the module's memory information. The optimization module is used to determine the computation time of different stages of the pipeline based on the computation time information and the communication information; and to optimize each initial distributed training strategy according to the computation time and the memory threshold of the large model device to obtain the target distributed training strategy.

9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the method for optimizing large model module-level training strategies for heterogeneous clusters as described in any one of claims 1 to 7.

Images