A heterogeneous cluster hybrid parallel training method and system combined with a freezing mechanism

Abstract

The application discloses a heterogeneous cluster mixed parallel training method and system combined with a freezing mechanism and belongs to the technical field of neural networks. The application is combined with the characteristics of a target heterogeneous hardware cluster, predicts the calculation time and storage requirements of each layer in a model on different types of hardware, adopts dynamic programming to solve the optimal pipeline segmentation and data parallel configuration, and obtains a mixed parallel training scheme. When the freezing state changes in the training process, the system reallocates the released video memory and computing power resources, dynamically adjusts the parallel scheme, and realizes load balancing and training efficiency improvement. The application effectively reduces resource occupation and improves the performance of large-scale model training in a heterogeneous environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of neural networks, relates to deep learning technology, and particularly relates to a heterogeneous cluster hybrid parallel training method and system combining a freezing mechanism. BACKGROUND

[0002] With the rapid development of deep learning, models based on the Transformer architecture (such as GPT-5, DeepSeek-R1, BERT 3, etc.) have achieved remarkable results in natural language processing, multi-modal understanding, and intelligent decision-making. However, the parameter size of deep learning models grows exponentially, reaching hundreds of billions or even trillions, which puts extremely high demands on computing and communication resources. The computing power and memory resources of a single computing device (such as a single GPU) cannot support such a large model training, so distributed parallel training methods based on large-scale heterogeneous GPU clusters have become the mainstream choice for training large models.

[0003] Existing parallel training strategies mainly include data parallelism, model parallelism, and pipeline parallelism. Data parallelism accelerates by replicating the model on multiple devices and processing different data shards, but as the model parameters expand, the parameter synchronization communication volume and memory occupancy increase dramatically. Model parallelism divides model parameters or computing layers across multiple devices to alleviate memory pressure, but when there is frequent cross-device communication or uneven layer division, some GPUs may be idle, and overall throughput may decrease. Pipeline parallelism divides the model into multiple stages, processes different stages on different devices, and overlaps computation and communication through micro-batches to improve device utilization, but its performance depends on reasonable stage division and scheduling strategies, and is still constrained by activation value storage and load imbalance.

[0004] To balance performance and resource efficiency in different dimensions, researchers have proposed hybrid parallelism. Hybrid parallelism combines the advantages of data parallelism and pipeline parallelism, combining data parallelism, model parallelism, and pipeline parallelism in multiple dimensions. For example, pipeline parallelism is used between stages, and data parallelism is used within stages to balance communication and computation load, which can balance model size expansion and training efficiency to some extent. Typical training frameworks that support hybrid parallelism include Megatron-LM and DeepSpeed, etc.

[0005] Although the above solutions have achieved certain results in large-scale model training, they generally have the limitation of static division and fixed scheduling strategies, that is, the parallel strategy (including pipeline stage division and the number of devices for data parallelism within each stage) is determined according to the model structure and GPU cluster hardware environment before training begins, and remains unchanged throughout the training process. With the changes in layer parameter update rate and gradient distribution during model training, static division often leads to overloading of some devices and idling of some devices, resulting in decreased overall computing efficiency.

[0006] On the other hand, the model freezing technology is introduced to further reduce the training overhead. This technology identifies the model layers that have basically converged or contribute less to the task by monitoring the gradient size or activation feature changes, and skips the back propagation and parameter update operations of these layers, thereby saving computing and memory resources. However, the introduction of the freezing mechanism into hybrid parallel training brings new challenges:

[0007] 1. Dynamic load imbalance: The freezing mechanism causes the computational load of different parts of the model to change dynamically during training. The static hybrid parallel scheme cannot adapt to this change, and the originally balanced pipeline stages may become unbalanced due to the freezing of some layers, resulting in new computational bottlenecks and reducing overall efficiency.

[0008] 2. Complexity of heterogeneous environment: Modern GPU clusters often contain GPUs of different models and performance from the same manufacturer (heterogeneous computing resources), and there may be different bandwidth connections between nodes and within nodes (heterogeneous network interconnection). Static or simple dynamic adjustment strategies cannot fully utilize the characteristics of heterogeneous resources, such as failing to allocate compute-intensive tasks to powerful GPUs or failing to prioritize high-bandwidth connections for data parallel communication.

[0009] 3. Difficulty in planning optimal solutions: In the context of dynamically changing freezing states and heterogeneous hardware environments, it is a complex optimization problem to accurately evaluate the performance of different hybrid parallel schemes (stage division, data parallelism allocation) in real time and quickly find or approach the optimal solution. Traditional methods based on static models or simple heuristic rules are difficult to meet the requirements.

[0010] Existing technical solutions (including methods supported by frameworks such as Megatron-LM and DeepSpeed) are difficult to achieve dynamic optimal balance between training efficiency and resource utilization in the context of model dynamic freezing and heterogeneous computing resources. Therefore, there is an urgent need for a training system and method that can adapt to model dynamic freezing, fully utilize heterogeneous computing resources, and optimize hybrid parallel strategies in real time to achieve efficient training of large-scale deep learning models. SUMMARY

[0011] To solve the problem that existing technologies cannot adapt to the dynamic changes of model freezing and efficiently utilize heterogeneous hardware resources when training large-scale models, the present application provides a heterogeneous cluster hybrid parallel training method and system combined with freezing mechanism, which dynamically optimizes heterogeneous cluster resource allocation and hybrid parallel strategy by identifying and utilizing the GPU computing and video memory resources released by frozen layers in the model, thereby improving training efficiency and reducing resource occupation.

[0012] In order to achieve the above object, the technical scheme adopted by the present application is:

[0013] A heterogeneous cluster hybrid parallel training method combined with a freezing mechanism, comprising the following steps:

[0014] Step 1, model analysis: receiving a deep learning model, analyzing the model network structure, and combining the characteristics of the target heterogeneous hardware cluster to predict the computing time and storage requirements of each layer in the model on different types of hardware, and considering the influence of the freezing state on the storage requirements and computing time;

[0015] Step 2, scheme planning: according to the model layer freezing condition, model analysis result and heterogeneous hardware information, the optimal pipeline segmentation and data parallel configuration is solved by dynamic programming to obtain a hybrid parallel training scheme; in the dynamic programming process, the constraint that the memory occupation of all model layers allocated on a specific GPU device does not exceed the upper limit of the device needs to be met at the same time;

[0016] Step 3, model training: executing the training process of the hybrid parallel training scheme, when the freezing state changes during the training process, the system re-allocates the released memory and computing power resources, dynamically adjusts the parallel scheme, and realizes load balancing and training efficiency improvement.

[0017] Further, the step 1 specifically comprises:

[0018] (1) using a model analysis tool to obtain the type, dimension and connection method of each layer of the model;

[0019] (2) using a hardware detection tool to obtain the GPU model, computing power, memory size and interconnection bandwidth information of each GPU in the heterogeneous cluster;

[0020] (3) computing time prediction: for each layer, according to its operation type, input and output dimension, parameter quantity, and the computing power of the GPU type on which it will be run , using a performance model or an empirical formula to predict the computing time :

[0021] ,

[0022] Storage requirement prediction: calculate the memory required by each layer, including parameter occupied memory , optimizer state occupied memory , activation occupied memory and gradient occupied memory :

[0023] ,

[0024] ,

[0025] ,

[0026] ,

[0027] The final result shows the actual memory occupied by one layer:

[0028] ,

[0029] in, Number of input / output channels The kernel size is the convolution kernel size. To output the height and width of the feature map, and The height and width of the input feature map.

[0030] Furthermore, step 2 specifically includes the following process:

[0031] The goal of the optimization scheme is to minimize the expected total time for a single training batch. ,in From the start-up phase time Stable phase time and end phase time constitute:

[0032] ,

[0033] ,

[0034] ,

[0035] ,

[0036] Conditional constraints are imposed on the actual amount of GPU memory used by the model during training:

[0037] ,

[0038] in, This is the bottleneck stage. Stages Forward and backward propagation times on the allocated heterogeneous GPUs, , Each represents a stage Forward and backward propagation times, Representative stage The backpropagation time, This refers to the quantity in a micro-batch. For the stage The set of parameters involved In the equipment group All-Reduce overhead on the GPU, Max memory capacity of the GPU device device, Total number of layers in the model, i.e., the sum of the number of all convolutional layers, residual blocks, and fully connected layers contained in the network, The layer , The frozen state of layer , The memory requirement of the layer , The heterogeneous GPU set;

[0039] First identify the pipeline bottleneck stage , and then use dynamic programming to solve the optimal hybrid parallel configuration; when the frozen state of the model changes, re-plan the resources released by the frozen layers; define the problem objective function as :

[0040] ,

[0041] ,

[0042] ,

[0043] wherein, Total number of layers in the model, i.e., the sum of the number of all convolutional layers, residual blocks, and fully connected layers contained in the network, the frozen state list Will affect the calculation and communication time of each stage, Total GPU number, Heterogeneous GPU set, Layer number segmentation point, Allocated GPU number, Specific GPU subset, the optimization process needs to consider The heterogeneity of GPUs in and the interconnection bandwidth between them, Indicates all possible combinations of selecting GPUs from all GPU sets ;

[0044] When predicting the level of calculation time, consider the impact of the frozen state on calculation: if layer is in the frozen state, its forward propagation still needs to be performed but does not participate in backpropagation, and the backpropagation time =0;

[0045] In terms of computing resources, the gradient of the frozen layer and the optimizer state memory requirement are zero, and the memory and computing resources released by the freezing are reallocated;

[0046] The problem is solved using the idea of dynamic programming to obtain the optimal solution.

[0047] The frozen state of the model layers is monitored during the training process; when a layer is frozen or unfrozen, the computational load and memory occupancy of each pipeline stage are re-evaluated; according to the updated load and available resources, the optimal hybrid parallel scheme is recalculated and applied.

[0048] Further, the process of solving the problem using the idea of dynamic programming includes:

[0049] Frozen state list For known information, define the state For the previous Layer, a heterogeneous GPU subset is used The minimum pipeline time at this time, where the heterogeneous GPU subset Contains GPU; by iteratively expanding the state, consider assigning the next layer ) to a new GPU subset , calculate the new stage time and total time resulting therefrom, and update the DP table to obtain the final Optimal solution.

[0050] Further, step 3 uses a computation graph to represent and schedule operations, and constructs a forward and backward propagation subgraph for each pipeline stage. The core scheduling strategy is 1F1B.

[0051] Further, the 1F1B strategy refers to scheduling the backward propagation of a micro-batch as early as possible after the forward propagation of the micro-batch is completed within a pipeline stage, and alternating with the forward propagation of subsequent micro-batches.

[0052] Further, step 3 uses the following communication mechanism for data transmission when processing cross-stage communication:

[0053] During forward propagation, when the data parallelism of the downstream stage increases or the GPU type / bandwidth changes, use a separate layer to copy or distribute data; when the data parallelism of the downstream stage decreases or needs to be aggregated to a specific type / bandwidth of GPU, use an aggregation layer to collect and integrate data;

[0054] During backward propagation, when the data parallelism of the downstream stage increases or the GPU type / bandwidth changes, use gradient replication; when the data parallelism of the downstream stage decreases or needs to aggregate gradients to a specific type / bandwidth of GPU, use gradient aggregation, where the overhead of All-Reduce is affected by the bandwidth between the heterogeneous GPUs involved.

[0055] The application also provides a heterogeneous cluster hybrid parallel training system combined with the freezing mechanism, comprising a model analysis module, a scheme planning module and a model training module, and is used for realizing the above-mentioned heterogeneous cluster hybrid parallel training method combined with the freezing mechanism.

[0056] Compared with the prior art, the application has the following advantages and beneficial effects:

[0057] 1. Strong adaptability: through dynamic scheme planning, the change of the calculation load caused by the model layer freezing in the training process can be adapted, and high training efficiency can be maintained at all times.

[0058] 2. Heterogeneous awareness: the differences in GPU performance (calculation speed, memory) and network topology heterogeneity (bandwidth) are explicitly considered, targeted task allocation and communication optimization are performed, and the heterogeneous cluster resources are fully utilized.

[0059] 3. Efficiency improvement: the expansibility of hybrid parallel, the memory efficiency of 1F1B scheduling, the calculation saving of the freezing mechanism and the optimization of heterogeneous hardware are combined, and the training time of a large-scale model is significantly shortened.

[0060] 4. Resource optimization: through dynamic adjustment of the parallel strategy, the calculation, memory and bandwidth resources are more reasonably allocated and utilized, and the overall resource utilization rate is improved. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is an internal module schematic diagram of the heterogeneous hybrid parallel training method combined with the freezing mechanism provided by the application.

[0062] Figure 2 is a typical Gantt chart example of hybrid parallel training, which shows the 1F1B scheduling strategy.

[0063] Figure 3 is a separation layer and aggregation layer design schematic diagram for processing cross-stage communication in hybrid parallel.

[0064] Figure 4 is a bottleneck stage algorithm.

[0065] Figure 5 is a dynamic planning solution algorithm. DETAILED DESCRIPTION

[0066] The technical solutions provided by the application will be described in detail below in combination with specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the application and not to limit the scope of the application.

[0067] REFERENCE Figure 1The overall implementation framework of the heterogeneous mixed parallel training method combined with the freezing mechanism comprises three main modules: a model analysis module, a scheme planning module, and a model training module. The framework is equipped with a dynamic adjustment design for the freezing mechanism. The above three modules are software modules that realize corresponding functions, and therefore the present application actually comprises three steps, namely a model analysis step, a scheme planning step, and a model training step, which correspond to the specific functions realized by the three modules respectively. The functions realized by the model analysis module, the scheme planning module, and the model training module can constitute a system for realizing the heterogeneous mixed parallel training method combined with the freezing mechanism. In order to cooperate with the drawings, the following is described in the form of modules.

[0068] The framework first calls the model analysis module, obtains the model information to be trained and the heterogeneous device information, and measures the training performance indicators, which are passed to the scheme planning module as known information. The model analysis module is called only once.

[0069] The scheme planning module is responsible for formulating and dynamically adjusting the mixed parallel strategy; the model training module actually executes the training process according to the configuration output by the scheme planning module. In order to adapt to the freezing mechanism during training, the system allows the scheme planning module and the model training module to call each other multiple times. When freezing occurs during the model training process, the system will pause the training process, update the model layer freezing information, and re-call the scheme planning module to construct a heterogeneous mixed parallel training scheme under the current latest state. Then, the model training module is restarted for training.

[0070] The detailed description of the functions of each module (step) is as follows.

[0071] Model analysis module:

[0072] The goal of this module is to obtain model information, model layer freezing conditions, and heterogeneous hardware information, and to predict performance indicators accordingly. When calculating the time of a layer at a prediction level, if the layer is in a frozen state, its forward propagation still needs to be executed but does not participate in backpropagation, the backpropagation time is 0, and the gradient and optimizer state memory requirements are zeroed, thereby releasing the corresponding video memory and computing resources, which can increase the data parallelism of some stages or adjust the stage boundaries to better balance the load on the heterogeneous devices.

[0073] Specifically, first, model structure analysis is performed. Receive the deep learning model file provided by the user (for example, in PyTorch, TensorFlow, or ONNX format). Use tool libraries such as ONNX to parse the model, traverse the computation graph, and obtain detailed information of each layer, including layer type (such as convolution, fully connected, Transformer layer, etc.), input and output tensor dimensions, parameter quantity, and connection relationship between layers. ​

[0074] Second, the computational resource and hardware condition analysis is performed. The system needs to be deployed on a heterogeneous GPU cluster, which can contain different models (at least two models) of GPUs (such as NVIDIA A100, V100, etc., which have different computing capabilities and memory sizes) and different interconnection methods (such as high-speed NVLink, PCIe bus, Ethernet, and the heterogeneous cluster should contain at least two different transmission rates of GPU interconnection). Use nvidia-smi or lower-level libraries to query the specific model, computing power (FLOPS), available memory of each GPU in the cluster, and detect the interconnection topology and point-to-point bandwidth between GPUs. These heterogeneous information is the basis for subsequent planning.

[0075] Finally, the performance index prediction is performed. Based on the model layer freezing condition, the parsed model structure, and the collected heterogeneous hardware information, the execution performance of each part of the model on the specific hardware is predicted.

[0076] 1. Calculation time prediction: for each layer, according to its operation type, input and output dimensions, parameter quantity, and the computing power of the GPU on which it will be run , use the performance model or empirical formula to predict the calculation time:

[0077] .

[0078] 2. Space cost prediction: calculate the memory required for each layer, including parameter-occupied memory , optimizer state-occupied memory , activation-occupied memory , and gradient-occupied memory , and finally obtain the actual memory occupied by a layer:

[0079] ,

[0080] ,

[0081] ,

[0082] ,

[0083] The total memory occupied by a layer that is not frozen can be obtained by combining the above formula :

[0084] ,

[0085] Key, this module considers the impact of freezing state on resource demand, which can be further described as:

[0086] ,

[0087] where, is the frozen state of layer , means that the layer is frozen and does not occupy the computation and gradient / optimizer related memory, means that the layer is not frozen and normally occupies the memory. is the number of input and output channels, is the size of the convolution kernel, is the height and width of the output feature map, and is the height and width of the input feature map.

[0088] These results (time and memory occupancy of each layer on different heterogeneous GPUs) will be passed as input to the scheme planning module.

[0089] Scheme planning module:

[0090] This module is the core of the application and is responsible for formulating and dynamically adjusting the mixed parallel strategy for the current model layer freezing situation.

[0091] In terms of computation time, since the frozen layer no longer performs backpropagation, when the layer is frozen ( =0), Therefore, the total computation time of the stage where the frozen layer is located decreases, but the forward propagation of the frozen layer still needs to be performed.

[0092] In terms of memory occupancy, the gradient and optimizer state memory requirements of the frozen layer are zero, thus freeing up the corresponding video memory and computing resources, which can be used to increase the data parallelism of some stages or adjust the stage boundaries to better balance the load on heterogeneous devices.

[0093] The core goal of scheme planning is to find a mixed parallel scheme (including pipeline stages and data parallel configuration within each stage) that divides the model layer (including frozen layers) into a set of heterogeneous GPUs ( ) so that the training time of a single batch is minimized, and the training time is composed of the start time , the stable time and the end time , the specific calculation process is as follows:

[0094] ,

[0095] where, is the bottleneck stage, i.e., the stage with the longest time consumption in mixed parallel, is the stage Forward propagation time, This is the bottleneck stage. The sum of all previous forward propagation times.

[0096] ,

[0097] in, Represents the number of micro-batches. , Each represents a stage The forward and backward propagation times, i.e., the stages The sum of the time for all forward and backward propagations except for the first forward propagation and the last backward propagation.

[0098] ,

[0099] in, Representative stage The All-Reduce overhead (measured by the program) has the following parameter set: GPU devices are ; Representative stage The backpropagation time; That is, the stage The sum of the backpropagation time of all subsequent stages and all All-Reduce overhead.

[0100] ,

[0101] ,

[0102] Where T is the computation time for one batch. This represents the maximum video memory capacity of the device. Considering the frozen result obtained for model analysis Layer memory requirements, This refers to the total number of layers in the model, which is the sum of all convolutional layers, residual blocks, and fully connected layers in the network. For the model number Layers. During training, the hybrid parallel scheme is dynamically adjusted based on changes in the frozen state of the model layers: when the frozen state changes, the system dynamically reassesses memory usage and computational load. By utilizing the memory and computing resources released by the frozen layer to adjust the hybrid parallel configuration, load balancing and training efficiency can be improved while maintaining memory constraints.

[0103] When calculating these times, the effects of heterogeneity are considered as follows:

[0104] 1. and (stage Forward / backward propagation time depends on the stage The freezing status and the specific model and quantity of GPUs allocated to that stage are measured by the program to obtain the actual values. If the stage... If data parallelism is involved, then it is the time required to perform one data parallel operation (computation + communication) on this group of heterogeneous GPUs. (Stage) The frozen layer in the process does not require backpropagation.

[0105] 2. (Parameter set) The overhead of All-Reduce depends on the number of parameters and the subset of GPUs participating in data parallelism. Internal interconnect bandwidth. The bandwidth between different GPU pairs may vary, affecting the efficiency of All-Reduce.

[0106] 3. The data transfer time across stages also depends on the connection bandwidth between the sending and receiving GPUs.

[0107] 4. The size of the model that a GPU can accommodate is limited by its video memory capacity.

[0108] The specific implementation process of the plan is as follows:

[0109] (I) Bottleneck Identification Stage

[0110] Due to the stable phase Typically dominates the total time , find The largest stage Crucially, as shown in Algorithm 1, this algorithm dynamically determines the bottleneck stage Z of the pipeline parallelism by scanning the time cost of each training stage in reverse, in order to minimize the training time of a batch. Its core idea is: first, initialize the bottleneck stage Z and the scan pointer s; then, backtrack from the end of the pipeline, comparing the fixed parallel cost of the current stage stage by stage. ) and the cumulative cost of candidate bottleneck stage Z ( If the current stage has a higher cost, it is updated to a new bottleneck stage, and finally the stage with the highest cumulative time cost is selected as the bottleneck optimization target. Through this reverse greedy strategy, the algorithm can... Adaptive adjustment of the parallel scheme within time complexity, considering That is, the number of stages, usually has Therefore, this algorithm can calculate the bottleneck stage in a short time. This effectively balances the computational load across different stages of the pipeline. Specific algorithm 1 is as follows:Figure 4 as shown.

[0111] (ii) Determine the hybrid parallel scheme under the current frozen state, solve the optimal configuration

[0112] Determine the bottleneck stage by Algorithm 1 After that, the hybrid parallel scheme needs to be further determined, the goal of which is to minimize the overall pipeline latency by recursively splitting the pipeline stages and enumerating the device allocation strategies. This problem can be formalized as follows:

[0113] ,

[0114] ,

[0115] ,

[0116] where is the total number of layers of the model, i.e., the sum of the number of all convolutional layers, residual blocks, and fully connected layers contained in the network, the frozen state list will affect the computation and communication time of each stage, is the total number of GPUs, is the heterogeneous GPU set, is the layer segmentation point, is the number of allocated GPUs, is the specific GPU subset, the optimization process needs to consider the heterogeneity of GPUs in and the interconnection bandwidth between them, denotes all possible combinations of GPUs selected from all GPU sets .

[0117] This problem can be solved using the idea of dynamic programming. The frozen state list is known information, and the state is the minimum pipeline time when the first layers are processed using the heterogeneous GPU subset (containing GPUs). By iteratively expanding the state, considering assigning the next segment of layers ( ) to a new GPU subset (the heterogeneity of which is known), the new stage time and total time resulting from this are calculated, and the DP table is updated. Finally is the optimal solution, as shown in Algorithm 2 in Figure 5 .

[0118] Model training module:

[0119] The module actually executes the training process according to the configuration output by the scheme planning module. It uses a computation graph to represent and schedule the operations. A subgraph for forward and backward propagation is constructed for each pipeline stage. The core scheduling strategy is 1F1B, i.e., one forward propagation followed by one backward propagation, as shown in Figure 2 The scheduler ensures that the backward propagation of a micro-batch must start after the completion of its forward propagation and is interleaved with the forward propagation of subsequent micro-batches, which effectively reduces the peak memory occupancy.

[0120] To handle the complex communication requirements brought by heterogeneous environments and mixed parallelism (e.g., one stage uses 1 GPU, the next stage uses 2 GPUs of different models connected through slower PCIe), the module contains special communication layers: a split layer for distributing data from a small number of devices to multiple devices in forward propagation, and an aggregate layer for collecting the results of multiple devices to a small number of devices; in backward propagation, gradient replication and gradient aggregation (such as All-Reduce) are used to handle similar situations. The design of these communication operations needs to consider the capability differences between the source and target devices and the connection bandwidth between them.

[0121] The key of this module is to handle cross-stage communication, especially in heterogeneous environments and dynamic parallelism, as shown in Figure 3 :

[0122] 1. Forward propagation: if the output of a stage needs to be passed to the downstream stage , and the data parallelism of stage is greater than that of stage (or the receiving GPU type / bandwidth needs special handling), a split layer is inserted at the end of stage . This layer is responsible for copying or splitting the activation tensor on a single (or a small number of) GPU and sending it to all related GPUs in stage , taking into account the actual bandwidth between them. Conversely, if the parallelism is reduced or needs to be aggregated to a GPU of a specific type / bandwidth, an aggregate layer is inserted at the beginning of stage , which collects data from multiple source GPUs and passes it to the subsequent computation after integration.

[0123] 2. Backward propagation: the gradient needs to be propagated backward. If the gradient of stage needs to be passed back to the upstream stage (the previous stage on the backward propagation path) , and the data parallelism of stage is greater than that of stage ​If the GPU type / bandwidth is different, then in stage to stage After passing the gradient, stage internally needs Gradient Replication to ensure that all data parallel copies receive the gradient. Conversely, if the parallelism of stage is less than that of stage , or if the gradient needs to be aggregated to a specific type / bandwidth of GPU, then in stage , after calculating the gradient, a Gradient Aggregation operation (usually All-Reduce) needs to be performed to accumulate the gradients of all copies, and then passed to the GPU of stage . The performance of All-Reduce ( ) depends heavily on the interconnect bandwidth between participating GPUs , which has been considered in the scheme planning part.

[0124] In addition, the module is provided with a dynamic adjustment function, which is the key to deal with the freezing mechanism. During the training process, when the system detects that the freezing state of some layers changes, the computational load ( becomes 0 or decreases) and memory requirements of these layers will change. This may cause the original load balancing to be broken, and the bottleneck stage may shift. At this time, the computational load ( ) and memory occupancy of each pipeline stage s are re-evaluated, and the scheme planning module needs to be re-invoked. It uses the updated layer state (new value) and the performance prediction provided by the model parsing module to re-run the optimization process (step (1) identify the bottleneck stage and step (2) determine the hybrid parallel scheme under the current freezing condition, solve the optimal configuration), find the optimal hybrid parallel scheme under the current state, and apply the new scheme (stage division, GPU allocation, etc.) to the model training module. The memory and computing resources released by the unfreezing can be reallocated, such as increasing the data parallelism of some stages, or adjusting the stage boundaries to better balance the load on heterogeneous devices.

[0125] Through the coordinated work of the three modules, the invention realizes an efficient hybrid parallel training method that can dynamically adapt to changes in model freezing on a heterogeneous GPU cluster. It not only solves the capacity and efficiency problem of large model training, but also fully utilizes the characteristics of heterogeneous hardware and can flexibly respond to dynamic changes during training.

[0126] Based on the method scheme, the application further provides a heterogeneous hybrid parallel training system combining a freezing mechanism, which is realized by computer software and includes a model analysis module, a scheme planning module and a model training module.

[0127] It should be noted that the above content only illustrates the technical idea of the application, and cannot limit the protection scope of the application. For ordinary skilled persons in the art, several improvements and refinements can be made without departing from the principles of the application, and these improvements and refinements all fall within the protection scope of the claims of the application.

Claims

1. A heterogeneous cluster hybrid parallel training method incorporating a freezing mechanism, characterized in that, Includes the following steps: Step 1, Model Analysis: Receive the deep learning model, analyze the model's network structure, and combine the characteristics of the target heterogeneous hardware cluster to predict the computation time and storage requirements of each layer in the model on different types of hardware, and consider the impact of the frozen state on storage requirements and computation time. Step 2, Scheme Planning: Based on the model layer freezing status, model parsing results, and heterogeneous hardware information, dynamic programming is used to solve for the optimal pipeline partitioning and data parallel configuration, resulting in a hybrid parallel training scheme. During the dynamic programming process, it is necessary to simultaneously satisfy the constraint that the memory usage of all model layers allocated on a specific GPU device does not exceed the device's upper limit. Specifically, this includes the following steps: The goal of the optimization scheme is to minimize the expected total time for a single training batch. ,in From the start-up phase time Stable phase time and end phase time constitute: ， ， ， ， Conditional constraints are imposed on the actual amount of GPU memory used by the model during training: ， in, This is the bottleneck stage. Stages Forward and backward propagation times on the allocated heterogeneous GPUs, , Each represents a stage Forward and backward propagation times, Representative stage The backpropagation time, This refers to the quantity in a micro-batch. For the stage The set of parameters involved In the equipment group All-Reduce overhead, This refers to the maximum video memory capacity of the GPU device. This represents the total number of layers in the model, which is the sum of all convolutional layers, residual blocks, and fully connected layers in the network. For the model number layer, For layer The frozen state The first step in the model analysis process, after considering the frozen state, is the calculation of the... Layer memory requirements, It is a heterogeneous GPU collection; First, identify the bottleneck stage of the production line. Then, dynamic programming is used to solve for the optimal hybrid parallel configuration; when the frozen state of the model changes, replanning will be performed in conjunction with the resources released from the frozen layer; the objective function of the problem is defined as follows. : ， ， ， in, The total number of layers in the model, i.e., the sum of all convolutional layers, residual blocks, and fully connected layers in the network, is the frozen state list. This will affect the computation and communication time at each stage. This represents the total number of GPUs. It is a heterogeneous GPU collection. As the dividing point of the layer, The number of GPUs allocated, For a specific subset of GPUs, the optimization process needs to consider The heterogeneity of GPUs and the interconnect bandwidth between them, Indicates from all GPU collections Selected All combinations of GPUs; When predicting the computation time for a hierarchical level, the impact of the frozen state on the computation should be considered: if the level If the process is frozen, its forward propagation still needs to be executed but it does not participate in the backward propagation. The backward propagation time is... =0; In terms of computing resources, the memory requirements of the gradients and optimizer states in the frozen layer are reduced to zero, and the memory and computing resources released by the freeze are reallocated. The optimal solution can be obtained by using the concept of dynamic programming. During training, monitor changes in the frozen state of model layers; when a layer is frozen or unfrozen, reassess the computational load and memory usage of each pipeline stage; based on the updated load and available resources, recalculate and apply the optimal hybrid parallel scheme. Step 3, Model Training: The training process of the hybrid parallel training scheme is executed. During the training process, when the frozen state changes, the system reallocates the released video memory and computing resources, dynamically adjusts the parallel scheme, and achieves load balancing and improved training efficiency.

2. The heterogeneous cluster hybrid parallel training method combining a freezing mechanism according to claim 1, characterized in that, Step 1 specifically includes: (1) Use model parsing tools to obtain the type, dimensions and connection methods of each layer of the model; (2) Use hardware detection tools to obtain information on the GPU model, computing power, memory size, and interconnect bandwidth between GPUs in the heterogeneous cluster; (3) Computation time prediction: For each layer, the prediction is based on its operation type, input / output dimensions, number of parameters, and the computing power of the GPU model on which it will be run. Predict computation time using performance models or empirical formulas. : ， Storage requirement forecasting: Calculate the memory required for each layer, including the memory occupied by parameters. Memory occupied by optimizer state Activation of occupied memory and the memory occupied by gradients : ， ， ， ， The final result shows the actual memory occupied by one layer: ， in, Number of input / output channels The kernel size is the convolution kernel size. To output the height and width of the feature map, and The height and width of the input feature map.

3. The heterogeneous cluster hybrid parallel training method combining a freezing mechanism according to claim 1, characterized in that, The process of solving problems using the concept of dynamic programming includes: Frozen status list Define the state based on the known information. Before processing Layers that utilize heterogeneous GPU subsets Minimum pipeline time for heterogeneous GPU subset Include One GPU; by iteratively expanding the state, consider moving to the next layer ( Assign a new subset of GPUs Calculate the resulting new stage time and total time, update the DP table, and obtain the final result. This is the optimal solution.

4. The heterogeneous cluster hybrid parallel training method combining a freezing mechanism according to claim 1, characterized in that, Step 3 uses a computation graph to represent and schedule operations, constructing forward and backward propagation subgraphs for each pipeline stage. The core scheduling strategy is 1F1B.

5. The heterogeneous cluster hybrid parallel training method combining a freezing mechanism according to claim 4, characterized in that, The 1F1B strategy refers to scheduling the backpropagation of a microbatch as early as possible after the forward propagation of a microbatch is completed within a pipeline stage, alternating with the forward propagation of subsequent microbatches.

6. The heterogeneous cluster hybrid parallel training method combining a freezing mechanism according to claim 1, characterized in that, Step 3 uses the following communication mechanism for data transmission when handling cross-stage communication: During forward propagation, when the data parallelism of downstream stages increases or the receiving GPU type / bandwidth differs, a separation layer is used to copy or distribute data; when the data parallelism of downstream stages decreases or needs to be converged to a specific type / bandwidth GPU, an aggregation layer is used to collect and integrate data. During backpropagation, gradient copying is used when the data parallelism of the downstream stage increases or the GPU type / bandwidth differs; gradient aggregation is used when the data parallelism of the downstream stage decreases or gradients need to be aggregated to a specific type / bandwidth of GPU, where the overhead of All-Reduce is affected by the bandwidth between the heterogeneous GPUs involved.

7. A heterogeneous cluster hybrid parallel training system incorporating a freezing mechanism, characterized in that, It includes a model parsing module, a scheme planning module, and a model training module, used to implement the heterogeneous cluster hybrid parallel training method with freezing mechanism as described in any one of claims 1-6.

Images