Multi-dimensional cooperative partition and communication optimization distributed training method for heterogeneous cluster

Abstract

The application relates to the technical field of deep learning models, and in particular provides a multi-dimensional cooperative partition and communication optimization distributed training method for a heterogeneous cluster. The method comprises the following steps: analyzing a heterogeneous device and a model through an analysis module to provide input data for optimization solving; utilizing a three-dimensional cooperative optimization partition algorithm to incorporate the selection of an activation value recalculation strategy into the process of partition optimization, and converting a video memory constraint into a soft constraint of calculation cost relaxation; and performing topology-aware adaptive communication, dynamically switching a communication transmission mode based on a partitioned link attribute label, so that the throughput and stability of large model distributed training are improved.

Description

Technical Field

[0001] This invention relates to the field of deep learning model technology, and in particular to a distributed training method for multi-dimensional collaborative partitioning and communication optimization for heterogeneous clusters. Background Technology

[0002] As the parameter size of deep learning models (especially large language model LLMs such as the GPT series, LLaMA, and PaLM) grows exponentially from hundreds of millions to trillions, the memory capacity and computing power of a single GPU accelerator card can no longer meet the loading and training requirements of the complete model. Distributed pipeline parallel technology, by splitting the model layer by layer across multiple devices to achieve multi-device collaborative training, has become the mainstream solution to this problem. However, in real-world industrial and research environments, heterogeneous clusters (heterogeneous computing, memory, and communication) are more common, and traditional pipeline parallel technology is difficult to adapt to such complex scenarios, exposing many key shortcomings:

[0003] (1) The model partitioning lacks comprehensive collaborative optimization. Existing partitioning algorithms (such as GPipe and PipeDream) are mostly based on the assumption of cluster isomorphism and only focus on the optimization of a single resource dimension (such as computing load balancing or meeting memory constraints). Even if some solutions (such as HetPipe) consider the heterogeneous characteristics of computing, they do not simultaneously take into account the correlation between communication bandwidth differences and memory constraints. For example, allocating model layers only according to computing power may lead to peak activation values ​​of low computing power devices (usually accompanied by small memory capacity) causing memory overflow; partitioning only with memory usage as the target may cause high computing power devices to be idle; when ignoring communication topology differences, it is easy to map adjacent stages with high communication volume to low bandwidth links, forming communication bottlenecks and ultimately offsetting the benefits of computing optimization.

[0004] (2) Insufficient elastic optimization capability for memory constraints; traditional solutions treat physical memory as an insurmountable hard constraint, and once the predicted memory usage exceeds the device's limit during the partitioning stage, the partitioning scheme is directly discarded. Although memory pressure can be alleviated by manually enabling recomputation mechanisms, existing technologies lack a collaborative optimization design between recomputation and model partitioning; some solutions indiscriminately enable full-layer recomputation, which leads to a surge in computational overhead and a decrease in training throughput. In essence, existing technologies do not incorporate recomputation as a trade-off elastic cost into the partitioning optimization framework, and cannot achieve a dynamic balance between memory usage and computational efficiency, which greatly limits the search space for the globally optimal partitioning scheme.

[0005] (3) The runtime communication strategy is disconnected from the offline partitioning; communication transmission efficiency is one of the core performance bottlenecks in training large models in heterogeneous clusters, but existing communication transmission strategies generally have adaptability defects: when using full-link compressed transmission, although it can reduce the communication latency of low-speed links such as cross-node Ethernet, it will generate redundant compression or decompression computation overhead on high-speed links such as NVLink / InfiniBand, resulting in a waste of GPU computing power, and even offsetting the compression benefits due to resource competition between computing and communication; when using full-link direct transmission, although it can make full use of the bandwidth advantage of high-speed links, it will cause a surge in communication latency due to the direct transmission of massive tensors in low-bandwidth heterogeneous interconnection scenarios, resulting in blockage of the training pipeline. More importantly, the existing offline partitioning process does not take into account the characteristics of communication links and the adaptation requirements of transmission strategies into the decision-making basis, resulting in the layer-device mapping scheme being unable to provide guidance for runtime communication optimization, and ultimately the load balancing benefits of the partitioning strategy are completely offset by the communication bottleneck. Summary of the Invention

[0006] In view of this, the present invention provides a multi-dimensional collaborative partitioning and communication optimization distributed training method for heterogeneous clusters, in order to improve the throughput and stability of large model distributed training.

[0007] In a first aspect, the present invention provides a distributed training method for multi-dimensional collaborative partitioning and communication optimization for heterogeneous clusters, the method comprising:

[0008] Step 1: Analyze the heterogeneous devices and models using the analysis module to provide input data for optimization solutions;

[0009] Step 2: Based on Step 1, the selection of activation value recalculation strategy is incorporated into the partitioning optimization process using the three-dimensional collaborative optimization partitioning algorithm, transforming the memory constraint into a soft constraint with relaxed computational cost.

[0010] Step 3: Based on Step 2, perform topology-aware adaptive communication and dynamically switch communication transmission modes based on the divided link attribute tags.

[0011] Optionally, step 1 includes:

[0012] Suppose the deep neural network model to be trained contains N layers, denoted as L = {L1, L2, ..., L...} N The heterogeneous computing device set is D={d1,d2,…,d}. M Each device has different computing power and video memory capacity;

[0013] Step 11: Computational Device Performance Profiling: By running standardized computational kernel functions on each heterogeneous device and measuring their actual execution time, the computational time coefficients of each device are fitted; simultaneously, the maximum available video memory capacity of each device is detected through video memory allocation tests. ;

[0014] Step 12, Communication Topology Detection: By performing point-to-point P2P communication tests, measure the actual transmission rate between any two devices in the cluster and construct the communication bandwidth matrix B;

[0015] Step 13, Model-level Feature Extraction: Perform static analysis or trial runs on the model to be trained, and analyze the activation values, parameter values, gradient values, and forward and backward computation times of each layer of the model.

[0016] Optionally, step 2 includes:

[0017] Step 21: Optimize objective and constraint modeling. The objective is to obtain the optimal model partitioning scheme to minimize pipeline bottleneck time. Defined as the maximum execution time across all stages:

[0018] ;

[0019] Stage execution time modeling, execution time of the kth stage. It consists of computation time and communication time:

[0020] ;

[0021] (1) The computation time includes forward propagation, backward propagation, and recomputation overhead:

[0022] ;

[0023] in, Indicates equipment The time required to complete a single floating-point operation; , These represent the computational costs of forward propagation and backward propagation, respectively; recalculation of decision variables. ∈{0,1}, when =0 indicates a reserved layer. The activation value, when =1 indicates that the activation value of the current layer is recalculated; the additional computational cost of recalculation is defined as: ;

[0024] (2) The communication time between stage k and stage k+1 depends on the size of the transmitted activation value and the link bandwidth:

[0025] ;

[0026] in, This represents the size of the output tensor of the last layer in stage k. This indicates the physical device number mapped to logical stage k; for the last stage K, no further transmission is needed. ;

[0027] (3) Memory constraints:

[0028] ;

[0029] in, ;

[0030] When the fixed video memory plus the original activated video memory value is greater than the device capacity At that time, by setting =1 to release video memory;

[0031] Step 22: Design a recomputation efficiency model and dynamically select the optimal recomputation layer:

[0032] The recomputation efficiency is defined as follows: For layer i, the recomputation efficiency is:

[0033] ;

[0034] in, The larger the value, the higher the cost-effectiveness of enabling the current layer's weight calculation;

[0035] The recomputation decision algorithm defines a recomputation efficiency index: the recomputation efficiency of the current layer is equal to the size of its activation value memory divided by its forward propagation computation. The higher the recomputation efficiency, the higher the cost-effectiveness of enabling recomputation in the current layer.

[0036] Based on the recomputation efficiency metric, the workflow is as follows: First, the recomputation efficiency of each layer during the computation phase is calculated and sorted in descending order; then, the total video memory requirement is calculated without enabling any recomputation; if the total video memory requirement is greater than the device capacity, recomputation of each layer is enabled one by one in descending order of recomputation efficiency, and the total video memory requirement is reduced accordingly for each layer enabled; the above process is repeated until the total video memory requirement meets the device capacity constraint, or recomputation of all layers is enabled; if the video memory requirement is still greater than the device capacity after enabling recomputation of all layers, the current partitioning scheme is determined to be infeasible; otherwise, the greedy algorithm outputs the optimal recomputation strategy that satisfies the video memory constraint.

[0037] Step 23: Dynamic programming model partitioning. Use dynamic programming algorithm to search for the globally optimal partitioning scheme and define the state and transition equations.

[0038] Optionally, defining the state and transition equations in step 23 includes:

[0039] a. State definition: DP[i][j] represents the minimum pipeline bottleneck time for dividing the first i layers of the model into j logical stages;

[0040] b. Boundary conditions: DP[0][0]=0 indicates that the 0-layer model is divided into 0 stages and the bottleneck time is 0.

[0041] DP[i][0]= +∞ means that when i>0, it is not feasible to divide the non-zero layer model into 0 stages;

[0042] DP[0][j]= +∞ means that when j>0, it is not feasible to divide the 0-layer model into j stages;

[0043] c. State transition equation: DP[i][j] = min { max (DP[u-1][j-1], Tstage (u,i,m)) |j ≤ u ≤ i, m is the target device of stage j};

[0044] Where u is the starting layer number of stage j; i is the ending layer number of stage j; and the set of layers contained in stage j is S. j ={u, u+1, ..., i}; Tstage (u,i,m) is the execution time when stage j is mapped to device m; iterate through all u and target device m, and select the partitioning scheme that minimizes max (DP [u-1][j-1], Tstage (u,i,m));

[0045] d. Output the optimal solution: Iterate through j from 1 to the maximum number of devices M, and find the minimum value of DP[N][j]. The corresponding partitioning scheme is the optimal one, where N is the minimum number of layers in the model. Output:

[0046] I. Layer sets S1, S2, ..., S of each logical stage K ;

[0047] II. Target device mapping at each stage: m1, m2, ..., m K ;

[0048] III. Link attribute marking between adjacent stages: Based on the target device mapping suggestion, query the bandwidth matrix B. If B[m k ][m k+1 If the value is greater than or equal to the preset high bandwidth threshold, it is marked as high bandwidth and pass-through; otherwise, it is marked as low bandwidth and requires compression.

[0049] Optionally, step 3 includes:

[0050] Step 31: Intercepting communication requests and determining link attributes. Monitor all tensor transmission requests during the forward and backward propagation of the model in real time, and obtain the link attribute tag corresponding to the current transmission based on the partitioning results.

[0051] Step 32, Dual-mode adaptive communication transmission, direct transmission mode: When the transmission link is a high-speed link within the node, the GPU's P2P memory copy instruction is called to write to the receiving GPU;

[0052] Differential stream compression transmission mode: When the transmission link is a low-speed link across nodes, the change in activation value of the same sample between different epochs is compressed;

[0053] Step 33: Adopt a storage-compute separation and asynchronous prefetching mechanism:

[0054] IV. State Cache Unloading: Completely unload the historical state cache from the GPU to the CPU;

[0055] V. Asynchronous prefetch pipeline: When the GPU performs non-communication layer computation tasks, it prefetches the historical state required for the next communication stage into the GPU's temporary video memory buffer;

[0056] VI. Use-and-discard management: After communication and computation are completed, the temporary video memory buffer is released immediately, so that the long-term occupation of GPU video memory by the compression module is reduced to zero, which is optimized in conjunction with the elastic video memory mechanism of the partitioning module.

[0057] Optionally, step 32 performs the following operations:

[0058] Historical state initialization: Before training, the sending and receiving ends synchronously initialize the historical state buffer, setting the initial value to a zero tensor.

[0059] Differential calculation and quantization compression at the sending end; historical state retrieval: retrieving the tensor H of the previous round's historical activation value stored in the local historical state cache module. t-1 The current tensor and the current active value tensor A to be transmitted t The dimensions and data types are completely identical;

[0060] Difference tensor generation: for A t With H t-1 Perform element-wise difference operations to obtain the difference tensor. t=A t -H t-1 By utilizing the strong correlation of activation values ​​at adjacent training times, the difference tensor can contain zero and minimum elements.

[0061] Quantization compression: A quantization strategy is adopted to compress floating-point differential tensors. The mapping of t to a low-bit integer bitstream is specifically as follows: the quantization reference value is determined by fitting the distribution of differential tensor elements, and then the element values ​​are nonlinearly mapped to the integer range of the target bit width through logarithmic transformation to generate a low-redundancy bitstream.

[0062] Historical state update: After quantization is complete, update the local historical state cache H t-1 Replace with the current active value tensor A t This is to ensure that the cache state is synchronized with the current training progress;

[0063] Receiver-side inverse quantization decoding and state overlay: Receive low-bit bitstream, restore differential tensor through inverse quantization, overlay with local historical state to reconstruct original activation value tensor, and synchronously update historical state to ensure consistency between sender and receiver states.

[0064] In a second aspect, embodiments of the present invention provide a computer-readable storage medium, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to execute the distributed training method for multi-dimensional collaborative partitioning and communication optimization for heterogeneous clusters in the first aspect or any possible implementation of the first aspect.

[0065] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the distributed training method for multidimensional collaborative partitioning and communication optimization for heterogeneous clusters in the first aspect or any possible implementation of the first aspect.

[0066] The technical solution provided by this invention includes analyzing heterogeneous devices and models through an analysis module to provide input data for optimization; incorporating the selection of activation value recalculation strategy into the partitioning optimization process using a three-dimensional collaborative optimization partitioning algorithm, transforming memory constraints into soft constraints with relaxed computational costs; and performing topology-aware adaptive communication, dynamically switching communication transmission modes based on the partitioned link attribute markings. This method improves the throughput and stability of distributed training of large models. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 A flowchart of a multi-dimensional collaborative partitioning and communication optimization distributed training method for heterogeneous clusters provided in an embodiment of the present invention;

[0069] Figure 2A flowchart of adaptive communication provided in an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention are also intended to include the plural forms unless the context clearly indicates otherwise.

[0073] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0074] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0075] Figure 1 The flowchart of the multi-dimensional collaborative partitioning and communication optimization distributed training method for heterogeneous clusters provided in the embodiments of the present invention is as follows: Figure 1 As shown, the method includes:

[0076] Step 1: Analyze the heterogeneous devices and models using the analysis module to provide input data for optimization solutions.

[0077] In this embodiment of the invention, step 1 includes:

[0078] Suppose the deep neural network model to be trained contains N layers, denoted as L = {L1, L2, ..., L...} NThe heterogeneous computing device set is D={d1,d2,…,d}. M Each device has different computing power and video memory capacity; for example... Figure 1 As shown, N is 8, M is 3, that is, L={L1,L2, L3,L4,L5,L6,L7,L8}, D={d1,d2,d3};

[0079] Step 11: Computational Device Performance Profiling: By running standardized computational kernel functions (such as matrix multiplication GEMM) on each heterogeneous device, the actual execution time is measured, and the computational time coefficient of each device is fitted; simultaneously, the maximum available video memory capacity of each device is detected through video memory allocation tests. ;

[0080] Step 12, Communication Topology Detection: By performing point-to-point P2P communication tests, the actual transmission rate between any two devices in the cluster is measured, and a communication bandwidth matrix B is constructed. The communication bandwidth matrix B can accurately reflect the bandwidth differences of different interconnection technologies such as NVLink, PCIe, and InfiniBand.

[0081] Step 13, Model-level Feature Extraction: Perform static analysis or trial runs on the model to be trained, and analyze the activation values, parameter values, gradient values, and forward and backward computation times of each layer of the model.

[0082] Step 2: Based on Step 1, the selection of activation value recalculation strategy is incorporated into the partitioning optimization process using the three-dimensional collaborative optimization partitioning algorithm, transforming the memory constraint into a soft constraint with relaxed computational cost.

[0083] The innovation of step 2 lies in incorporating the selection of activation value recalculation strategy into the partitioning optimization process, transforming the memory constraint from a traditional hard constraint into a soft constraint that can be relaxed through computational cost. The main determinations are which layers are included in each stage and what specifications of equipment are required for the current stage. With minimizing pipeline bottleneck time as the core objective, a dynamic programming algorithm is used to search for the globally optimal partitioning scheme. Each decision step simultaneously considers the three major constraints of computation, memory, and communication. Specific steps are as follows:

[0084] In this embodiment of the invention, step 2 includes:

[0085] Step 21: Optimize objective and constraint modeling. The objective is to obtain the optimal model partitioning scheme to minimize pipeline bottleneck time. Defined as the maximum execution time across all stages:

[0086] ;

[0087] Stage execution time modeling, execution time of the kth stage. It consists of computation time and communication time:

[0088] ;

[0089] (1) The computation time includes forward propagation, backward propagation, and recomputation overhead:

[0090] ;

[0091] in, Indicates equipment The time required to complete a single floating-point operation; , These represent the computational costs of forward propagation and backward propagation, respectively; recalculation of decision variables. ∈{0,1}, when =0 indicates a reserved layer. The activation value, when =1 indicates that the activation value of the current layer is recalculated; the additional computational cost of recalculation is defined as: ;

[0092] (2) The communication time between stage k and stage k+1 depends on the size of the transmitted activation value and the link bandwidth:

[0093] ;

[0094] in, This represents the size of the output tensor of the last layer in stage k. This indicates the physical device number mapped to logical stage k; for the last stage K, no further transmission is needed. ;

[0095] (3) Memory constraints:

[0096] ;

[0097] in, ;

[0098] When the fixed video memory plus the original activated video memory value is greater than the device capacity At that time, by setting =1 (enable recomputation at a certain level) to release video memory, but will increase computational overhead;

[0099] Step 22: To minimize the additional computational cost while satisfying memory constraints, design a recomputation efficiency model and dynamically select the optimal recomputation layer:

[0100] The recomputation efficiency is defined as follows: For layer i, the recomputation efficiency is:

[0101] ;

[0102] in, The larger the value (in GB / FLOP), the higher the cost-effectiveness of enabling the current layer's computation (more memory savings per unit of computational overhead).

[0103] The recomputation decision algorithm defines a recomputation efficiency index: the recomputation efficiency of the current layer is equal to the size of its activation value memory divided by its forward propagation computation. The higher the recomputation efficiency, the higher the cost-effectiveness of enabling recomputation in the current layer.

[0104] Based on the recomputation efficiency metric, the workflow is as follows: First, the recomputation efficiency of each layer during the computation phase is calculated and sorted in descending order; then, the total video memory requirement is calculated without enabling any recomputation; if the total video memory requirement is greater than the device capacity, recomputation of each layer is enabled one by one in descending order of recomputation efficiency, and the total video memory requirement is reduced accordingly for each layer enabled; the above process is repeated until the total video memory requirement meets the device capacity constraint, or recomputation of all layers is enabled; if the video memory requirement is still greater than the device capacity after enabling recomputation of all layers, the current partitioning scheme is determined to be infeasible; otherwise, the greedy algorithm outputs the optimal recomputation strategy that satisfies the video memory constraint.

[0105] Step 23: Dynamic programming model partitioning. The dynamic programming algorithm is used to search for the globally optimal partitioning scheme. The state and transition equations are defined as follows:

[0106] a. State definition: DP[i][j] represents the minimum pipeline bottleneck time for dividing the first i layers of the model into j logical stages;

[0107] b. Boundary conditions: DP[0][0]=0 indicates that the 0-layer model is divided into 0 stages and the bottleneck time is 0.

[0108] DP[i][0]= +∞ means that when i>0, it is not feasible to divide the non-zero layer model into 0 stages;

[0109] DP[0][j]= +∞ means that when j>0, it is not feasible to divide the 0-layer model into j stages;

[0110] c. State transition equation: DP[i][j] = min { max (DP[u-1][j-1], Tstage (u,i,m)) |j ≤ u ≤ i, m is the target device of stage j};

[0111] Where u is the starting layer number of stage j; i is the ending layer number of stage j; and the set of layers contained in stage j is S. j={u, u+1, ..., i}; Tstage (u,i,m) is the execution time (including computation time and communication time) when stage j (layer ui) is mapped to device m; iterate through all u and target device m, and select the partitioning scheme that minimizes max (DP [u-1][j-1], Tstage (u,i,m));

[0112] d. Output the optimal solution: Iterate through j from 1 to the maximum number of devices M, and find the minimum value of DP[N][j]. The corresponding partitioning scheme is the optimal one, where N is the minimum number of layers in the model. Output:

[0113] I. Layer sets S1, S2, ..., S of each logical stage K ;

[0114] II. Target device mapping at each stage: m1, m2, ..., m K ;

[0115] III. Link attribute marking between adjacent stages: Based on the target device mapping suggestion, query the bandwidth matrix B. If B[m k ][m k+1 If the value is greater than or equal to the preset high bandwidth threshold, it is marked as high bandwidth and pass-through; otherwise, it is marked as low bandwidth and requires compression.

[0116] Step 3: Based on Step 2, perform topology-aware adaptive communication and dynamically switch communication transmission modes based on the divided link attribute tags.

[0117] Based on the segmented link attribute tags, the communication transmission mode is dynamically switched, and either the direct transmission mode or the differential stream compression transmission mode is selected for differentiated transmission. This achieves differentiated transmission with zero latency for high-speed links and high compression for low-speed links. At the same time, a storage-computation separation and asynchronous prefetching mechanism are introduced to optimize the use of video memory through unloading and prefetching.

[0118] In embodiments of the present invention, such as Figure 2 As shown, step 3 includes:

[0119] Step 31: Intercepting communication requests and determining link attributes. Monitor all tensor transmission requests during the forward and backward propagation of the model in real time, and obtain the link attribute tag corresponding to the current transmission based on the partitioning results.

[0120] Step 32, Dual-mode adaptive communication transmission, direct transmission mode: When the transmission link is a high-speed link within the node, the GPU's P2P memory copy instruction is called to write to the receiving GPU, realizing zero-latency data transmission and avoiding unnecessary computing power consumption;

[0121] Differential stream compression transmission mode: When the transmission link is a low-speed link across nodes, it compresses the change in activation value of the same sample between different epochs, and performs the following operations:

[0122] Historical state initialization: Before training, the sending and receiving ends synchronously initialize the historical state buffer, setting the initial value to a zero tensor.

[0123] Differential calculation and quantization compression at the sending end; historical state retrieval: retrieving the tensor H of the previous round's historical activation value stored in the local historical state cache module. t-1 The current tensor and the current active value tensor A to be transmitted t The dimensions and data types are completely identical;

[0124] Difference tensor generation: for A t With H t-1 Perform element-wise difference operations to obtain the difference tensor. t=A t -H t-1 By utilizing the strong correlation of activation values ​​at adjacent training times, the difference tensor can contain zero and minimum elements.

[0125] Quantization compression: A quantization strategy is adopted to compress floating-point differential tensors. The mapping of t to a low-bit integer bitstream is specifically as follows: the quantization reference value is determined by fitting the distribution of differential tensor elements, and then the element values ​​are nonlinearly mapped to the integer range of the target bit width through logarithmic transformation to generate a low-redundancy bitstream.

[0126] Historical state update: After quantization is complete, update the local historical state cache H t-1 Replace with the current active value tensor A t This is to ensure that the cache state is synchronized with the current training progress;

[0127] Receiver inverse quantization decoding and state superposition: Receive low-bit bit stream, restore differential tensor through inverse quantization, superimpose with local historical state to reconstruct original activation value tensor, and synchronously update historical state to ensure consistency between the transmitting and receiving ends;

[0128] Step 33: Adopt a storage-compute separation and asynchronous prefetching mechanism:

[0129] IV. State Cache Unloading: Completely unload the historical state cache from the GPU to the CPU;

[0130] V. Asynchronous prefetch pipeline: When the GPU performs non-communication layer computation tasks (such as recomputation, regular forward or backward computation), it prefetches the historical state required for the next communication stage into the GPU's temporary video memory buffer. This design completely overlaps the data transmission latency with the GPU computation latency, eliminating the idle computing power of the GPU due to waiting for data, and significantly improving the pipeline efficiency of large model training in heterogeneous clusters.

[0131] VI. Use-and-discard management: After communication and computation are completed, the temporary video memory buffer is released immediately, so that the long-term occupation of GPU video memory by the compression module is reduced to zero, which is optimized in conjunction with the elastic video memory mechanism of the partitioning module.

[0132] This invention proposes a pipelined parallelism method that balances 3D resource characteristics with dynamic communication optimization. It designs a full-link optimization framework combining heterogeneous-aware 3D collaborative planning with runtime topology-aware adaptive communication. The analysis module performs performance profiling of the hardware resources (computing power, link bandwidth, and topology) of the heterogeneous cluster. Based on model structural characteristics, a static execution strategy is generated through a 3D collaborative optimization partitioning algorithm, including model partitioning, device mapping, recomputation decisions, and link attribute marking. During online training, the static strategy is executed, and dual-mode adaptive communication transmission is performed based on the link attribute markings. Simultaneously, a storage-computation separation and asynchronous prefetching mechanism are introduced to offload historical states to the CPU, achieving collaborative optimization of communication bandwidth and GPU memory resources. Through deep collaboration, the core pain points of pipelined parallel training in heterogeneous clusters are addressed, fully releasing the resource potential of heterogeneous clusters and improving the throughput and stability of distributed training of large models.

[0133] The technical solution provided by this invention includes analyzing heterogeneous devices and models through an analysis module to provide input data for optimization; incorporating the selection of activation value recalculation strategy into the partitioning optimization process using a three-dimensional collaborative optimization partitioning algorithm, transforming memory constraints into soft constraints with relaxed computational costs; and performing topology-aware adaptive communication, dynamically switching communication transmission modes based on the partitioned link attribute markings. This method improves the throughput and stability of distributed training of large models.

[0134] The various steps in the embodiments of the present invention can be performed by an electronic device. This electronic device includes, but is not limited to, tablet computers, portable PCs, and desktop computers.

[0135] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is running, it controls the electronic device containing the computer-readable storage medium to execute the above-described embodiment of the multi-dimensional collaborative partitioning and communication optimization distributed training method for heterogeneous clusters.

[0136] Figure 3 A schematic diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device 21 includes a processor 211, a memory 212, and a computer program 213 stored in the memory 212 and executable on the processor 211. When the computer program 213 is executed by the processor 211, it implements the distributed training method for multidimensional collaborative partitioning and communication optimization for heterogeneous clusters in the embodiment. To avoid repetition, it will not be described in detail here.

[0137] Electronic device 21 includes, but is not limited to, processor 211 and memory 212. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 21 and does not constitute a limitation on electronic device 21. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0138] The processor 211 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0139] The memory 212 can be an internal storage unit of the electronic device 21, such as a hard disk or RAM of the electronic device 21. The memory 212 can also be an external storage device of the electronic device 21, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or FlashCard equipped on the electronic device 21. Furthermore, the memory 212 can include both internal and external storage units of the electronic device 21. The memory 212 is used to store computer programs and other programs and data required by network devices. The memory 212 can also be used to temporarily store data that has been output or will be output.

[0140] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0141] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed training method for multi-dimensional collaborative partitioning and communication optimization in heterogeneous clusters, characterized in that, The method includes: Step 1: Analyze the heterogeneous devices and models using the analysis module to provide input data for optimization solutions; Step 2: Based on Step 1, the selection of activation value recalculation strategy is incorporated into the partitioning optimization process using the three-dimensional collaborative optimization partitioning algorithm, transforming the memory constraint into a soft constraint with relaxed computational cost. Step 3: Based on Step 2, perform topology-aware adaptive communication and dynamically switch communication transmission modes based on the partitioned link attribute tags. Step 1 includes: Suppose the deep neural network model to be trained contains N layers, denoted as L = {L1, L2, ..., L...} N The heterogeneous computing device set is D={d1,d2,…,d}. M Each device has different computing power and video memory capacity; Step 11: Computational Device Performance Profiling: By running standardized computational kernel functions on each heterogeneous device and measuring their actual execution time, the computational time coefficients of each device are fitted; simultaneously, the maximum available video memory capacity of each device is detected through video memory allocation tests. ; Step 12, Communication Topology Detection: By performing point-to-point P2P communication tests, measure the actual transmission rate between any two devices in the cluster and construct the communication bandwidth matrix B; Step 13, Model-level Feature Extraction: Perform static analysis or trial runs on the model to be trained, and analyze the activation values, parameter values, gradient values, and forward and backward computation times of each layer of the model. Step 2 includes: Step 21: Optimize objective and constraint modeling. The objective is to obtain the optimal model partitioning scheme to minimize pipeline bottleneck time. Defined as the maximum execution time across all stages: ； Stage execution time modeling, execution time of the kth stage. It consists of computation time and communication time: ； (1) The computation time includes forward propagation, backward propagation, and recomputation overhead: ； in, Indicates device The time required to complete a single floating-point operation; , These represent the computational costs of forward propagation and backward propagation, respectively; recalculation of decision variables. ∈{0,1}, when =0 indicates a reserved layer. The activation value, when =1 indicates that the activation value of the current layer is recalculated; the additional computational cost of recalculation is defined as: ; (2) The communication time between stage k and stage k+1 depends on the size of the transmitted activation value and the link bandwidth: ； in, This represents the size of the output tensor of the last layer in stage k. This indicates the physical device number mapped to logical stage k; for the last stage K, no further transmission is needed. ; (3) Memory constraints: ； in, ; When the fixed video memory plus the original activated video memory value is greater than the device capacity At that time, by setting =1 to release video memory; Step 22: Design a recomputation efficiency model and dynamically select the optimal recomputation layer: The recomputation efficiency is defined as follows: For layer i, the recomputation efficiency is: ； in, The larger the value, the higher the cost-effectiveness of enabling the current layer's weight calculation; The recomputation decision algorithm defines a recomputation efficiency index: the recomputation efficiency of the current layer is equal to the size of its activation value memory divided by its forward propagation computation. The higher the recomputation efficiency, the higher the cost-effectiveness of enabling recomputation in the current layer. Based on the recomputation efficiency metric, the workflow is as follows: First, the recomputation efficiency of each layer during the computation phase is calculated and sorted in descending order; then, the total video memory requirement is calculated without enabling any recomputation; if the total video memory requirement is greater than the device capacity, recomputation of each layer is enabled one by one in descending order of recomputation efficiency, and the total video memory requirement is reduced accordingly for each layer enabled; the above process is repeated until the total video memory requirement meets the device capacity constraint, or recomputation of all layers is enabled; if the video memory requirement is still greater than the device capacity after enabling recomputation of all layers, the current partitioning scheme is determined to be infeasible; otherwise, the greedy algorithm outputs the optimal recomputation strategy that satisfies the video memory constraint. Step 23: Dynamic programming model partitioning. Use dynamic programming algorithm to search for the globally optimal partitioning scheme and define the state and transition equations; The definition of the state and transition equations in step 23 includes: a. State definition: DP[i][j] represents the minimum pipeline bottleneck time for dividing the first i layers of the model into j logical stages; b. Boundary conditions: DP[0][0] = 0 indicates that the 0-layer model is divided into 0 stages and the bottleneck time is 0. DP[i][0] = +∞ means that when i>0, it is not feasible to divide the non-zero layer model into 0 stages; DP[0][j] = +∞ means that when j>0, it is not feasible to divide the 0-layer model into j stages; c. State transition equation: DP[i][j] = min { max (DP[u-1][j-1], Tstage (u,i,m)) | j≤ u ≤ i, m is the target device of stage j}; Where u is the starting layer number of stage j; i is the ending layer number of stage j; and the set of layers contained in stage j is S. j = {u,u+1, ..., i}; Tstage (u,i,m) is the execution time when stage j is mapped to device m; iterate through all u and target device m, and select the partition scheme that minimizes max (DP [u-1][j-1], Tstage (u,i,m)); d. Output the optimal solution: Iterate through j from 1 to the maximum number of devices M, and find the minimum value of DP[N][j]. The corresponding partitioning scheme is the optimal one, where N is the minimum number of layers in the model. Output: I. Layer sets S1, S2, ..., S of each logical stage K ; II. Target device mapping at each stage: m1, m2, ..., m K ; III. Link attribute marking between adjacent stages: Based on the target device mapping suggestion, query the bandwidth matrix B. If B[m k ][m k+1 If the bandwidth is greater than or equal to the preset high bandwidth threshold, it is marked as high bandwidth and pass-through; otherwise, it is marked as low bandwidth and requires compression. Step 3 includes: Step 31: Intercepting communication requests and determining link attributes. Monitor all tensor transmission requests during the forward and backward propagation of the model in real time, and obtain the link attribute tag corresponding to the current transmission based on the partitioning results. Step 32, Dual-mode adaptive communication transmission, direct transmission mode: When the transmission link is a high-speed link within the node, the GPU's P2P memory copy instruction is called to write to the receiving GPU; Differential stream compression transmission mode: When the transmission link is a low-speed link across nodes, the change in activation value of the same sample between different epochs is compressed; Step 33: Adopt a storage-compute separation and asynchronous prefetching mechanism: IV. State Cache Unloading: Completely unload the historical state cache from the GPU to the CPU; V. Asynchronous prefetch pipeline: When the GPU performs non-communication layer computation tasks, it prefetches the historical state required for the next communication stage into the GPU's temporary video memory buffer; VI. Use-and-discard management: After communication and computation are completed, the temporary video memory buffer is released immediately, so that the long-term occupation of GPU video memory by the compression module is reduced to zero, which is optimized in conjunction with the elastic video memory mechanism of the partitioning module.

2. The method according to claim 1, characterized in that, Step 32 performs the following operations: Historical state initialization: Before training, the sending and receiving ends synchronously initialize the historical state buffer, setting the initial value to a zero tensor. Differential calculation and quantization compression at the sending end; historical state retrieval: retrieving the tensor H of the previous round's historical activation value stored in the local historical state cache module. t-1 The current tensor and the current active value tensor A to be transmitted t The dimensions and data types are completely identical; Difference tensor generation: for A t With H t-1 Perform element-wise difference operations to obtain the difference tensor. t=A t -H t-1 By utilizing the strong correlation of activation values ​​at adjacent training times, the difference tensor can contain zero and minimum elements. Quantization compression: A quantization strategy is adopted to compress floating-point differential tensors. The mapping of t to a low-bit integer bitstream is specifically as follows: the quantization reference value is determined by fitting the distribution of differential tensor elements, and then the element values ​​are nonlinearly mapped to the integer range of the target bit width through logarithmic transformation to generate a low-redundancy bitstream. Historical state update: After quantization is complete, update the local historical state cache H t-1 Replace with the current active value tensor A t This is to ensure that the cache state is synchronized with the current training progress; Receiver-side inverse quantization decoding and state overlay: Receive low-bit bitstream, restore differential tensor through inverse quantization, overlay with local historical state to reconstruct original activation value tensor, and synchronously update historical state to ensure consistency between sender and receiver states.

3. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to execute the distributed training method for multidimensional collaborative partitioning and communication optimization for heterogeneous clusters as described in any one of claims 1 to 2.

4. An electronic device, characterized in that, include: One or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the device, cause the device to perform the distributed training method for multidimensional collaborative partitioning and communication optimization for heterogeneous clusters as described in any one of claims 1 to 2.

Images