A node networking method and device for model training

Abstract

The present disclosure provides a node networking method and device for model training, wherein the method includes: before performing hybrid parallel training on a large model, determining the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration; if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, allocating consecutive nodes belonging to the same leaf-layer switch to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, continuing to allocate consecutive nodes belonging to the same leaf-layer switch to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the nodes are allocated. This can reduce communication across leaf-layer switches, and the communication between nodes under the same leaf-layer switch has higher affinity and lower latency, thereby improving the overall training performance.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning technology, and in particular to a node networking method and device for model training. Background Art

[0002] In the field of machine learning systems (MLSys), large language models are typically trained using a hybrid parallel training approach, including tensor parallelism (TP), pipeline parallelism (PP), and data parallelism (DP).

[0003] Hybrid parallel training introduces additional communication overhead, and communication performance directly impacts the effectiveness of parallel training. In large-scale distributed training, the network topology of nodes significantly impacts performance. Therefore, selecting appropriate physical nodes and arranging their positions within the hybrid parallel strategy significantly impacts the performance of large model training.

[0004] Related technologies fail to fully consider network affinity when numbering nodes and grouping them in hybrid parallelism. This results in poor communication performance during data-parallel or pipeline-parallel communication due to the long network links between nodes in the communication group, impacting overall training performance. Therefore, selecting appropriate physical nodes and properly arranging their positions within a hybrid parallel strategy significantly impacts the performance of large model training. Summary of the Invention

[0005] The present disclosure provides a node networking method and device for model training, which is used to solve the defect in related technologies that network affinity is not fully considered when numbering nodes and hybrid parallel grouping, resulting in poor communication performance and affecting overall training performance.

[0006] The present disclosure provides a node networking method for model training, which is used in a distributed cluster of a leaf-spine network architecture. The distributed cluster includes leaf switches and spine switches; each spine switch is connected to all leaf switches, and the leaf switches are used to connect nodes of a data parallel group;

[0007] The method comprises:

[0008] Before performing hybrid parallel training on the large model, determine the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration;

[0009] If the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group based on the required number of nodes for each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the nodes are allocated to each data parallel group.

[0010] According to a node networking method for model training provided by the present invention, if the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group based on the required number of nodes for each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, a node is first allocated to each data parallel group in the positive order of the data parallel groups. If the data parallel groups do not meet the node allocation requirements, a node is allocated to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

[0011] According to a node networking method for model training provided by the present disclosure, before performing multiple rounds of node allocation for each data parallel group, the method further includes:

[0012] All nodes are sorted based on their affinity. Nodes under the same leaf-layer switch have the highest affinity.

[0013] The leaf switches are sorted in descending order of the number of nodes of each leaf switch.

[0014] According to a node networking method for model training provided by the present disclosure, after allocating nodes to each data parallel group, the method further includes:

[0015] Set a starting global number for each data parallel group;

[0016] Global numbers are assigned to multiple nodes assigned to the data parallel group in sequence, wherein the global numbers of the nodes are incremented based on the starting global number of the data parallel group.

[0017] According to a node networking method for model training provided by the present disclosure, in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups, specifically including:

[0018] In the positive order of the data parallel groups, for each data parallel group:

[0019] Check the number of remaining consecutive nodes of each leaf layer switch;

[0020] If the number of remaining consecutive nodes of one of the leaf switches is greater than the number of nodes required by the data parallel group, the leaf switch is determined as the target leaf switch, and the required number of consecutive nodes of the target leaf switch is allocated to the data parallel group;

[0021] If the number of remaining continuous nodes of all the leaf switches is not greater than the number of nodes required by the data parallel group, the leaf switch with the largest number of remaining continuous nodes is determined as the target leaf switch, and the remaining number of continuous nodes of the target leaf switch is allocated to the data parallel group.

[0022] According to a node networking method for model training provided by the present disclosure, a node is first assigned to each data parallel group in the positive order of the data parallel groups, and then a node is assigned to each data parallel group in the reverse order of the data parallel groups, specifically including:

[0023] In each round of allocation, first, according to the positive order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group; then, according to the reverse order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group.

[0024] The present disclosure also provides a node networking device for model training, which is used in a distributed cluster of a leaf-spine network architecture, wherein the distributed cluster includes leaf switches and spine switches; each spine switch is connected to all leaf switches, and the leaf switches are used to connect nodes of a data parallel group;

[0025] The device comprises:

[0026] A proportion determination module is used to determine the proportion of data parallel communication overhead and pipeline parallel communication overhead of the large model in each training iteration before hybrid parallel training is performed on the large model;

[0027] The pipeline parallel priority processing module is used to perform multiple rounds of node allocation for each data parallel group if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, based on the required number of nodes for each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the node allocation for each data parallel group is completed.

[0028] The data parallel priority processing module is used to perform multiple rounds of node allocation for each data parallel group if the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, based on the required number of nodes for each data parallel group and the number of nodes of the leaf layer switch: in each round of allocation, a node is first allocated to each data parallel group in the positive order of the data parallel groups. If the data parallel group does not meet the node allocation requirements, a node is continuously allocated to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

[0029] The present disclosure also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a node networking method for model training as described in any one of the above is implemented.

[0030] The present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a node networking method for model training as described in any one of the above.

[0031] The present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements the node networking method for model training as described in any of the above.

[0032] The node networking method and device for model training provided by the present disclosure determine whether to execute the pipeline parallel priority strategy or the data parallel priority strategy based on the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration before hybrid parallel training is performed on the large model. If the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, the pipeline parallel priority strategy is executed. In each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups in each round of allocation until the nodes are allocated to each data parallel group. By giving priority to the pipeline parallel communication overhead and performing node allocation accordingly, communication across leaf-layer switches can be reduced, and communication between nodes under the same leaf-layer switch has higher affinity and lower latency, thereby improving the overall training performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, a brief introduction will be given below to the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0034] Figure 1 This is a schematic diagram of the allocation results of artificial intelligence chips provided by related technologies.

[0035] Figure 2 It is a distributed cluster of the Spine-Leaf network architecture provided by this disclosure.

[0036] Figure 3 It is a flowchart of the node networking method for model training provided by the present invention.

[0037] Figure 4 It is a structural diagram of the node networking device for model training provided by the present invention.

[0038] Figure 5 It is a structural diagram of the electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of this disclosure more clear, the technical solutions of this disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of this disclosure, not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments of this disclosure without creative effort shall fall within the scope of protection of this disclosure.

[0040] First, the terms involved in the embodiments of the present disclosure are explained.

[0041] Tensor Parallelism (TP): Tensor parallelism is a parallelization strategy that splits a single tensor in a model (such as a weight matrix) into multiple small blocks, each of which is processed by a different AI chip.

[0042] Pipeline Parallelism (PP): Pipeline parallelism is a parallelization strategy that assigns different stages of a model (such as different layers or sub-networks) to different AI chips (such as GPU cards) to form a processing pipeline.

[0043] Data Parallelism (DP): Data parallelism is a parallelization strategy that replicates a model across multiple AI chips (such as GPU cards), with each replica processing a different subset of the data.

[0044] Hybrid parallelism: This refers to a method that combines multiple parallelization techniques to accelerate the training process in deep learning and large-scale machine learning model training. These parallelization techniques primarily include tensor parallelism (TP), pipeline parallelism (PP), and data parallelism (DP). By simultaneously applying these techniques, hybrid parallelism aims to maximize resource utilization, improve training efficiency, and reduce training time.

[0045] In related technologies, before conducting distributed training of large models, it is necessary to first determine the global number of each AI chip. The core logic of the global numbering of AI chips lies in how to number the nodes, because the node number determines the global number of the AI ​​chip, which is crucial for communication and coordination between nodes in distributed training. For example, if the node number is N and there are 8 AI chips under each node, then the global number of the AI ​​chip in the node is: node number * 8 + AI chip local number. Among them, the number of AI chips and the local number are usually fixed.

[0046] There are two numbering methods when starting a distributed training task:

[0047] Static numbering: When starting a task on each node, the node number specified for that node is displayed. In this static numbering mode, the number allocation logic is implemented by the management platform. That is, when starting a task on each node, the node number can be explicitly specified, and the central scheduling system or resource manager is responsible for number allocation and management.

[0048] Dynamic numbering: Each node does not specify a node number when launching a task. Instead, the node's agent (torchagent) automatically coordinates with agents on other nodes based on a predefined communication protocol to dynamically determine a unique number for each node. This process ensures efficient and conflict-free number assignment for all nodes, even as the number of nodes changes or new nodes join the cluster.

[0049] Whether static or dynamic, node numbers are typically assigned based on the order in which they are scheduled to execute tasks or based on the lexicographical order of node names. This approach doesn't take into account the actual location of nodes in the network topology and may not optimize inter-node communication efficiency.

[0050] With the node and AI chip numbers assigned, the training framework assigns the AI ​​chips to corresponding hybrid parallel groups. Current mainstream hybrid parallel training frameworks (such as Megatron-LM) typically assign AI chips in the order of pipeline parallelism, data parallelism, and tensor parallelism. After all AI chips are sorted by number, they are first grouped according to the pipeline parallelization (PP) dimension, with the AI ​​chips within each group corresponding to a pipeline parallelization stage (PP stage). The AI ​​chips within each PP group are then grouped according to the data parallelism (DP) dimension. Finally, the AI ​​chips within each group are assigned tensor parallelization group numbers.

[0051] Taking a specific hybrid parallel strategy as an example, assuming the TP8-DP3-PP2 configuration, that is, there are 8 splits in tensor parallelism, 3 copies in data parallelism, and 2 stages in pipeline parallelism. In this case, if each node is equipped with 8 AI chips, the allocation of AI chips will follow the above sorting and grouping logic. The specific allocation results can be referred to Figure 1 , which details how the AI ​​chips on each node are assigned to different parallel groups under this hybrid parallel strategy. This allocation strategy aims to optimize communication efficiency and computing load to improve the performance of large-scale model training.

[0052] Among them, Node: In a distributed system, each node may represent a server or a computing unit, and each node contains 8 AI chips.

[0053] PP0, PP1: Refers to different stages or stage numbers of pipeline parallelism. For example, PP0 and PP1 represent two different execution stages in pipeline parallelism.

[0054] DP0, DP1, DP2: Refers to different groups or replicas of data parallelism. For example, DP0, DP1, and DP2 represent three different data shards in data parallel training, each of which is processed independently on different nodes.

[0055] TP0-7: Refers to different parts or splits of Tensor Parallelism. For example, TP0 to TP7 means that the weight matrix of a model is split into 8 parts, each of which is processed on a different GPU card.

[0056] A limitation of current node numbering methods is that they do not consider network topology when assigning node numbers. This means that although nodes are assigned numbers, these numbers do not reflect the network affinity between them, which can affect communication efficiency in distributed training. In particular, when grouping nodes for pipeline parallelism (PP) and data parallelism (DP), network affinity cannot be guaranteed. Because the nodes themselves are numbered without considering network topology, communication between PPs or DPs can be degraded due to distant network locations.

[0057] The current training framework does not consider network affinity when assigning node numbers and hybrid parallel groupings, which may lead to the following problems:

[0058] 1) During data-parallel communication or pipeline-parallel communication, the long network links between nodes in the communication group lead to poor communication performance, affecting overall training performance.

[0059] 2) Data-parallel training groups experience significant differences in network topology, resulting in some groups training faster than others. Since DPs require regular synchronization, this can lead to a "barrel effect," where the performance of the entire task is determined by the worst DP group. Poor synchronization between DPs can lead to numerous bubbles and low resource utilization.

[0060] 3) Since nodes are numbered according to the scheduling order, which has a certain degree of randomness, the network topology state is inconsistent every time a task is started, which may cause performance fluctuations.

[0061] 4) An unreasonable layout of the network topology between nodes may lead to uneven traffic distribution on network links, causing congestion on individual links, introducing performance issues, and even stability issues.

[0062] As introduced in the previous technical solution, the final position of a GPU card in the hybrid parallel group is determined by two factors: the number of the node where the GPU is located and the strategy of the training framework to allocate GPUs to hybrid parallel groups. In order to reduce the intrusion into the logic of the training framework, the method proposed in the present invention affects the final arrangement of nodes under the hybrid parallel strategy by intervening in the node numbering to achieve better performance. Since it can only affect the numbering of node granularity, the method of the present invention is mainly applicable to scenarios where the entire computing node is used as a unit to participate in parallel computing at the tensor parallel (TP) level. In this scenario, all resources on each node (for example, multiple GPU cards on a node) are used to split different parts of the model to achieve parallel processing at the tensor level. This method makes full use of the computing resources inside the node and improves the efficiency of tensor parallelism.

[0063] Before introducing the method of the embodiment of the present disclosure, the cluster network topology applied by the embodiment of the present disclosure is schematically described. Figure 2 , Figure 2 Figure 2 shows a distributed cluster in the spine-leaf network architecture used in the embodiments of the present disclosure. The distributed cluster includes leaf switches and spine switches. Each spine switch is connected to all leaf switches, and the leaf switches are used to connect nodes in data parallel groups. Nodes represent physical servers that perform computing tasks, and each node is connected to one or more leaf switches.

[0064] Figure 2 The nodes in the network are connected through leaf switches, which means that the communication between nodes under the same leaf switch has higher affinity and lower latency.

[0065] See also Figure 2 The Leaf layer consists of multiple leaf switches, which aggregate traffic from servers and connect directly to the spine switches in the Spine layer. Spine switches interconnect all Leaf switches in a full mesh topology. Servers on the same Leaf switch can communicate directly through the Leaf switch, while servers on different Leaf switches must communicate through three hops (Leaf->Spine->Leaf). Therefore, the efficiency of inter-server communication is affected by network affinity. To achieve network affinity, the platform responsible for managing the computing resources of the distributed cluster must transparently transmit the node's Leaf information to the module responsible for node numbering through environment variables after completing resource scheduling. An example is as follows: '{"leaf": "lA","spine": "sA"}'.

[0066] See also Figure 3 , the method of the embodiment of the present disclosure includes:

[0067] 301. Before performing hybrid parallel training on the large model, determine the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration.

[0068] The execution subject of the method of the embodiment of the present disclosure may be a platform. The platform serves as the management center of the entire distributed cluster resources and tasks, and is responsible for executing a series of operations including resource scheduling, node numbering, and environment variable setting.

[0069] Before training begins, collect data parallel (DP) and pipeline parallel (PP) communication data from large model training iterations through simulation or historical data analysis. Calculate the data volume, communication frequency, and communication latency of DP and PP communication during each training iteration to determine their respective communication costs. Then, based on this collected data, analyze the contribution of DP and PP communication costs to the total communication cost.

[0070] In one embodiment, a threshold may also be set. For example, if the PP communication overhead exceeds 50% of the total communication overhead, it is considered that the PP communication overhead accounts for a large proportion; if the DP communication overhead exceeds the threshold, it is considered that the DP communication overhead accounts for a large proportion.

[0071] Through step 301, it can be decided whether to adopt the DP priority strategy or the PP priority strategy for subsequent node allocation according to the proportion of DP and PP communication overheads.

[0072] 302. If the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group based on the required number of nodes for each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the nodes are allocated to each data parallel group.

[0073] In data-parallel training, the entire training dataset is split into multiple subsets, each of which is processed by one or more nodes (usually GPU cards). Each such group of nodes, that is, a collection of nodes processing the same subset of data, is called a "data-parallel group" or "DP group" for short.

[0074] The purpose of data parallelism is to distribute the training task across multiple nodes, with each node processing a subset of the data and synchronizing gradient updates during training to ensure that the model copies on all nodes remain consistent.

[0075] In the context of node allocation, "data-parallel group positive order" means allocating nodes to each DP group in order, starting with the first DP group, according to the data-parallel group number or order. For example, if there are four DP groups (DP0, DP1, DP2, DP3), positive order allocation would start with DP0, followed by DP1, then DP2, and finally DP3. Conversely, "data-parallel group reverse order" means allocating nodes in the reverse order of positive order. Continuing with the above example, reverse order allocation would start with DP3, followed by DP2, then DP1, and finally DP0.

[0076] If the pipeline parallel communication overhead ratio is greater than the data parallel communication overhead ratio, the pipeline parallel PP priority mode is implemented. The performance of the PP communication group determines the training performance of each DP group. To avoid the "barrel effect" caused by uneven performance between DP groups, the strategy design needs to balance the PP communication group performance of different DP groups.

[0077] In an actual distributed cluster, different leaf switches may have different numbers of nodes connected to them. Therefore, not all leaf switches have the same number of nodes.

[0078] Before assigning nodes to a DP group, all nodes must be sorted by affinity, then by leaf node, with the number of nodes under each leaf node decreasing. Assume that a leaf-spine network cluster includes three leaf switches and a total of 32 nodes, belonging to switches Leaf A, Leaf B, and Leaf C.

[0079] First, sort all nodes by affinity. Affinity here refers to the communication affinity between nodes on the same leaf switch. Therefore, first group all nodes according to the leaf switch to which they belong. Then, sort the leaf switches from most to least nodes. Based on the conditions, Leaf A has 10 nodes; Leaf B has 10 nodes; and Leaf C has 12 nodes. Therefore, the order after the second-level sorting is Leaf C, Leaf A, and Leaf B.

[0080] Each DP group requires a fixed number of nodes to form. For example, if a DP group requires 8 nodes, and there are 4 DP groups, then a total of 32 nodes are required to meet the node allocation requirements of the DP group.

[0081] When allocating nodes, the goal is to place nodes of the same DP group under the same leaf switch as much as possible to reduce communication across leaf switches. However, due to the uneven distribution of nodes under leaf switches, there may not be enough consecutive nodes to meet the needs of a DP group.

[0082] When a DP group needs to allocate nodes, the algorithm searches for the largest available contiguous node block. If the number of nodes in the block is smaller than the number of nodes currently required by the DP group, the nodes in the block are directly allocated to the DP group. Otherwise, the algorithm attempts to find a block closest to the required number of nodes for the DP group and splits it to ensure that each DP group can be allocated a node. In this embodiment, a "block" refers to a group of contiguous nodes that belong to the same leaf-layer switch and therefore have high communication affinity.

[0083] It should be noted that reverse order allocation is not necessarily performed. Instead, if there are still nodes remaining after forward order allocation, reverse order allocation is performed to ensure balance between DP groups.

[0084] For example, assume there are 4 data parallel (DP) groups, and each DP group requires 8 nodes.

[0085] The leaf-spine network cluster includes three leaf switches, each with the following number of nodes: Leaf A: 10 nodes; Leaf B: 10 nodes; Leaf C: 12 nodes. This totals 32 nodes, meeting the requirement of 8 nodes per each of the four DP groups. The order after secondary sorting is Leaf C, Leaf A, and Leaf B.

[0086] Positive order allocation:

[0087] DP0 allocation: Starting from Leaf C, DP0 selects 8 nodes (Leaf C has 4 remaining nodes).

[0088] DP1 allocation: Starting from Leaf A, DP1 selects 8 consecutive nodes (there are 2 nodes left in Leaf A).

[0089] DP2 allocation: Starting from Leaf B, DP2 selects 8 consecutive nodes (Leaf B has 2 remaining nodes).

[0090] DP3 allocation: At this point, both Leaf A and Leaf B have only 2 nodes left, which are insufficient to allocate to DP3. Therefore, DP3 can only select 4 nodes from Leaf C, which is still 4 nodes short.

[0091] At this step, DP3 has not yet allocated enough nodes, so reverse allocation is needed to complete the node allocation of DP3 and try to balance the node allocation of other DP groups.

[0092] Reverse order allocation:

[0093] At this point, Leaf A and Leaf B each have 2 nodes left.

[0094] DP3 selects 2 nodes from Leaf A and then selects 2 nodes from Leaf B.

[0095] Final allocation results:

[0096] DP0: 8 nodes from Leaf C.

[0097] DP1: 8 nodes from Leaf A.

[0098] DP2: 8 nodes from Leaf B.

[0099] DP3: All four nodes are on Leaf C, two nodes are from Leaf B, and two nodes are from Leaf A.

[0100] The advantages of the pipeline parallelism (PP) priority strategy include:

[0101] Affinity Optimization: Before node allocation, all nodes are sorted based on their inter-node affinity. Nodes on the same leaf switch have the highest affinity, so nodes in the same pipeline stage are prioritized for placement on the same leaf switch. During node allocation, ensuring that nodes in each pipeline stage are assigned to the same leaf switch whenever possible reduces communication latency and bandwidth consumption between pipeline stages. This is because nodes on the same leaf switch have higher affinity and lower latency, thus optimizing the PP communication group.

[0102] Balanced performance: This allocation strategy helps to balance the performance among different DP groups because each DP group is assigned to nodes with high communication affinity as much as possible.

[0103] Resource utilization: This approach allows for more efficient use of cluster resources, avoiding resource waste due to uneven node allocation.

[0104] Furthermore, the method further comprises:

[0105] If the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group based on the required number of nodes for each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, a node is first allocated to each data parallel group in the positive order of the data parallel groups. If the data parallel group does not meet the node allocation requirements, a node is allocated to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

[0106] If the data parallel communication overhead accounts for a large proportion, the data parallel DP priority mode is executed.

[0107] Assume there are four data parallel (DP) groups (DP0, DP1, DP2, and DP3), each requiring three nodes. The platform allocates 12 nodes for this task, distributed across three leaf switches, each with four nodes (Leaf A: 4 nodes; Leaf B: 4 nodes; Leaf C: 4 nodes). After secondary sorting, the order is Leaf A, Leaf B, and Leaf C.

[0108] The allocation process is as follows:

[0109] First round of allocation (in positive order):

[0110] DP0 selects a node from Leaf A.

[0111] DP1 selects a node from Leaf A.

[0112] DP2 selects a node from Leaf A.

[0113] DP3 selects a node from Leaf A.

[0114] At this point, Leaf A has 0 nodes remaining, Leaf B has 4 nodes remaining, and Leaf C has 4 nodes remaining.

[0115] First round of allocation (in reverse order):

[0116] DP3 selects a node from Leaf B.

[0117] DP2 selects a node from Leaf B.

[0118] DP1 selects a node from Leaf B.

[0119] DP0 selects a node from LeafB.

[0120] At this point, Leaf A has 0 nodes remaining, Leaf B has 0 nodes remaining, and Leaf C has 4 nodes remaining.

[0121] Second round of allocation (in positive order):

[0122] DP0 selects a node from Leaf C.

[0123] DP1 selects a node from Leaf C.

[0124] DP2 selects a node from Leaf C.

[0125] DP3 selects a node from Leaf C.

[0126] Final allocation results:

[0127] DP0: One node is on Leaf A, one node is on Leaf B, and one node is on Leaf C.

[0128] DP1: One node is on Leaf A, one node is on Leaf B, and one node is on Leaf C.

[0129] DP2: 1 node on Leaf A, 1 node on Leaf B, and 1 node on Leaf C.

[0130] DP3: 1 node on Leaf A, 1 node on Leaf B, and 1 node on Leaf C.

[0131] This round-robin node allocation method ensures that each DP group obtains nodes from different leaf switches, reducing the need for cross-leaf switch communication. In pipeline parallelism, communication between different stages (PP stages) typically requires traversing multiple nodes. If these nodes are distributed across different leaf switches, communication efficiency is affected. The DP priority policy can reduce this cross-leaf switch communication, thereby indirectly optimizing the performance of the PP communication group.

[0132] The DP priority strategy ensures that each DP group receives a relatively balanced amount of resources by allocating nodes in a round-robin manner. This balanced resource allocation helps reduce the situation where a specific DP group becomes a bottleneck due to insufficient resources. This is also beneficial for the PP communication group, as the performance of the PP phase is often affected by the slowest DP group.

[0133] The method of the embodiment of the present disclosure can select different priority strategies for different overhead ratios, and reasonably arrange the position of each node under the hybrid parallel strategy, thereby improving the throughput of the entire distributed training, thereby improving the performance of large model training.

[0134] In summary, for the PP priority strategy:

[0135] Secondary sorting: All nodes are sorted by affinity based on the affinity between nodes. Nodes under the same leaf-layer switch have the highest affinity. Leaf-layer switches are sorted in descending order of the number of nodes per leaf-layer switch.

[0136] Assign nodes to DP groups: In the positive order of the data parallel groups, for each data parallel group: check the number of remaining continuous nodes of each leaf-layer switch; if the number of remaining continuous nodes of one of the leaf-layer switches is greater than the required number of nodes of the data parallel group, then determine the leaf-layer switch as the target leaf-layer switch, and assign the required number of nodes of the target leaf-layer switch to the data parallel group; if the number of remaining continuous nodes of all leaf-layer switches is not greater than the required number of nodes of the data parallel group, then determine the leaf-layer switch with the largest number of remaining continuous nodes as the target leaf-layer switch, and assign the remaining number of continuous nodes of the target leaf-layer switch to the data parallel group.

[0137] If the data parallel group still fails to meet the node allocation requirements after the forward allocation is completed, the data parallel group will be allocated in reverse order. The reverse allocation process is similar to the forward allocation process and will not be described here.

[0138] Node numbering: Each data parallel group is assigned a starting global number. The starting global numbers of adjacent data parallel groups differ by dp_num (the number of nodes in each DP group). Nodes assigned to a data parallel group are assigned global numbers in order, with the global numbers increasing based on the starting global number of the data parallel group.

[0139] For example, if there are four DP groups and each DP group requires three nodes, then the nodes of DP0 might be numbered 0, 1, and 2; the nodes of DP1 might be numbered 3, 4, and 5; the nodes of DP2 might be numbered 6, 7, and 8; and the nodes of DP3 might be numbered 9, 10, and 11. Here, the starting numbers of each DP group are 0, 3, 6, and 9, respectively, and the difference between these numbers is dp_num (3 in this example).

[0140] For DP priority strategy:

[0141] Secondary sorting: All nodes are sorted by affinity based on the affinity between nodes. Nodes under the same leaf-layer switch have the highest affinity. Leaf-layer switches are sorted in descending order of the number of nodes per leaf-layer switch.

[0142] Assign nodes to DP groups: In each round of allocation, first select a corresponding target leaf-layer switch for each data parallel group in the positive order of the data parallel groups and the order of the leaf-layer switches, and select a node from the selected target leaf-layer switches to assign to the data parallel group; then select a corresponding target leaf-layer switch for each data parallel group in the reverse order of the data parallel groups and the order of the leaf-layer switches, and select a node from the selected target leaf-layer switches to assign to the data parallel group, until all nodes are allocated to each data parallel group.

[0143] Node numbering: Each data parallel group is assigned a starting global number. The starting global numbers of adjacent data parallel groups differ by dp_num (the number of nodes in each DP group). Nodes assigned to a data parallel group are assigned global numbers in order, with the global numbers increasing based on the starting global number of the data parallel group.

[0144] For example, if there are four DP groups and each DP group requires three nodes, then the nodes of DP0 might be numbered 0, 1, and 2; the nodes of DP1 might be numbered 3, 4, and 5; the nodes of DP2 might be numbered 6, 7, and 8; and the nodes of DP3 might be numbered 9, 10, and 11. Here, the starting numbers of each DP group are 0, 3, 6, and 9, respectively, and the difference between these numbers is dp_num (3 in this example).

[0145] The purpose of secondary sorting is to ensure that nodes are assigned to the same leaf switch as much as possible. This reduces cross-leaf switch communication, thereby optimizing communication affinity, reducing communication latency, and improving bandwidth utilization. Furthermore, secondary sorting ensures that all DP groups receive resources fairly, preventing performance limitations for some DP groups due to uneven resource allocation.

[0146] The purpose of node numbering is to align with the way the training framework assigns nodes to hybrid parallel groups, ensuring that the framework can correctly identify the DP group or PP stage to which each node belongs. Node numbering after assigning nodes to DP groups ensures that nodes in the same DP group or PP stage are correctly grouped together, reducing cross-group communication.

[0147] The difference between PP priority strategy and DP priority strategy:

[0148] The DP-first strategy focuses on optimizing communication between data-parallel (DP) groups. This is because in data-parallel training, each DP group frequently performs gradient aggregation and parameter updates, operations that significantly impact overall training performance. DP-first prioritizes communication between data-parallel groups by allocating resources node by node and alternating between forward and reverse order to ensure resource balance and communication affinity.

[0149] The primary goal of the PP-first strategy is to optimize communication affinity between pipeline parallel (PP) stages. This means minimizing communication latency and bandwidth consumption between pipeline stages. PP-first prioritizes communication between pipeline parallel stages by assigning consecutive nodes within each DP group to the same leaf switch, thereby optimizing communication efficiency within the PP stage. This ensures that nodes within each pipeline parallel (PP) stage are assigned to the same leaf switch whenever possible, thereby reducing communication latency and bandwidth consumption between PP stages.

[0150] The method of the embodiment of the present disclosure can achieve the following technical effects:

[0151] 1) By intervening in the node numbering, the nodes are arranged reasonably in the hybrid parallel grouping. By using the affinity between nodes, the communication efficiency of the hybrid parallel can be effectively improved.

[0152] 2) You can combine parameter configuration and training strategy to select a PP-first or DP-first affinity strategy to improve the throughput of the entire distributed training.

[0153] 3) Reduce task randomness. After affinity networking, the network topology of each communication group is relatively stable, avoiding performance fluctuations caused by changes in network topology.

[0154] 4) The reasonable layout of the communication group network topology is conducive to the traffic balance of the entire communication link, which can effectively avoid problems such as link congestion and improve the stability of the system.

[0155] The node networking device for model training provided by the present disclosure is described below. The node networking device for model training described below and the node networking method for model training described above can be referenced to each other.

[0156] This disclosure provides a node networking device for model training, see Figure 4 , including: a proportion determining module 401, used to determine the proportion of data parallel communication overhead and pipeline parallel communication overhead of the large model in each training iteration before performing hybrid parallel training on the large model;

[0157] The pipeline parallel priority processing module 402 is used to perform multiple rounds of node allocation for each data parallel group based on the required number of nodes for each data parallel group and the number of nodes of the leaf-layer switch if the pipeline parallel communication overhead accounts for a large proportion: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the nodes are allocated to each data parallel group.

[0158] Optionally, the device also includes: a data parallel priority processing module, which is used to perform multiple rounds of node allocation for each data parallel group if the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, based on the required number of nodes for each data parallel group and the number of nodes of the leaf layer switch: in each round of allocation, first allocate a node to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements, continue to allocate a node to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

[0159] Optionally, the pipeline parallel priority processing module 402 is specifically configured to:

[0160] In the positive order of the data parallel groups, for each data parallel group:

[0161] Check the number of remaining consecutive nodes of each leaf layer switch;

[0162] If the number of remaining consecutive nodes of one of the leaf switches is greater than the number of nodes required by the data parallel group, the leaf switch is determined as the target leaf switch, and the required number of consecutive nodes of the target leaf switch is allocated to the data parallel group;

[0163] If the number of remaining continuous nodes of all the leaf switches is not greater than the number of nodes required by the data parallel group, the leaf switch with the largest number of remaining continuous nodes is determined as the target leaf switch, and the remaining number of continuous nodes of the target leaf switch is allocated to the data parallel group.

[0164] Optionally, the data parallel priority processing module is specifically configured to:

[0165] In each round of allocation, first, according to the positive order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group; then, according to the reverse order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group.

[0166] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the node networking method for model training, which includes: before performing hybrid parallel training on the large model, determining the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration; if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, performing multiple rounds of node allocation for each data parallel group based on the required number of nodes of each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then continue to allocate at least one continuous node belonging to the same leaf-layer switch to each data parallel group containing the remaining nodes in each round of allocation in the reverse order of the data parallel groups until the nodes are allocated to each data parallel group.

[0167] In addition, the logical instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program code.

[0168] On the other hand, the present disclosure also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the node networking method for model training provided by the above methods. The method includes: before performing hybrid parallel training on the large model, determining the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration; if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, performing multiple rounds of node allocation for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then continue to allocate at least one continuous node belonging to the same leaf-layer switch to each data parallel group containing the remaining nodes in each round of allocation in the reverse order of the data parallel groups until the nodes are allocated to each data parallel group.

[0169] On the other hand, the present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a node networking method for executing the model training provided by the above-mentioned methods, the method comprising: before performing hybrid parallel training on a large model, determining the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration; if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, performing multiple rounds of node allocation for each data parallel group based on the required number of nodes of each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then continuing to allocate at least one continuous node belonging to the same leaf-layer switch to each data parallel group containing the remaining nodes in each round of allocation in the reverse order of the data parallel groups, until the nodes are allocated to each data parallel group.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0171] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

1. A node networking method for model training, characterized in that: A distributed cluster for a leaf-spine network architecture, wherein the distributed cluster includes leaf switches and spine switches; each spine switch is connected to all leaf switches, and the leaf switches are used to connect nodes of a data parallel group; The method comprises: Before performing hybrid parallel training on the large model, determine the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration; If the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group based on the required number of nodes for each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the nodes are allocated to each data parallel group.

2. The node networking method for model training according to claim 1, characterized in that: The method also includes: if the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group based on the required number of nodes for each data parallel group and the number of nodes of the leaf layer switch: in each round of allocation, a node is first allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel groups do not meet the node allocation requirements, a node is continuously allocated to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

3. The node networking method for model training according to claim 1 or 2, characterized in that: Before performing multiple rounds of node allocation for each data parallel group, the method further includes: All nodes are sorted based on their affinity. Nodes under the same leaf-layer switch have the highest affinity. The leaf switches are sorted in descending order of the number of nodes of each leaf switch.

4. The node networking method for model training according to claim 1 or 2, characterized in that: After allocating nodes to each data parallel group, the method further includes: Set a starting global number for each data parallel group; Global numbers are assigned to multiple nodes assigned to the data parallel group in sequence, wherein the global numbers of the nodes are incremented based on the starting global number of the data parallel group.

5. The node networking method for model training according to claim 1, characterized in that: In each round of allocation, at least one consecutive node belonging to the same leaf switch is allocated to each data parallel group in the positive order of the data parallel groups, specifically including: In the positive order of the data parallel groups, for each data parallel group: Check the number of remaining consecutive nodes of each leaf layer switch; If the number of remaining consecutive nodes of one of the leaf switches is greater than the number of nodes required by the data parallel group, the leaf switch is determined as the target leaf switch, and the required number of consecutive nodes of the target leaf switch is allocated to the data parallel group; If the number of remaining continuous nodes of all the leaf switches is not greater than the number of nodes required by the data parallel group, the leaf switch with the largest number of remaining continuous nodes is determined as the target leaf switch, and the remaining number of continuous nodes of the target leaf switch is allocated to the data parallel group.

6. The node networking method for model training according to claim 2, characterized in that: First, assign a node to each data parallel group in the positive order of the data parallel groups, and then assign a node to each data parallel group in the reverse order of the data parallel groups. Specifically, In each round of allocation, first, according to the positive order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group; then, according to the reverse order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group.

7. A node networking device for model training, characterized in that: A distributed cluster for a leaf-spine network architecture, wherein the distributed cluster includes leaf switches and spine switches; each spine switch is connected to all leaf switches, and the leaf switches are used to connect nodes of a data parallel group; The device comprises: A proportion determination module is used to determine the proportion of data parallel communication overhead and pipeline parallel communication overhead of the large model in each training iteration before hybrid parallel training is performed on the large model; The pipeline parallel priority processing module is used to perform multiple rounds of node allocation for each data parallel group if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, based on the required number of nodes for each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one consecutive node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the node allocation for each data parallel group is completed.

8. The node networking device for model training according to claim 7, characterized in that: The device further comprises: The data parallel priority processing module is used to perform multiple rounds of node allocation for each data parallel group if the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, based on the required number of nodes for each data parallel group and the number of nodes of the leaf layer switch: in each round of allocation, a node is first allocated to each data parallel group in the positive order of the data parallel groups. If the data parallel group does not meet the node allocation requirements, a node is continuously allocated to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the node networking method for model training according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the node networking method for model training according to any one of claims 1 to 6 is implemented.

Images