Node networking method and device for model training

Abstract

The invention provides a node networking method and device for model training, and the method comprises the steps: determining the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of a large model in each training iteration before the mixed parallel training of the large model; if the pipeline parallel communication overhead proportion is greater than the data parallel communication overhead proportion, sequentially allocating continuous nodes belonging to the same leaf layer switch to each data parallel group according to the positive sequence of the data parallel groups; and if the data parallel group does not meet the node distribution requirement after the positive sequence distribution is finished, continuously distributing the continuous nodes belonging to the same leaf-layer switch to each data parallel group containing the residual nodes in sequence according to the reverse sequence of the data parallel group until the node distribution is finished, thereby reducing the communication of the cross-leaf-layer switch, and improving the node distribution efficiency. And communication among the nodes under the same leaf layer switch has higher affinity and lower delay, so that the overall training performance is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine learning, and in particular to a node networking method and device for model training. Background Art

[0002] In the field of machine learning systems (MLSys), the training of large language models usually adopts a hybrid parallel training method, including tensor parallelism (TP), pipeline parallelism (PP) and data parallelism (DP).

[0003] The hybrid parallel training method introduces additional communication overhead, and the communication performance directly affects the effect of parallel training. In large-scale distributed training, the network topology position of the node has a significant impact on the performance. Therefore, how to select appropriate physical nodes and reasonably arrange the positions of each node under the hybrid parallel strategy will greatly affect the performance of large model training.

[0004] In the related art, network affinity is not fully considered when numbering nodes and hybrid parallel grouping. This leads to poor communication performance during data parallel communication or pipeline parallel communication due to the long network link distance between nodes in the communication group, which affects the overall training performance. Therefore, how to select appropriate physical nodes and reasonably arrange the positions of each node under the hybrid parallel strategy will greatly affect the performance of large model training. Summary of the invention

[0005] The present disclosure provides a node networking method and device for model training, which are used to solve the defect in the related art that network affinity is not fully considered during node numbering and mixed parallel grouping, resulting in poor communication performance and affecting the overall training performance.

[0006] The present disclosure provides a node networking method for model training, which is used in a distributed cluster of a leaf-spine network architecture, wherein the distributed cluster includes a leaf layer switch and a spine layer switch; each spine layer switch is respectively connected to all leaf layer switches, and the leaf layer switch is used to connect nodes of a data parallel group; The method comprises: Before performing hybrid parallel training on the large model, determining the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration; If the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group according to the required number of nodes for each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the node allocation for each data parallel group is completed.

[0007] According to a node networking method for model training provided by the present invention, if the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf layer switch: in each round of allocation, a node is first allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel groups do not meet the node allocation requirements, a node is continuously allocated to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

[0008] According to a node networking method for model training provided by the present disclosure, before performing multiple rounds of node allocation for each data parallel group, the method further includes: According to the affinity between nodes, all nodes are sorted by affinity; among them, the nodes under the same leaf layer switch have the highest affinity; The leaf switches are sorted in descending order of the number of nodes of each leaf switch.

[0009] According to a node networking method for model training provided by the present disclosure, after allocating nodes to each data parallel group, the method further includes: Set a starting global number for each data parallel group; Global numbers are assigned to multiple nodes assigned to the data parallel group in sequence, wherein the global numbers of the nodes are incremented based on the starting global number of the data parallel group.

[0010] According to a node networking method for model training provided by the present disclosure, in each round of allocation, at least one continuous node belonging to the same leaf layer switch is allocated to each data parallel group in the positive order of the data parallel groups, specifically including: In the positive order of the data parallel groups, for each data parallel group: Check the number of remaining consecutive nodes of each leaf layer switch; If the number of remaining consecutive nodes of one of the leaf switches is greater than the number of nodes required by the data parallel group, the leaf switch is determined as the target leaf switch, and the required number of consecutive nodes of the target leaf switch is allocated to the data parallel group; If the number of remaining continuous nodes of all the leaf switches is not greater than the number of nodes required by the data parallel group, the leaf switch with the largest number of remaining continuous nodes is determined as the target leaf switch, and the remaining number of continuous nodes of the target leaf switch is allocated to the data parallel group.

[0011] According to a node networking method for model training provided by the present disclosure, a node is first allocated to each data parallel group in the positive order of the data parallel groups, and then a node is allocated to each data parallel group in the reverse order of the data parallel groups, specifically including: In each round of allocation, first, according to the positive order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group; then, according to the reverse order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group.

[0012] The present disclosure also provides a node networking device for model training, which is used in a distributed cluster of a leaf-spine network architecture, wherein the distributed cluster includes a leaf layer switch and a spine layer switch; each spine layer switch is respectively connected to all leaf layer switches, and the leaf layer switch is used to connect nodes of a data parallel group; The device comprises: A proportion determination module, used to determine the proportion of data parallel communication overhead and pipeline parallel communication overhead of the large model in each training iteration before hybrid parallel training is performed on the large model; The pipeline parallel priority processing module is used to perform multiple rounds of node allocation for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf-layer switch if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the node allocation for each data parallel group is completed.

[0013] The data parallel priority processing module is used to perform multiple rounds of node allocation for each data parallel group if the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, based on the required number of nodes for each data parallel group and the number of nodes of the leaf layer switch: in each round of allocation, first allocate a node to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements, continue to allocate a node to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

[0014] The present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a node networking method for model training as described in any one of the above is implemented.

[0015] The present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a node networking method for model training as described in any one of the above.

[0016] The present disclosure also provides a computer program product, including a computer program, which, when executed by a processor, implements a node networking method for model training as described in any one of the above.

[0017] The node networking method and device for model training provided by the present invention determine whether to execute the pipeline parallel priority strategy or the data parallel priority strategy according to the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration before hybrid parallel training is performed on the large model. If the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, the pipeline parallel priority strategy is executed. In each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirement after the positive order allocation is completed, then in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the node allocation for each data parallel group is completed. By giving priority to the pipeline parallel communication overhead and performing node allocation accordingly, communication across leaf-layer switches can be reduced, and communication between nodes under the same leaf-layer switch has higher affinity and lower latency, thereby improving the overall training performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a schematic diagram of the allocation results of the artificial intelligence chip provided by the relevant technology.

[0020] Figure 2 It is a distributed cluster of the leaf-spine network architecture (Spine-Leaf) provided by the present invention.

[0021] Figure 3 It is a flowchart of the node networking method for model training provided by the present invention.

[0022] Figure 4 It is a structural diagram of a node networking device for model training provided by the present invention.

[0023] Figure 5 It is a schematic diagram of the structure of an electronic device provided by the present disclosure. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions in the present disclosure will be clearly and completely described below in conjunction with the drawings in the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0025] First, the terms involved in the embodiments of the present disclosure are explained.

[0026] Tensor Parallelism (TP): Tensor parallelism is a parallelization strategy that splits a single tensor in a model (such as a weight matrix) into multiple small blocks, each of which is processed by a different AI chip.

[0027] Pipeline Parallelism (PP): Pipeline parallelism is a parallelization strategy that assigns different stages of a model (such as different layers or sub-networks) to different AI chips (such as GPU cards) to form a processing pipeline.

[0028] Data Parallelism (DP): Data parallelism is a parallelization strategy that replicates a model on multiple AI chips (such as GPU cards), with each replica processing a different subset of the data.

[0029] Hybrid parallel strategy: refers to a method that combines multiple parallelization technologies to accelerate the training process in deep learning and large-scale machine learning model training. These parallelization technologies mainly include tensor parallelism (TP), pipeline parallelism (PP), and data parallelism (DP). The hybrid parallel strategy aims to maximize resource utilization, improve training efficiency, and reduce training time by applying these strategies simultaneously.

[0030] In the related technology, before conducting large-model distributed training, it is necessary to determine the global number of each artificial intelligence AI chip. The core logic of the global numbering of AI chips lies in how to number the nodes, because the node number determines the global number of the AI ​​chip, which is crucial for communication and coordination between nodes in distributed training. For example, the node number is N, and there are 8 artificial intelligence chips under each node. The global number of the AI ​​chip in the node is: node number * 8 + local number of the AI ​​chip. Among them, the number of AI chips and the local number, these two parameters are usually fixed.

[0031] There are two ways to number a distributed training task: Static numbering: When starting a task on each node, the number of the node is displayed. In the static numbering mode, the number allocation logic is implemented by the management platform, that is, when starting a task on each node, the number of the node can be clearly specified, and the central scheduling system or resource manager is responsible for the allocation and management of the number.

[0032] Dynamic numbering: When each node starts a task, the node number is not specified. The agent process (torchagent) on the node will automatically coordinate with the agent processes on other nodes according to the predefined communication protocol to dynamically determine the unique number of each node. This process ensures that all nodes can be assigned numbers efficiently and without conflict even when the number of nodes changes or new nodes join the cluster.

[0033] Whether static or dynamic, node numbers are usually assigned based on the order in which the nodes are scheduled to execute tasks or based on the lexicographic order of the node names. This approach does not take into account the actual location of the nodes in the network topology, and therefore may not optimize the communication efficiency between nodes.

[0034] After the nodes and AI chips are numbered, the training framework assigns the AI ​​chips to the corresponding hybrid parallel groups. The current mainstream hybrid parallel training frameworks (such as Megatron-LM) are usually assigned in the order of pipeline parallelism, data parallelism, and tensor parallelism. After all AI chips are sorted by number, they are first grouped according to the PP dimension, and the AI ​​chips in each group correspond to a pipeline parallel stage (PP stage); then the AI ​​chips in each PP group are grouped according to the data parallel (DP) dimension; finally, the AI ​​chips in each group are assigned the tensor parallel group number.

[0035] Taking a specific hybrid parallel strategy as an example, assume that the configuration of TP8-DP3-PP2 is adopted, that is, there are 8 splits in tensor parallelism, 3 copies in data parallelism, and 2 stages in pipeline parallelism. In this case, if each node is equipped with 8 AI chips, the allocation of AI chips will follow the above sorting and grouping logic. The specific allocation results can be referred to Figure 1 , which shows in detail how the AI ​​chips of each node are allocated to different parallel groups under this hybrid parallel strategy. Such an allocation strategy is designed to optimize communication efficiency and computing load to improve the performance of large-scale model training.

[0036] Among them, Node: In a distributed system, each node may represent a server or a computing unit, and each node contains 8 AI chips.

[0037] PP0, PP1: Refers to different stages or stage numbers of pipeline parallelism. For example, PP0 and PP1 represent two different execution stages in pipeline parallelism.

[0038] DP0, DP1, DP2: Refers to different groups or copies of data parallelism. For example, DP0, DP1, and DP2 represent three different data shards in data parallel training, and each shard is processed independently on different nodes.

[0039] TP0-7: refers to different parts or splits of Tensor Parallelism. For example, TP0 to TP7 means that the weight matrix of a model is split into 8 parts, each of which is processed on a different GPU card.

[0040] A limitation of current node numbering methods is that they do not consider the network topology when assigning node numbers. This means that although nodes are assigned numbers, these numbers do not reflect the network affinity between nodes, which may affect the communication efficiency in distributed training. In particular, the network affinity of nodes cannot be guaranteed when grouping pipeline parallelism (PP) and data parallelism (DP). Since the nodes themselves are numbered without considering the network topology, communication between PPs or DPs may become poor due to the distant network locations of the nodes.

[0041] The current training framework does not consider the affinity of the network when numbering nodes and mixing parallel groups, which may cause the following problems: 1) During data-parallel communication or pipeline-parallel communication, the long network link distance between nodes in the communication group leads to poor communication performance, which affects the overall training performance.

[0042] 2) The network topology of different groups in data parallel training is very different, which leads to some data parallel groups training fast and some data parallel groups training slowly. Since DPs need to be synchronized regularly, a barrel effect will be formed, and the performance of the entire task is determined by the worst DP group; poor synchronization between DPs will form a large number of bubbles and low resource utilization.

[0043] 3) Since the nodes are numbered in the scheduling order, which has a certain degree of randomness, the network topology state is inconsistent each time a task is started, which may cause performance fluctuations.

[0044] 4) An unreasonable layout of the network topology between nodes may lead to uneven network link traffic, causing congestion on individual links, introducing performance issues, and even stability issues.

[0045] As introduced in the previous technical solution, the final position of a GPU card in the hybrid parallel group is determined by two factors: the number of the node where the GPU is located and the strategy of the training framework to allocate the GPU to the hybrid parallel group. In order to reduce the intrusion into the logic of the training framework, the method proposed in the present disclosure affects the arrangement of the final nodes under the hybrid parallel strategy by intervening in the numbering of nodes to achieve better performance. Since only the numbering of the node granularity can be affected, the method of the present disclosure is mainly applicable to the scenario where the entire computing node is used as a unit to participate in parallel computing at the tensor parallel (TP) level. In this scenario, all resources on each node (for example, multiple GPU cards on a node) are used to split different parts of the model to achieve tensor-level parallel processing. This method makes full use of the computing resources inside the node and improves the efficiency of tensor parallelism.

[0046] Before introducing the method of the embodiment of the present disclosure, the cluster network topology applied by the embodiment of the present disclosure is schematically described. Figure 2 , Figure 2 The figure shows a distributed cluster of the spine-leaf network architecture used in the embodiment of the present disclosure. The distributed cluster includes leaf switches and spine switches; each spine switch is connected to all leaf switches respectively, and the leaf switches are used to connect the nodes of the data parallel group. The nodes represent physical servers that perform computing tasks, and each node is connected to one or more leaf switches.

[0047] Figure 2 The nodes in the network are connected through leaf switches, which means that the communication between nodes under the same leaf switch has higher affinity and lower latency.

[0048] See also Figure 2 , the Leaf layer consists of multiple leaf switches, which aggregate traffic from the servers and are directly connected to the spine switches of the Spine layer. The Spine switches interconnect all Leaf switches in the full mesh topology. Servers under the same Leaf switch can communicate directly through the Leaf switch, and servers under different Leaf switches need to communicate through three hops (Leaf->Spine->Leaf), so the efficiency of communication between servers is affected by network affinity. In order to achieve network affinity, the platform responsible for managing the computing resources of the distributed cluster needs to pass the node's Leaf information to the module responsible for node numbering through environment variables and other means after completing resource scheduling. The example is as follows: '{"leaf": "lA","spine": "sA"}'.

[0049] See also Figure 3 , the method of the embodiment of the present disclosure includes: 301. Before performing hybrid parallel training on the large model, determine the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration.

[0050] The execution subject of the method of the embodiment of the present disclosure may be a platform. The platform serves as the management center of the entire distributed cluster resources and tasks, and is responsible for executing a series of operations including resource scheduling, node numbering, and environment variable setting.

[0051] Before training begins, collect data parallel (DP) and pipeline parallel (PP) communication data in large model training iterations through simulation or historical data analysis. Calculate the data volume, communication frequency, and communication delay of DP and PP communications in each training iteration to determine their respective communication overheads. Then, based on the collected data, analyze the proportion of DP and PP communication overhead in the total communication overhead.

[0052] In one implementation, a threshold may also be set. For example, if the PP communication overhead exceeds 50% of the total communication overhead, it is considered that the PP communication overhead accounts for a large proportion; if the DP communication overhead exceeds the threshold, it is considered that the DP communication overhead accounts for a large proportion.

[0053] Through step 301, it can be decided whether to adopt a DP priority strategy or a PP priority strategy for subsequent node allocation according to the proportion of DP and PP communication overheads.

[0054] 302. If the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the node allocation for each data parallel group is completed.

[0055] In data parallel training, the entire training dataset is split into multiple subsets, each of which is processed by one or more nodes (usually GPU cards). Each such node group, i.e., a collection of nodes processing the same subset of data, is called a "data parallel group" or "DP group" for short.

[0056] The purpose of data parallelism is to distribute the training task across multiple nodes, with each node processing a subset of the data and synchronizing gradient updates during training to ensure that the model copies on all nodes remain consistent.

[0057] In the context of node allocation, "data parallel groups in positive order" means allocating nodes to each DP group in sequence, starting from the first DP group, according to the number or order of the data parallel groups. For example, if there are four DP groups (DP0, DP1, DP2, DP3), positive order allocation will start with DP0, then DP1, then DP2, and finally DP3. In contrast, "data parallel groups in reverse order" means allocating nodes in the reverse order of positive order. Continuing with the above example, reverse order allocation will start with DP3, then DP2, then DP1, and finally DP0.

[0058] If the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, the pipeline parallel PP priority mode is executed. The performance of the PP communication group determines the training performance of each DP group. In order to avoid the barrel effect caused by uneven performance between DP groups, it is necessary to balance the performance of PP communication groups of different DP groups when designing strategies.

[0059] In an actual distributed cluster, different leaf switches may be connected to different numbers of nodes. Therefore, not all leaf switches have the same number of nodes.

[0060] Before assigning nodes to the DP group, all nodes must be sorted in the second level, including: first sorting all nodes by affinity, and then sorting by Leaf, requiring the number of nodes under Leaf to be from most to least. Assume that the leaf-spine network cluster includes 3 leaf switches, with a total of 32 nodes, belonging to switches Leaf A, Leaf B, and Leaf C respectively.

[0061] First, sort all nodes by affinity. Here, affinity refers to the communication affinity between nodes under the same Leaf switch. Therefore, first group all nodes according to the Leaf switch they belong to. Then sort the Leaf switches in order from most to least nodes in each Leaf switch. According to the conditions, Leaf A: 10 nodes; Leaf B: 10 nodes; Leaf C: 12 nodes. Therefore, the order after the secondary sorting is Leaf C, Leaf A, and Leaf B.

[0062] Each DP group requires a fixed number of nodes to form. For example, if a DP group requires 8 nodes, if there are 4 DP groups, then a total of 32 nodes are required to meet the node allocation requirements of the DP group.

[0063] When allocating nodes, the goal is to place nodes of the same DP group under the same leaf switch as much as possible to reduce communication across leaf switches. However, due to the uneven distribution of the number of nodes under the leaf switches, there may not be enough consecutive nodes to meet the needs of a certain DP group.

[0064] When a DP group needs to allocate nodes, the algorithm will look for the largest continuous node block currently available. If the number of nodes in the block is less than the number of nodes remaining required by the current DP group, the nodes in the block will be directly allocated to the DP group. Otherwise, try to find a block closest to the size of the remaining nodes required by the DP group and split it to ensure that each DP group can be allocated a node. In this embodiment, a "block" refers to a group of continuous nodes that belong to the same leaf layer switch, so the communication affinity between them is high.

[0065] It should be noted that reverse order allocation is not necessarily performed, but if there are remaining nodes after forward order allocation, reverse order allocation is performed to ensure balance between DP groups.

[0066] For example, suppose there are 4 data parallel (DP) groups and each DP group requires 8 nodes.

[0067] The leaf-spine network cluster includes three leaf switches, each with the following number of nodes: Leaf A: 10 nodes; Leaf B: 10 nodes; Leaf C: 12 nodes. There are 32 nodes in total, meeting the requirement of 8 nodes for each of the four DP groups. The order after the secondary sorting is Leaf C, Leaf A, and Leaf B.

[0068] Positive order allocation: DP0 allocation: Starting from Leaf C, DP0 selects 8 nodes (Leaf C has 4 remaining nodes).

[0069] DP1 allocation: Starting from Leaf A, DP1 selects 8 consecutive nodes (Leaf A has 2 remaining nodes).

[0070] DP2 allocation: Starting from Leaf B, DP2 selects 8 consecutive nodes (Leaf B has 2 remaining nodes).

[0071] DP3 allocation: At this time, Leaf A and Leaf B only have 2 nodes left, which are not enough to be allocated to DP3, so DP3 can only select 4 nodes from Leaf C, which is still 4 nodes short.

[0072] At this step, DP3 has not yet allocated enough nodes, so reverse allocation is needed to complete the node allocation of DP3 and try to balance the node allocation of other DP groups.

[0073] Reverse allocation: At this point, Leaf A and Leaf B each have 2 nodes left.

[0074] DP3 selects 2 nodes from Leaf A and then selects 2 nodes from Leaf B.

[0075] Final distribution results: DP0: 8 nodes from Leaf C.

[0076] DP1: 8 nodes from Leaf A.

[0077] DP2: 8 nodes come from Leaf B.

[0078] DP3: All 4 nodes are in Leaf C, 2 nodes are from Leaf B, and 2 nodes are from Leaf A.

[0079] The advantages of the pipeline parallelism (PP) priority strategy include: Affinity optimization: Before node allocation, all nodes are first sorted according to the affinity between them. Nodes under the same leaf switch have the highest affinity, so nodes in the same pipeline stage are placed under the same leaf switch first. During node allocation, by ensuring that nodes in each pipeline stage are allocated to the same leaf switch as much as possible, communication latency and bandwidth consumption between pipeline stages can be reduced. This is because communication between nodes under the same leaf switch has higher affinity and lower latency, thereby optimizing the PP communication group.

[0080] Balanced performance: This allocation strategy helps to balance the performance among different DP groups because each DP group is assigned to nodes with high communication affinity as much as possible.

[0081] Resource utilization: In this way, cluster resources can be used more effectively and resource waste caused by uneven node allocation can be avoided.

[0082] Furthermore, the method further comprises: If the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group according to the required number of nodes for each data parallel group and the number of nodes of the leaf layer switch: in each round of allocation, a node is first allocated to each data parallel group in the positive order of the data parallel groups. If the data parallel group does not meet the node allocation requirements, a node is continuously allocated to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

[0083] If the data parallel communication overhead accounts for a large proportion, the data parallel DP priority mode is executed.

[0084] Assume that there are 4 data parallel (DP) groups (DP0, DP1, DP2, DP3), and each DP group requires 3 nodes. The platform allocates 12 nodes for this task, which are distributed under 3 Leaf switches, and each Leaf switch has 4 nodes (Leaf A: 4 nodes; Leaf B: 4 nodes; Leaf C: 4 nodes). The order after the secondary sort is Leaf A, Leaf B, and Leaf C.

[0085] The allocation process is as follows: First round of allocation (in order): DP0 selects a node from Leaf A.

[0086] DP1 selects a node from Leaf A.

[0087] DP2 selects a node from Leaf A.

[0088] DP3 selects a node from Leaf A.

[0089] At this time, Leaf A has 0 nodes remaining, Leaf B has 4 nodes remaining, and Leaf C has 4 nodes remaining.

[0090] First round of allocation (in reverse order): DP3 selects a node from Leaf B.

[0091] DP2 selects a node from Leaf B.

[0092] DP1 selects a node from Leaf B.

[0093] DP0 selects a node from LeafB.

[0094] At this time, Leaf A has 0 nodes remaining, Leaf B has 0 nodes remaining, and Leaf C has 4 nodes remaining.

[0095] Second round of allocation (in order): DP0 selects a node from Leaf C.

[0096] DP1 selects a node from Leaf C.

[0097] DP2 selects a node from Leaf C.

[0098] DP3 selects a node from Leaf C.

[0099] Final distribution results: DP0: 1 node on Leaf A, 1 node on Leaf B, and 1 node on Leaf C.

[0100] DP1: 1 node is on Leaf A, 1 node is on Leaf B, and 1 node is on Leaf C.

[0101] DP2: 1 node on Leaf A, 1 node on Leaf B, and 1 node on Leaf C.

[0102] DP3: 1 node is on Leaf A, 1 node is on Leaf B, and 1 node is on Leaf C.

[0103] This method of allocating nodes in turns reduces the need for cross-Leaf switch communication by ensuring that each DP group can obtain nodes from different Leaf switches. In pipeline parallelism, communication between different stages (PP stages) usually needs to cross multiple nodes. If these nodes are distributed under different Leaf switches, the communication efficiency will be affected. Through the DP priority strategy, this cross-Leaf switch communication can be reduced, thereby indirectly optimizing the performance of the PP communication group.

[0104] The DP priority strategy ensures that each DP group can obtain relatively balanced resources by allocating nodes in turn. This balanced resource allocation helps reduce the situation where a specific DP group becomes a bottleneck due to insufficient resources. This is also beneficial for the PP communication group because the performance of the PP stage is often affected by the slowest DP group.

[0105] The method of the disclosed embodiment can select different priority strategies for different overhead ratios, and reasonably arrange the position of each node under the hybrid parallel strategy, thereby improving the throughput of the entire distributed training, thereby improving the performance of large model training.

[0106] In summary, for the PP priority strategy: Secondary sorting: All nodes are sorted by affinity according to the affinity between nodes; the nodes under the same leaf switch have the highest affinity; the leaf switches are sorted in descending order of the number of nodes in each leaf switch.

[0107] Allocate nodes to DP groups: in the positive order of the data parallel groups, for each data parallel group: check the number of remaining continuous nodes of each leaf-layer switch; if the number of remaining continuous nodes of one of the leaf-layer switches is greater than the required number of nodes of the data parallel group, then determine the leaf-layer switch as the target leaf-layer switch, and allocate the required number of continuous nodes of the target leaf-layer switch to the data parallel group; if the number of remaining continuous nodes of all leaf-layer switches is not greater than the required number of nodes of the data parallel group, then determine the leaf-layer switch with the largest number of remaining continuous nodes as the target leaf-layer switch, and allocate the remaining number of continuous nodes of the target leaf-layer switch to the data parallel group.

[0108] If the data parallel group still fails to meet the node allocation requirements after the forward allocation of the data parallel group is completed, the reverse allocation of the data parallel group is performed. The reverse allocation process is similar to the forward allocation process, and will not be repeated here.

[0109] Node numbering: Set a starting global number for each data parallel group. The starting global numbers of each adjacent data parallel group differ by dp_num (the number of nodes in each DP group). For multiple nodes assigned to the data parallel group, global numbers are assigned in order, where the global numbers of the nodes are incremented based on the starting global number of the data parallel group.

[0110] For example, if there are 4 DP groups and each DP group requires 3 nodes, the node numbers of DP0 may be 0, 1, 2; the node numbers of DP1 are 3, 4, 5; the node numbers of DP2 are 6, 7, 8; and the node numbers of DP3 are 9, 10, 11. Here, the starting numbers of each DP group are 0, 3, 6, 9, and the difference between these numbers is dp_num (3 in this example).

[0111] For DP priority strategy: Secondary sorting: All nodes are sorted by affinity according to the affinity between nodes; the nodes under the same leaf switch have the highest affinity; the leaf switches are sorted in descending order of the number of nodes in each leaf switch.

[0112] Allocate nodes to DP groups: In each round of allocation, first select a corresponding target leaf-layer switch for each data parallel group in the positive order of the data parallel groups and the order of the leaf-layer switches, and select a node from the selected target leaf-layer switch to allocate to the data parallel group; then select a corresponding target leaf-layer switch for each data parallel group in the reverse order of the data parallel groups and the order of the leaf-layer switches, and select a node from the selected target leaf-layer switch to allocate to the data parallel group, until allocating nodes to each data parallel group is completed.

[0113] Node numbering: Set a starting global number for each data parallel group. The starting global numbers of each adjacent data parallel group differ by dp_num (the number of nodes in each DP group). For multiple nodes assigned to the data parallel group, global numbers are assigned in order, where the global numbers of the nodes are incremented based on the starting global number of the data parallel group.

[0114] For example, if there are 4 DP groups and each DP group requires 3 nodes, the node numbers of DP0 may be 0, 1, 2; the node numbers of DP1 are 3, 4, 5; the node numbers of DP2 are 6, 7, 8; and the node numbers of DP3 are 9, 10, 11. Here, the starting numbers of each DP group are 0, 3, 6, 9, and the difference between these numbers is dp_num (3 in this example).

[0115] The purpose of the secondary sorting is to ensure that nodes are assigned to the same Leaf switch as much as possible, which can reduce the communication across Leaf switches, thereby optimizing communication affinity, reducing communication delay and improving bandwidth utilization. In addition, the secondary sorting can ensure that all DP groups can obtain resources fairly, avoiding the performance limitation of some DP groups due to uneven resource allocation.

[0116] The purpose of node numbering is to match the way the training framework assigns nodes to hybrid parallel groups to ensure that the training framework can correctly identify the DP group or PP stage to which each node belongs. Numbering nodes after assigning nodes to DP groups allows nodes in the same DP group or PP stage to be correctly organized together to reduce cross-group communication.

[0117] The difference between PP priority strategy and DP priority strategy: The DP-first strategy focuses on optimizing the communication between data parallel (DP) groups. This is because in data parallel training, each DP group needs to frequently perform gradient aggregation and parameter updates, which have an important impact on the overall training performance. DP-first focuses on optimizing the communication between data parallel groups by allocating nodes one by one and alternating between forward and reverse order to ensure resource balance and communication affinity.

[0118] The main goal of the PP priority strategy is to optimize the communication affinity between pipeline parallel (PP) stages. This means that we need to minimize the communication delay and bandwidth consumption between the various stages of the pipeline. PP priority focuses on optimizing the communication between the parallel stages of the pipeline, by assigning consecutive nodes to the same Leaf switch for each DP group to optimize the communication efficiency of the PP stage. The purpose of this is to ensure that the nodes of each stage (PP stage) in the pipeline parallel are assigned to the same Leaf switch as much as possible, thereby reducing the communication delay and bandwidth consumption between the PP stages.

[0119] The method of the embodiment of the present disclosure can achieve the following technical effects: 1) By intervening in the numbering of nodes, the nodes can be reasonably arranged in the hybrid parallel grouping. Through the affinity between nodes, the communication efficiency of the hybrid parallel can be effectively improved.

[0120] 2) You can combine parameter configuration and training strategy to select PP-first or DP-first affinity strategy to improve the throughput of the entire distributed training.

[0121] 3) Reduce the randomness of tasks. After affinity networking, the network topology of each communication group is relatively stable, avoiding performance fluctuations caused by changes in network topology.

[0122] 4) The reasonable layout of the communication group network topology is conducive to the traffic balance of the entire communication link, which can effectively avoid problems such as link congestion and improve the stability of the system.

[0123] The node networking device for model training provided by the present disclosure is described below. The node networking device for model training described below and the node networking method for model training described above can be referenced to each other.

[0124] The present disclosure provides a node networking device for model training, see Figure 4 , including: a proportion determination module 401, used to determine the proportion of data parallel communication overhead and pipeline parallel communication overhead of the large model in each training iteration before hybrid parallel training is performed on the large model; The pipeline parallel priority processing module 402 is used to perform multiple rounds of node allocation for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf-layer switch if the pipeline parallel communication overhead accounts for a large proportion: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the node allocation for each data parallel group is completed.

[0125] Optionally, the device also includes: a data parallel priority processing module, which is used to perform multiple rounds of node allocation for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf layer switch if the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead: in each round of allocation, first allocate a node to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements, continue to allocate a node to each data parallel group in the reverse order of the data parallel groups until allocating nodes to each data parallel group is completed.

[0126] Optionally, the pipeline parallel priority processing module 402 is specifically used for: In the positive order of the data parallel groups, for each data parallel group: Check the number of remaining consecutive nodes of each leaf layer switch; If the number of remaining consecutive nodes of one of the leaf switches is greater than the number of nodes required by the data parallel group, the leaf switch is determined as the target leaf switch, and the required number of consecutive nodes of the target leaf switch is allocated to the data parallel group; If the number of remaining continuous nodes of all the leaf switches is not greater than the number of nodes required by the data parallel group, the leaf switch with the largest number of remaining continuous nodes is determined as the target leaf switch, and the remaining number of continuous nodes of the target leaf switch is allocated to the data parallel group.

[0127] Optionally, the data parallel priority processing module is specifically used to: In each round of allocation, first, according to the positive order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group; then, according to the reverse order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group.

[0128] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute the node networking method for model training, which includes: before performing hybrid parallel training on the large model, determining the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration; if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, performing multiple rounds of node allocation for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then continue to allocate at least one continuous node belonging to the same leaf-layer switch to each data parallel group containing the remaining nodes in each round of allocation in the reverse order of the data parallel groups until the nodes are allocated to each data parallel group.

[0129] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the relevant technology or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0130] On the other hand, the present disclosure also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the node networking method for model training provided by the above methods, the method including: before performing hybrid parallel training on a large model, determining the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration; if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, performing multiple rounds of node allocation for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then continue to allocate at least one continuous node belonging to the same leaf-layer switch to each data parallel group containing the remaining nodes in each round of allocation in the reverse order of the data parallel groups until the nodes are allocated to each data parallel group.

[0131] On the other hand, the present disclosure also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a node networking method for executing the model training provided by the above-mentioned methods, the method comprising: before performing hybrid parallel training on a large model, determining the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration; if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, performing multiple rounds of node allocation for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirement after the positive order allocation is completed, then in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the nodes are allocated to each data parallel group.

[0132] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0133] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A node networking method for model training, characterized in that: In a distributed cluster for a leaf-spine network architecture, the distributed cluster includes a leaf layer switch and a spine layer switch; each spine layer switch is connected to all leaf layer switches respectively, and the leaf layer switch is used to connect nodes of a data parallel group; The method comprises: Before performing hybrid parallel training on the large model, determining the proportion of data parallel communication overhead and the proportion of pipeline parallel communication overhead of the large model in each training iteration; If the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group according to the required number of nodes for each data parallel group and the number of nodes of the leaf-layer switch: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the node allocation for each data parallel group is completed.

2. The node networking method for model training according to claim 1, characterized in that: The method also includes: if the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, multiple rounds of node allocation are performed for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf layer switch: in each round of allocation, a node is first allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel groups do not meet the node allocation requirements, a node is continuously allocated to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

3. The node networking method for model training according to claim 1 or 2, characterized in that: Before performing multiple rounds of node allocation for each data parallel group, the method further includes: According to the affinity between nodes, all nodes are sorted by affinity; among them, the nodes under the same leaf layer switch have the highest affinity; The leaf switches are sorted in descending order of the number of nodes of each leaf switch.

4. The node networking method for model training according to claim 1 or 2, characterized in that: After allocating nodes to each data parallel group, the method further includes: Set a starting global number for each data parallel group; Global numbers are assigned to multiple nodes assigned to the data parallel group in sequence, wherein the global numbers of the nodes are incremented based on the starting global number of the data parallel group.

5. The node networking method for model training according to claim 1, characterized in that: In each round of allocation, at least one consecutive node belonging to the same leaf layer switch is allocated to each data parallel group in the positive order of the data parallel groups, specifically including: In the positive order of the data parallel groups, for each data parallel group: Check the number of remaining consecutive nodes of each leaf layer switch; If the number of remaining consecutive nodes of one of the leaf switches is greater than the number of nodes required by the data parallel group, the leaf switch is determined as the target leaf switch, and the required number of consecutive nodes of the target leaf switch is allocated to the data parallel group; If the number of remaining continuous nodes of all the leaf switches is not greater than the number of nodes required by the data parallel group, the leaf switch with the largest number of remaining continuous nodes is determined as the target leaf switch, and the remaining number of continuous nodes of the target leaf switch is allocated to the data parallel group.

6. The node networking method for model training according to claim 2, characterized in that: First, a node is allocated to each data parallel group in the positive order of the data parallel groups, and then a node is allocated to each data parallel group in the reverse order of the data parallel groups, including: In each round of allocation, first, according to the positive order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group; then, according to the reverse order of the data parallel groups and the order of the leaf-layer switches, a corresponding target leaf-layer switch is selected for each data parallel group, and a node is selected from the selected target leaf-layer switches and allocated to the data parallel group.

7. A node networking device for model training, characterized in that: In a distributed cluster for a leaf-spine network architecture, the distributed cluster includes a leaf layer switch and a spine layer switch; each spine layer switch is connected to all leaf layer switches respectively, and the leaf layer switch is used to connect nodes of a data parallel group; The device comprises: A proportion determination module, used to determine the proportion of data parallel communication overhead and pipeline parallel communication overhead of the large model in each training iteration before hybrid parallel training is performed on the large model; The pipeline parallel priority processing module is used to perform multiple rounds of node allocation for each data parallel group according to the required number of nodes of each data parallel group and the number of nodes of the leaf-layer switch if the proportion of pipeline parallel communication overhead is greater than the proportion of data parallel communication overhead: in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements after the positive order allocation is completed, then in each round of allocation, at least one continuous node belonging to the same leaf-layer switch is allocated to each data parallel group containing the remaining nodes in the reverse order of the data parallel groups until the node allocation for each data parallel group is completed.

8. The node networking device for model training according to claim 7, characterized in that: The device also includes: The data parallel priority processing module is used to perform multiple rounds of node allocation for each data parallel group if the proportion of data parallel communication overhead is greater than the proportion of pipeline parallel communication overhead, based on the required number of nodes for each data parallel group and the number of nodes of the leaf layer switch: in each round of allocation, first allocate a node to each data parallel group in the positive order of the data parallel groups; if the data parallel group does not meet the node allocation requirements, continue to allocate a node to each data parallel group in the reverse order of the data parallel groups until all nodes are allocated to each data parallel group.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the node networking method for model training according to any one of claims 1 to 6 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the node networking method for model training according to any one of claims 1 to 6 is implemented.

Images