Large model distributed parallel training method, device, equipment and program product

Abstract

The invention provides a large model distributed parallel training method, device, equipment and program product, and the method comprises the steps: dividing each GPU in a distributed network into a plurality of GPU groups according to the communication type of each GPU connection pair in the distributed network, and determining a target parallel strategy for carrying out the parallel training of a machine learning model according to the obtained GPU groups. According to the technical scheme, the same sub-training task is forbidden to be executed on the GPUs in the different GPU groups under the target parallel strategy, so that it is guaranteed that the communication types between the GPUs used for executing the same sub-training task are consistent. Furthermore, the communication efficiency of the GPU group can be determined according to the communication type of the GPU connection pair in each GPU group, so that the sub-training task is deployed in each GPU group according to the type of the sub-training task, and the parallel training efficiency of the machine learning model is further improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, and program product for large-scale distributed parallel training. Background Technology

[0002] Machine learning models are widely used in various task scenarios. To ensure the performance of these models, they need to be trained. As the scale of machine learning models and the amount of training data increase, a single computer or GPU becomes insufficient to complete the training task, necessitating distributed training using multiple machines and GPUs. Currently, the hardware environment required for distributed training often requires hundreds, thousands, or even tens of thousands of GPUs. Therefore, determining the method for parallel training of machine learning models is a pressing issue that needs to be addressed. Summary of the Invention

[0003] This application provides a distributed parallel training method for large models, including: Obtain the connection methods of each GPU connection pair in the distributed network; wherein, a GPU connection pair consists of two GPUs with a connection relationship; Based on the connection method of each GPU connection pair, the GPU connection pairs are grouped to obtain at least one GPU group; each GPU group includes at least one GPU connection pair, and the connection method of each GPU connection pair in the same GPU group is the same. Based on at least one GPU group, a target parallel strategy for distributed parallel training of large models is determined; wherein, under the target parallel strategy, the same sub-training task is prohibited from being executed on GPUs in different GPU groups; the target parallel strategy is used for distributed parallel training of large models on each GPU.

[0004] This application provides a distributed parallel training device for large models, including: The acquisition module is configured to acquire the communication type of GPU connection pairs in a distributed network; where a GPU connection pair refers to two GPUs that are connected, and the communication type of the GPU connection pair indicates the number of hops in data transmission between the two GPUs in the GPU connection pair. The grouping module is configured to group GPU connection pairs based on their communication type to obtain at least one GPU group; each GPU group includes at least one GPU connection pair, and the communication type of each GPU connection pair in the same GPU group is the same. The determination module is configured to determine the target parallel strategy for distributed parallel training of large models based on at least one GPU group; wherein, under the target parallel strategy, the same sub-training task is prohibited from being executed on GPUs in different GPU groups; the target parallel strategy is used for distributed parallel training of large models on each GPU.

[0005] This application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0006] This application provides a server, including: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; wherein the processor is used to execute the machine-executable instructions to implement the above-described method.

[0007] This application provides a machine-readable storage medium storing machine-executable instructions that can be executed by a processor; wherein the processor executes the machine-executable instructions to implement the above-described method.

[0008] As can be seen from the above technical solutions, this application provides a method for distributed parallel training of large models. In this method, the GPUs in the distributed network are divided into multiple GPU groups according to the communication type of each GPU connection pair in the distributed network, and a target parallel strategy for parallel training of the machine learning model is determined based on the obtained GPU groups. In this way, by prohibiting the execution of the same sub-training task under the target parallel strategy on GPUs in different GPU groups, the consistency of the communication type between GPUs used to execute the same sub-training task is ensured.

[0009] Furthermore, the communication efficiency of a GPU group can be determined based on the communication type of the GPU connection pairs in each GPU group, so that the sub-training task can be deployed in each GPU group according to the type of the sub-training task, thereby further improving the efficiency of parallel training of machine learning models. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the hardware environment of a distributed network in one embodiment of this application; Figure 2 This is a flowchart illustrating a distributed parallel training method for a large model according to one embodiment of this application. Figure 3 This is a schematic diagram of an example architecture of a distributed network in one embodiment of this application; Figure 4 This is a schematic diagram of an example architecture within a server node according to one embodiment of this application; Figure 5 This is a schematic diagram of another example architecture of a distributed network in one embodiment of this application; Figure 6 This is a schematic diagram of yet another example architecture of a distributed network in one embodiment of this application; Figure 7This is a schematic diagram of the structure of a large-scale model distributed parallel training device according to one embodiment of this application; Figure 8 This is a hardware structure diagram of a large-model distributed parallel training device according to one embodiment of this application. Detailed Implementation

[0011] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0012] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0013] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0014] As used in this paper, the term "model" refers to a system that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that uses multiple layers of processing units to process inputs and provide corresponding outputs. In this paper, "model" may also be referred to as a "machine learning model," a "machine learning network," or simply a "network," and these terms are used interchangeably. A model can also include different types of processing units or networks.

[0015] Figure 1 This is a schematic diagram of the hardware environment 100 of a distributed network according to one embodiment of this application. Figure 1As shown, environment 100 may include control unit 150 and distributed network 140. Distributed network 140 may include spine switches, leaf switches, and multiple server nodes. Each server node includes at least one GPU card. Environment 100 includes leaf switches 1, 2, 3, 4, 5, 6, 7, and 8, which may be individually or collectively referred to as leaf switch 120. Environment 100 includes server nodes 130-1, 130-2, and 130-3, which may be individually or collectively referred to as server node 130. Figure 1 The spine switch 110, leaf switch 120, server node 130, and the number of GPU cards in each server node shown are for illustrative purposes only and are not limited here.

[0016] like Figure 1 As shown, multiple GPUs within the same node can communicate directly. For example, in the first server node 130-1, GPU1 and GPU8 can communicate via a connection line between them. This connection line can include a PCI Expander Bridge (PXB) or NVLink, etc. GPUs in different server nodes can communicate via leaf switches and spine switches. For example, GPU1 in the first server node 130-1 can communicate with GPU1 in the second server node 130-2 via leaf switch 1. GPU1 in the first server node 130-1 can also communicate with GPU2 in the second server node 130-2 via a communication link consisting of leaf switch 1, spine switch 110, and leaf switch 2. The distributed network 140 is used to perform distributed training tasks for large models.

[0017] The control unit 150 is used to acquire training tasks (e.g., distributed parallel training tasks) published by the user. The control unit 150 can send the training tasks to the distributed network 140 and control the distributed network 140 to execute the training tasks. The control unit 150 can also be used to acquire the execution results of the training tasks and the operating status of each network device in the distributed network, and display these results to the user.

[0018] Figure 1 The environment 100 shown can be used for distributed training of large models. During the training of a machine learning model, the training task can be broken down into multiple sub-training tasks. Then, the corresponding sub-training tasks are executed using the various GPUs in environment 100. During the execution of the sub-training tasks, the GPUs communicate with each other to synchronize their intermediate computation results. The communication time between the GPUs directly affects the training speed of the large model.

[0019] It should be understood that Figure 1 The components and arrangements shown in environment 100 are merely examples, and a computing system suitable for implementing the exemplary implementations described in this application may include one or more different components, other components, and / or different arrangements. Implementations of this application are not limited in this respect.

[0020] The following describes the large-model distributed parallel training method of this application in conjunction with specific embodiments. Figure 2 This is a flowchart illustrating a distributed parallel training method for large models in one embodiment of this application. Figure 2 The large-model distributed parallel training method shown can be implemented or included in the distributed network 140 or the control unit 150 of the distributed network 140. Here, the method is illustrated using the implementation in the control unit 150 as an example. Figure 2 As shown, the method may include: Step 210: Control unit 150 acquires the connection method of GPU connection pairs in the distributed network. A GPU connection pair refers to two GPUs that are connected. The connection method of a GPU connection pair indicates how many switches the two GPUs in the pair are connected across, the type of connection line between the two GPUs, or the bandwidth of the connection line between the two GPUs. The connection method between GPUs directly affects the latency of communication between GPUs. The more switches the connection between two GPUs passes through and the lower the bandwidth of the connection line, the higher the communication latency between the two GPUs. The distributed network is used to perform large model training tasks. The distributed network includes multiple server nodes, and each server node includes multiple GPU cards. Any two GPUs communicate with each other to form a GPU connection pair. The connection method of a GPU connection pair can include intra-node communication (i.e., the two GPUs are located in the same server node) and inter-node communication (i.e., the two GPUs are located in different server nodes). Further, inter-node communication can include communication between server nodes under the same leaf switch, communication between server nodes under the same spine switch across leaf switches, or communication between server nodes under the same core switch across Spanish switches. Intra-node communication can include multiple types, determined by the type of connection between the two GPUs, such as NVLink type, PHB type, or PXB type.

[0021] Step 220: The control unit 150 groups the GPU connection pairs based on their connection method to obtain at least one GPU group. Each GPU group includes at least one GPU connection pair, and the connection method of each GPU connection pair in the same GPU group is the same.

[0022] For a given GPU group, all GPU connection pairs within that group share the same connection method. For example, each GPU connection pair can be configured for either inter-node communication or intra-node communication. Furthermore, if the connection method between GPU connection pairs is inter-node communication, and if one GPU connection pair's connection method is communication between server nodes under the same Leaf switch, then the connection methods for other GPU connection pairs are also communication between server nodes under the same Leaf switch.

[0023] Step 230: The control unit 150 determines the target parallel strategy for distributed parallel training of the large model based on at least one GPU group. Under the target parallel strategy, the same sub-training task is prohibited from being executed on GPUs in different GPU groups. The target parallel strategy is used for distributed parallel training of the large model on all GPUs.

[0024] In some embodiments, the control unit 150 may determine multiple initial parallel strategies using an exhaustive approach. Subsequently, it selects at least one candidate parallel strategy from the multiple initial parallel strategies that satisfies the principle that the same sub-training task is prohibited from execution on GPUs in different GPU groups. Furthermore, the control unit 150 may use the principle that "the same sub-training task is prohibited from execution on GPUs in different GPU groups" as an exhaustive constraint to obtain at least one candidate parallel strategy.

[0025] In some embodiments, the control unit 150 can determine a target parallel strategy for distributed parallel training of large models based on at least one determined GPU group. The parallel strategy refers to the technique used during distributed training to decompose the model training task into multiple sub-training tasks and execute them in parallel on multiple GPUs, mainly including data parallelism, model parallelism, and pipelined parallelism. During training, multiple GPU cards in a GPU group used to execute the same sub-training task can be considered as a set. The GPU set can include all or a portion of the GPUs in a GPU group. Under different parallel strategies, the model training task is split in different ways, and the sub-training tasks executed by each GPU card are also different.

[0026] The target parallelism strategy is used for distributed parallel training of large models across GPUs. For example, the control unit 150 can determine a splitting scheme for the model training tasks based on the identified multiple GPU groups. For instance, the model training tasks can be split based on the performance of the GPUs in each GPU group to ensure that each sub-training task can be executed on at least one GPU group. In some embodiments, the control unit 150 can deploy the split sub-training tasks to multiple GPU groups according to the target parallelism strategy to execute the corresponding sub-training tasks. For example, the control unit 150 can determine the sub-training tasks to be deployed to each GPU group based on the performance (e.g., communication capabilities) of that GPU group. Under the target parallelism strategy, the same sub-training task is prohibited from being executed on GPUs in different GPU groups, and a single GPU group can be used to execute multiple sub-training tasks.

[0027] In some embodiments, a single target parallel strategy can be determined based on at least one GPU group. In this case, the control unit 150 can perform parallel training on a large model based on the target parallel strategy. Furthermore, the control unit 150 can determine multiple target parallel strategies based on the determined multiple GPU groups. Subsequently, based on the hardware capabilities of the server nodes and GPU cards in the distributed network, the estimated time consumption and estimated memory consumption corresponding to the multiple target parallel strategies are determined. Based on the determined estimated time consumption and estimated memory consumption, the strategy with the best performance is determined from the multiple target parallel strategies.

[0028] This concludes the process. Figure 2 The process is shown below.

[0029] In some embodiments, the control unit 150 can determine the connection method of each GPU connection pair based on the network connection type and connection bandwidth between GPUs in the distributed network.

[0030] The control unit 150 can determine the connection method of each GPU connection pair in the distributed network based on the aggregated communication library associated with the distributed network. In some embodiments, for each GPU connection pair, based on the identification information of the two GPUs in the GPU connection pair, the control unit 150 determines the connection type, communication bandwidth, and communication latency of the GPU connection pair from the aggregated communication library associated with the distributed network. The connection method of the GPU connection pair is determined based on the connection type, communication bandwidth, and communication latency. For example, the control unit 150 can provide GPU identification information or identification information of the server node associated with the GPU (e.g., the first server node 0x01) to the API provided by the aggregated communication library to obtain the connection type, communication bandwidth, and communication latency between any two GPUs in the first server node.

[0031] The control unit 150 obtains the information returned by the API and determines the connection method of each GPU connection pair. For example, if the connection type between a GPU connection pair is across a Leaf switch and within the same Spanish switch, the control unit 150 can determine that the connection method of the GPU connection pair is the first connection method. If the connection type between a connection pair is within the same Leaf switch, the control unit 150 can determine that the connection method of the GPU connection pair is the second connection method.

[0032] In some embodiments, for each GPU connection pair, the control unit 150 can obtain the connection type, communication bandwidth, and communication latency of the GPU connection pair using a network controller associated with the distributed network, based on the identification information of the two GPUs in the GPU connection pair. The connection method of the GPU connection pair is determined based on the connection type, communication bandwidth, and communication latency. The network controller manages the topology type of the distributed network, multiple network devices in the distributed network, and the operating status of each network device. The control unit 150 can provide the identification information of a specified GPU to the network controller API to obtain the connection method of the corresponding GPU connection pair.

[0033] In some embodiments, for each GPU connection pair, the control unit 150 transmits test messages between the two GPUs in the GPU connection pair to determine the communication bandwidth and communication latency of the GPU connection pair. The connection method of the GPU connection pair is determined based on the communication bandwidth and communication latency. For example, for a given GPU connection pair, the control unit 150 can control the first GPU in the GPU connection pair to send test messages to the second GPU to measure the communication bandwidth and communication latency between the two GPUs. Subsequently, the control unit 150 can determine the connection method between the two GPUs based on the communication bandwidth and communication latency between the GPUs.

[0034] In some embodiments, to further improve the accuracy of the determined connection method, multiple test messages containing different amounts of data (e.g., 8-byte, 16-byte, 42-byte, 64-byte, or 128-byte messages) can be sent or received, and the communication bandwidth and latency can be determined based on the test results. Subsequently, the connection method between the two GPUs is determined based on the communication bandwidth and latency.

[0035] In some embodiments, the control unit 150 can determine the connection method for each GPU connection pair based on a configuration file related to the distributed network. The configuration file may include configuration information of the distributed network, such as server models and their internal connection relationships. The configuration file can be a file generated by the network administrator when building the distributed network.

[0036] In some embodiments, each GPU connection pair has a corresponding type value for its connection method. The type value of a connection pair can indicate the communication efficiency between the two GPUs. The type value of a GPU connection pair can be used to distinguish different connection methods, and the magnitude of the type value itself represents the ranking of the communication capabilities of the GPU connection pairs. The type value corresponding to any GPU connection pair is determined based on the connection method of the connection pair and the communication bandwidth between the two GPUs in the GPU connection pair. All GPU connection pairs within the same GPU group have the same type value. Figure 3 This is a schematic diagram of an example architecture 300 of a distributed network in one embodiment of this application. Figure 4 This is a schematic diagram of an example architecture 400 within a server node according to one embodiment of this application. In this embodiment, reference is made to... Figure 3 and Figure 4 The connection methods between GPU connection pairs are explained.

[0037] like Figure 3 As shown, the distributed network includes a core switch 310, a first spine switch 320, a second spine switch 321, a first leaf switch 330, a second leaf switch 331, and a third leaf switch 332. A first server node 340 and a second server node 341 are connected to the first leaf switch 330. The second leaf switch 331 is connected to the third server node 342. Both the first leaf switch 330 and the second leaf switch 331 are connected to the first spine switch 320. The third leaf switch 332 is connected to the fourth server node 343. The third leaf switch 332 is also connected to the second spine switch 321. Both the first spine switch 320 and the second spine switch 321 are connected to the core switch 310.

[0038] For example, the type values ​​for GPU connection pairs between server nodes can be incremented starting from 20, while the type values ​​for GPU connection pairs within a server node can be defined in the range of 0-20. (See reference) Figure 3The type value of GPU connection pairs connected to the same Leaf switch is set to 20. For example, the connection type value of the GPU connection pair consisting of the first GPU in the first server node 340 and the second GPU in the second server node 341 is 20. The type value of GPU connection pairs connected to different Leaf switches but connected to the same Spine switch is set to 30. For example, the connection type value of the connection pair consisting of the first GPU in the first server node 340 and the third GPU in the third server node 342 is 30. The type value of connection pairs connected to different Spine switches but connected to the same Core switch is set to 40. For example, the connection type value of the connection pair consisting of the first GPU in the first server node 340 and the fourth GPU in the fourth server node 343 is 40. The type value of GPU connection pairs connected to different Core switches is set to 50.

[0039] In some embodiments, the type value can be set according to the communication bandwidth of the GPU connection pair. For example, if the communication bandwidth between the first GPU and the fourth GPU is 10 G / s, then the type value of the connection pair formed by the first GPU and the fourth GPU is 41. If the communication bandwidth between the first GPU and the fourth GPU is 2 G / s, then the type value of the connection pair formed by the first GPU and the fourth GPU is 48.

[0040] When two GPUs in a GPU connection pair belong to the same node, the type value of the GPU connection pair can be determined based on the connection lines between the two GPUs. (Reference) Figure 4 If the two GPUs are connected via NVLink, the type value of the GPU connection pair is 0. If the two GPUs are connected via a PIX line, the type value of the GPU connection pair is 1. If the two GPUs are connected via a PXB line, the type value of the GPU connection pair is 2. If the two GPUs are connected via a SoC, the type value of the connection pair is 3.

[0041] In some embodiments, the connection method or type value of each obtained GPU connection pair can be represented or stored in the form of a matrix to facilitate user viewing and grouping of GPU connection pairs according to their connection method. Figure 5 This is a schematic diagram of yet another example architecture of the distributed network provided in the embodiments of this application. Figure 5 As shown, the distributed network includes a spine switch 510, a first leaf switch 520, a second leaf switch 521, a first server node 530, a second server node 531, and a third server node 532. Each server node includes 8 GPUs. Taking three server nodes with 24 GPU cards as an example, the network connection between the three server nodes is as follows: Figure 6 As shown. Figure 6 As shown, the first region 610 indicates the type value of the connection mode for GPU connection pairs within a node; the second region 620 indicates the type value of the connection mode for GPU connection pairs across server nodes but under the same leaf switch; and the third region 630 indicates the type value of the connection mode for GPU connection pairs across leaf switches but under the same spine switch.

[0042] In some embodiments, a single GPU can form multiple GPU connection pairs with multiple other GPUs. Therefore, during the grouping of GPU connection pairs, a single GPU connection can appear in multiple GPU groups. (See reference) Figure 5 The distributed network architecture shown can divide the multiple GPUs in the first server node 530, the second server node 531, and the third server node 532 into multiple GPU groups. For example, the multiple GPU groups include intra-node group 1, intra-node group 2, intra-node group 3, inter-node group 1, inter-node group 2, and inter-node group 3.

[0043] Intra-node group 1 includes all GPU connection pairs in the first server node 530, with a connection type value of 0. Intra-node group 2 includes all GPU connection pairs in the second server node 531, with a connection type value of 0. Intra-node group 3 includes all GPU connection pairs in the third server node 532, with a connection type value of 0. In each GPU connection pair included in inter-node group 1, the two GPUs are located in the first server node 530 and the second server node 531, respectively. The connection type value of the GPU connection pairs in inter-node group 1 is 20. In each GPU connection pair included in inter-node group 2, the two GPUs are located in the first server node 530 and the third server node 532, respectively. In each GPU connection pair included in inter-node group 3, the two GPUs are located in the second server node 531 and the third server node 532, respectively. The connection type value of the GPU connection pairs in inter-node groups 2 and 3 is 30.

[0044] In some embodiments, the process of determining a target parallel strategy based on multiple GPU groups may include: determining multiple initial parallel strategies through an exhaustive approach. Subsequently, the selection of parallel strategies is constrained by certain rules based on the GPU groups to filter out initial parallel strategies that do not conform to the principle that "the same sub-training task is prohibited from being executed on GPUs in different GPU groups," thereby determining multiple candidate parallel strategies. In this way, by retaining only the automatic parallel algorithm for the ensemble communication involved in the parallel strategy within the same logical device group, the communication efficiency of all GPU connection pairs involved in the ensemble communication is ensured to be balanced, preventing a situation where communication between two GPUs slows down the overall communication efficiency of the ensemble communication. Furthermore, based on the frequency of ensemble communication transmission, the sub-training task with the largest amount of ensemble communication transmission can be preferentially deployed in the GPU group with the smallest connection type value.

[0045] In some embodiments, if multiple candidate parallel strategies are determined according to GPU groups, the control unit 150 can determine the estimated time consumption of each candidate parallel strategy in performing a large model partial training task. Then, based on the estimated time consumption corresponding to each candidate parallel strategy, at least one target parallel strategy is determined from the multiple candidate parallel strategies. Here, the process of determining the estimated time consumption corresponding to each candidate parallel strategy will be described.

[0046] The control unit 150 can divide a large model training task into multiple sub-training tasks according to a candidate parallel strategy. In some embodiments, the control unit 150 can use a testing tool (e.g., NCCL TEST) to simulate the training process of the large model under the candidate parallel strategy, based on the GPUs involved in executing the multiple sub-training tasks and the amount of data sent during the task execution, to determine the estimated task time for executing each sub-training task. Subsequently, the estimated time corresponding to the candidate parallel strategy is determined based on the sum of the estimated task times corresponding to each sub-training task.

[0047] In some embodiments, the control unit 150 may determine the estimated task time corresponding to a sub-training task based on the task information corresponding to a certain sub-training task and information related to the GPU group used to execute the sub-training task.

[0048] In some embodiments, for a specific sub-training task among multiple sub-training tasks, the control unit 150 can determine the number of communications between GPUs during the execution of the sub-training task based on the number of GPUs executing the sub-training task. The control unit 150 can determine the number of communications between GPUs during the execution of the sub-training task based on the type of aggregate communication of the GPU set during the execution of the sub-training task and the number of GPUs used to execute the sub-training task (i.e., the number of GPUs in the GPU set). The GPU set may include all or a portion of the GPUs in a GPU group. Types of aggregate communication include, but are not limited to, broadcast, reduction, collection, distribution, and all-reduce. For example, if the type of aggregate communication is all-reduce, and the number of GPUs used to execute the sub-training task is n, then the number of communications is 2(n-1). If the type of aggregate communication is reduction-distribution, all-collection, or all-exchange, then the number of communications is n-1.

[0049] Subsequently, the control unit 150 can determine the estimated fixed time for executing the sub-training task based on the number of communications and the point-to-point constant latency between the GPU executing the sub-training task. Point-to-point constant latency refers to the base time, independent of message length, introduced by a single hardware DMA transfer "from the local GPU memory to the remote GPU memory". Point-to-point constant latency is basically composed of three parts: link physical propagation delay, switch / routing node forwarding delay, and GPU internal fixed overhead. Algorithm bandwidth indicates the bandwidth occupied by the algorithm application sending data, which can be obtained by dividing the amount of data sent by the transmission time.

[0050] The control unit 150 can determine the estimated task time for executing the sub-training task based on the algorithm bandwidth between the GPUs executing the sub-training task, the amount of data sent by each GPU during the execution of the sub-training task, and the number of communication attempts. For example, if the type of aggregate communication is full reduction, the estimated task time for executing a particular sub-training task can be determined using the following formula: (1) Where T is the estimated task time, and S is the total amount of data sent during the execution of the sub-training task. This refers to the amount of data sent by each GPU during the execution of this sub-training task.

[0051] Furthermore, the "point-to-point fixed delay" can be represented by the parameter Alpha, and the "1 / point-to-point algorithm bandwidth" can be represented by the parameter Beta. According to the above formula (1), the following calculation formula can be obtained: (2) If the type of aggregate communication is reduction-spread, full collection, or full exchange, the estimated time can be determined using the following formula: (3) In some embodiments, the control unit 150 can determine the Alpha and Beta values ​​through testing. For example, the control unit 150 can determine multiple test results by sending messages containing different amounts of data (with the data size gradually increasing, e.g., 8, 16, 32, 64, 128, etc.). The point-to-point fixed latency is then determined based on the average or median of the test results. Furthermore, the control unit 150 can measure the point-to-point algorithm bandwidth using large messages (e.g., messages with 1GB of data or the maximum message size allowed by the buffer).

[0052] In some embodiments, the control unit 150 may predetermine a time consumption function corresponding to the CPU group executing a sub-training task. The time consumption function indicates the relationship between the estimated time consumption and the fixed latency of the GPU group executing the sub-training task, the point-to-point fixed latency between the GPUs executing the sub-training task, and the unit data transfer latency between the GPUs executing the sub-training task. The control unit 150 can calculate the parameters in the function using linear regression. For example, if the ensemble communication type is full reduction, the estimated task time for executing a sub-training task can be calculated using the following formula: (4) Where a is the total point-to-point fixed latency of communication between GPUs during the execution of the sub-training task (i.e., the fixed latency of the GPU group), b is the point-to-point fixed latency between GPUs executing the sub-training task, and c is the unit data transmission latency between GPUs executing the sub-training task.

[0053] For a given GPU group, the control unit 150 can select a certain number of GPUs from that group and obtain multiple sets of actual communication latency data through communication testing. Subsequently, based on the obtained multiple sets of actual communication latency data, the values ​​of parameters a, b, and c corresponding to that GPU group are obtained through multiple linear regression or least squares method.

[0054] As shown in formula (4), the time consumption function and the amount of data sent by each GPU during the execution of the sub-training task can be used as a basis (i.e., The estimated time to execute the sub-training task is determined by the value of the sub-training task and the number of communications during the execution of the sub-training task.

[0055] Based on the same concept as the methods described above, this application proposes a large-model distributed parallel training device, see [link to relevant documentation]. Figure 7 The diagram shows the structure of a parameter configuration device. The device may include: an acquisition module 710, configured to acquire the connection methods of GPU connection pairs in a distributed network; wherein a GPU connection pair refers to two GPUs with a connection relationship, and the connection method of the GPU connection pair indicates the number of hops in data transmission between the two GPUs in the GPU connection pair. A grouping module 720, configured to group the GPU connection pairs in the distributed network based on their connection methods to obtain at least one GPU group; each GPU group includes at least one GPU connection pair, and the connection methods of all GPU connection pairs within the same GPU group are identical. A determination module 730, configured to determine a target parallel strategy for large-scale distributed parallel training based on at least one GPU group; wherein, under the target parallel strategy, the same sub-training task is prohibited from execution on GPUs in different GPU groups; the target parallel strategy is used for large-scale distributed parallel training on each GPU.

[0056] Optionally, the connection method of the GPU connection pair indicates any of the following types: when the two GPUs in the GPU connection pair belong to different nodes, the connection type between those different nodes; the connection type includes: connected to the same Leaf switch, connected to different Leaf switches and connected to the same Spine switch, connected to different Spine switches and connected to the same Core switch, or connected to different Core switches; when the two GPUs in the GPU connection pair belong to the same node, the type of connection line between those two GPUs.

[0057] Optionally, the connection method of any GPU connection pair has a corresponding type value, and the type value of any GPU connection pair is determined based on the connection method of the connection pair and the communication bandwidth between the two GPUs in the GPU connection pair; the type values ​​of each GPU connection pair in the same GPU group are the same.

[0058] Optionally, determining the target parallel strategy for distributed parallel training of large models based on at least one GPU group includes: selecting at least one candidate parallel strategy from the existing parallel strategies according to the principle that the same sub-training task is prohibited from being executed on GPUs in different GPU groups; if the number of candidate parallel strategies is 1, then the candidate parallel strategy is determined as the target parallel strategy; if the number of candidate parallel strategies is not 1, then the target parallel strategy is selected from the candidate parallel strategies.

[0059] Optionally, selecting a target parallel strategy from the candidate parallel strategies includes: for each candidate parallel strategy, determining the estimated time required for each GPU group to perform distributed parallel training of a large model under that candidate parallel strategy; and determining the target parallel strategy based on the estimated time required for each candidate parallel strategy.

[0060] Optionally, based on the candidate parallel strategy, the model training task is divided into multiple sub-training tasks; the estimated time of the candidate parallel strategy is determined based on the sum of the estimated task times of each sub-training task; wherein the estimated task time of each sub-training task is determined as follows: for each sub-training task, the estimated fixed time of execution of the sub-training task is determined based on the number of communication times between GPUs and the point-to-point fixed latency between GPUs executing the sub-training task; wherein the number of communication times is determined based on the number of GPUs executing the sub-training task; the estimated fixed time of execution of the sub-training task is determined based on the algorithm bandwidth between GPUs executing the sub-training task and the point-to-point fixed latency between GPUs executing the sub-training task; wherein the number of communication times is determined based on the number of GPUs executing the sub-training task; the estimated fixed time of execution of the sub-training task is determined based on the sum of the estimated task times of each GPU executing the sub-training task. The estimated task time for executing the sub-training task is determined by the amount of data sent by the GPU and the number of communications. Alternatively, for each of the multiple sub-training tasks, a time consumption function corresponding to the GPU group executing the sub-training task is determined. The time consumption function indicates the relationship between the estimated time and the fixed latency of the GPU group executing the sub-training task, the point-to-point fixed latency between the GPUs executing the sub-training task, and the unit data transmission latency between the GPUs executing the sub-training task. Based on the time consumption function, the amount of data sent by each GPU during the execution of the sub-training task, and the number of communications during the execution of the sub-training task, the estimated task time for executing the sub-training task is determined.

[0061] Optionally, obtaining the connection method of GPU connection pairs in the distributed network includes: for each GPU connection pair, determining the connection type, communication bandwidth, and communication latency of the GPU connection pair from a collection of communication libraries related to the distributed network based on the identification information of the two GPUs in the GPU connection pair; determining the connection method of the GPU connection pair based on the connection type, communication bandwidth, and communication latency; or, for each GPU connection pair, obtaining the connection type, communication bandwidth, and communication latency of the GPU connection pair using a network controller related to the distributed network based on the identification information of the two GPUs in the GPU connection pair; determining the connection method of the GPU connection pair based on the connection type, communication bandwidth, and communication latency; or, for each GPU connection pair, transmitting test messages between the two GPUs in the GPU connection pair to determine the communication bandwidth and communication latency of the GPU connection pair; determining the connection method of the GPU connection pair based on the communication bandwidth and communication latency; or, determining the connection method of each GPU connection pair based on a configuration file related to the distributed network.

[0062] Based on the same concept as the methods described above, this application proposes a large-model distributed parallel training system, see [link to relevant documentation]. Figure 8As shown, the large model distributed parallel training system includes a processor 81 and a machine-readable storage medium 82. The machine-readable storage medium 82 stores machine-executable instructions that can be executed by the processor 81. The processor 81 is used to execute the machine-executable instructions to implement the large model distributed parallel training method.

[0063] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions. When the computer instructions are executed by a processor, they can implement the parameter configuration method of the above example application.

[0064] The aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0065] Based on the same application concept as the above method, this application embodiment also provides a computer program product, including a computer program that implements the above-described large model parallel training method when executed by a processor.

[0066] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A distributed parallel training method for large models, characterized in that the method includes: Obtain the connection methods of each GPU connection pair in the distributed network; wherein, a GPU connection pair consists of two GPUs with a connection relationship; Based on the connection method of each GPU connection pair, the GPU connection pairs are grouped to obtain at least one GPU group; each GPU group includes at least one GPU connection pair, and the connection method of each GPU connection pair in the same GPU group is the same. Based on the at least one GPU group, a target parallel strategy for distributed parallel training of a large model is determined; wherein, under the target parallel strategy, the same sub-training task is prohibited from being executed on GPUs in different GPU groups; the target parallel strategy is used for distributed parallel training of a large model on each GPU.

2. The method according to claim 1, characterized in that, The connection method of the GPU connection pair indicates any of the following types: When two GPUs in a GPU connection pair belong to different nodes, the connection type between those different nodes includes: connecting to the same Leaf switch, connecting to different Leaf switches and connecting to the same Spine switch, connecting to different Spine switches and connecting to the same Core switch, or connecting to different Core switches. The type of connection line between two GPUs in a GPU connection pair when they belong to the same node.

3. The method according to claim 1, characterized in that, Each GPU connection pair has a corresponding type value for its connection method. The type value of each GPU connection pair is determined based on the connection method of the connection pair and the communication bandwidth between the two GPUs in the GPU connection pair. The type values ​​of all GPU connection pairs in the same GPU group are the same.

4. The method according to claim 1, characterized in that, Based on the at least one GPU group, the target parallel strategy for distributed parallel training of large models includes: Based on the principle that the same sub-training task is prohibited from being executed on GPUs in different GPU groups, at least one candidate parallel strategy is selected from the existing parallel strategies. If the number of candidate parallel strategies is 1, then the candidate parallel strategy is determined as the target parallel strategy; If the number of candidate parallel strategies is not 1, then the target parallel strategy is selected from among the candidate parallel strategies.

5. The method according to claim 4, characterized in that, The selection of the target parallel strategy from among the candidate parallel strategies includes: For each candidate parallel strategy, determine the estimated time for each GPU group to perform distributed parallel training of the large model under that candidate parallel strategy; The target parallel strategy is determined based on the estimated time consumption corresponding to each candidate parallel strategy.

6. The method according to claim 5, characterized in that, For each candidate parallel strategy, the estimated time for each GPU group to perform distributed parallel training of a large model under that candidate parallel strategy is determined, including: Based on this candidate parallel strategy, the model training task is split into multiple sub-training tasks; The estimated time of the candidate parallel strategy is determined based on the sum of the estimated time of each sub-training task. The estimated time for each sub-training task is determined in the following way: For each of the plurality of sub-training tasks, the estimated fixed execution time of the sub-training task is determined based on the number of communication calls between GPUs during the execution of the sub-training task and the point-to-point fixed latency between the GPUs executing the sub-training task; wherein, the number of communication calls is determined based on the number of GPUs executing the sub-training task; the estimated task execution time of the sub-training task is determined based on the algorithm bandwidth between the GPUs executing the sub-training task, the amount of data sent by each GPU during the execution of the sub-training task, and the number of communication calls; or, For each of the plurality of sub-training tasks, a time consumption function corresponding to the GPU group executing the sub-training task is determined; wherein, the time consumption function indicates the relationship between the estimated time consumption and the fixed latency of the GPU group executing the sub-training task, the point-to-point fixed latency between the GPUs executing the sub-training task, and the unit data transmission latency between the GPUs executing the sub-training task; based on the time consumption function, the amount of data sent by each GPU during the execution of the sub-training task, and the number of communications during the execution of the sub-training task, the estimated task consumption time for executing the sub-training task is determined.

7. The method according to claim 1, characterized in that, The methods for obtaining GPU connection pairs in a distributed network include: For each GPU connection pair, based on the identification information of the two GPUs in the pair, the connection type, communication bandwidth, and communication latency of the GPU connection pair are determined from the aggregated communication library related to the distributed network; the connection method of the GPU connection pair is determined based on the connection type, communication bandwidth, and communication latency; or, For each GPU connection pair, based on the identification information of the two GPUs in the pair, the connection type, communication bandwidth, and communication latency of the GPU connection pair are obtained using a network controller associated with the distributed network; the connection method of the GPU connection pair is determined based on the connection type, communication bandwidth, and communication latency; or, For each GPU connection pair, test messages are transmitted between the two GPUs in the pair to determine the communication bandwidth and latency of the GPU connection pair; based on the communication bandwidth and latency, the connection method of the GPU connection pair is determined; or, Based on the configuration file associated with the distributed network, the connection method for each GPU connection pair is determined.

8. A large-scale model distributed parallel training device, characterized in that the device includes... The acquisition module is configured to acquire the connection methods of each GPU connection pair in the distributed network; among which, A GPU connection pair consists of two GPUs that are connected. The grouping module is configured to group the GPU connection pairs based on their connection methods to obtain at least one GPU group; each GPU group includes at least one GPU connection pair, and the connection methods of each GPU connection pair in the same GPU group are the same. The determining module is configured to determine a target parallel strategy for large-scale model distributed parallel training based on the at least one GPU group; wherein, under the target parallel strategy, the same sub-training task is prohibited from being executed on GPUs in different GPU groups; the target parallel strategy is used for large-scale model distributed parallel training on each GPU.

9. A large-scale model distributed parallel training device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method of any one of claims 1-7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

Images