Distributed model training-oriented congestion control method, equipment and medium

Abstract

The invention relates to the technical field of computer networks, and discloses a distributed model training-oriented congestion control method and device, and a medium. The method comprises the following steps: when a cooperative switch receives a local gradient data packet, updating the total communication traffic required to be transmitted by each training task in a task total communication traffic table according to a corresponding field; when the cooperative switch receives the global gradient data packet, the residual communication traffic of each training task in the task total communication traffic table is updated according to the corresponding field, and the residual communication traffic is broadcasted to each working node, so that the working node updates the residual communication traffic of each training task in the residual communication traffic table; when network congestion is detected, calculating a real-time proportion, and dynamically adjusting a sending rate of a working node corresponding to each training task through an increasing function; a working node senses the existence of other training tasks, the residual communication traffic of the other training tasks in the communication stage of the current iteration round is obtained, and the training task with less residual communication traffic obtains a larger bandwidth, so that the communication stage staggering is realized, and the link utilization rate is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer networks, and particularly relates to a congestion control method, device and medium for distributed model training. Background Art

[0002] In the past few years, the field of artificial intelligence has experienced remarkable development. Especially in multiple application scenarios such as computer vision, natural language processing, and recommendation systems, the demand has been continuously rising. With the advancement of this trend, many landmark research results have emerged one after another, and they are often accompanied by complex deep learning models. These models not only have numerous parameters but also need to process huge training data sets. Therefore, training these deep learning models not only consumes a large amount of time but also poses a huge demand for computing resources. To accelerate this training process, distributed training schemes have been increasingly emphasized and widely applied. Currently, the more commonly used one is distributed model training based on data parallelism. The main steps of distributed model training include: data preparation, model preparation, forward propagation, backward propagation, gradient synchronization, and model update. The entire training process usually consists of multiple iterative rounds. In each iteration, each worker node is responsible for calculating the local gradient data and sending these local gradient data to the parameter server through the network for aggregation. These processes are respectively called gradient calculation and gradient aggregation. It is worth noting that the traffic during the process of distributed model training has obvious regularity, that is, the calculation phase and the communication phase alternate periodically. This provides an important basis for us to design a congestion control algorithm for distributed model training at the transport layer.

[0003] Data Center Quantized Congestion Notification (DCQCN) is the most widely used congestion control algorithm in the RoCEv2 network currently. It combines the Quantized Congestion Notification (QCN) algorithm and the DCTCP algorithm, and requires data center switches to support Weighted Random Early Detection (WRED) and Explicit Congestion Notification (ECN). The DCQCN algorithm can provide better fairness, achieve high bandwidth utilization, ensure low queue buffer occupancy, and less queue buffer jitter. The DCQCN algorithm consists of three parts: the Congestion Point (CP) in the switch, the Notification Point (NP) at the receiver, and the Reaction Point (RP) at the sender. When the switch discovers that the outport queue exceeds the threshold, it will mark the packet with an ECN congestion mark (the ECN field is set to 11) with a certain probability when forwarding the packet, to indicate the congestion existing in the network. The marking process is completed by the WRED function. The Reaction Point at the sender sends backpressure information to the Reaction Point at the sender according to the received ECN congestion mark. As a general congestion control algorithm for data centers, the DCQCN algorithm cannot effectively utilize the periodic characteristics of the communication phase in the distributed model training process. When network congestion occurs, the DCQCN algorithm usually divides the shared link bandwidth equally among multiple training task data streams. This seemingly reasonable "fairness" may cause serious problems in the distributed model training scenario. Specifically, when there are multiple tenants in the training system, that is, multiple tasks are performing distributed training simultaneously, the communication phases of different tasks will overlap, and the degree of overlap will continue to increase. In the worst case, it may cause the communication phases of these tasks to completely overlap. This reduces the link utilization rate and increases the transmission delay, and further makes the overall task training time longer.

[0004] The MLQCN algorithm is a technique that enhances the existing congestion control algorithm (DCQCN), aiming to accelerate the training tasks of deep neural networks (DNNs) in shared GPU clusters. This algorithm is based on an important principle: the training flow should dynamically adjust the sending rate according to the number of bytes sent. The MLQCN algorithm adjusts the aggressiveness of competing flows based on the remaining bytes in the current iteration, thus approximating the shortest remaining processing time strategy. After several iterations, the communication phases of competing jobs are separated into an interleaved state, significantly improving the utilization of network links, reducing the average iteration time, and enhancing the model training efficiency. However, the existing solutions have obvious limitations. Since each training task cannot perceive the existence of the other training tasks and cannot obtain the progress of the communication phases of other training tasks, there is uncertainty in selecting the task with the least remaining traffic, and it is usually only applicable to scenarios where the traffic of training tasks is equal or similar. Summary of the Invention

[0005] The present invention provides a congestion control method, device, and medium for distributed model training, which solves the above technical problems by means of collaborative perception based on worker nodes (or computing nodes).

[0006] To solve the above technical problems, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a congestion control method for distributed model training, including:

[0008] Determine the worker nodes, switches, and parameter servers participating in the distributed model training, and use the programmable switch connected to the parameter server as the collaborative switch; create a task remaining traffic table in the worker nodes and a task total traffic table in the collaborative switch;

[0009] Each worker node sends local gradient data packets to the parameter server for aggregation, and the parameter server sends the aggregated global gradient data packets to each worker node;

[0010] When the collaborative switch receives a local gradient data packet, it updates the total traffic that each training task needs to transmit recorded in the task total traffic table according to the corresponding field; when the collaborative switch receives a global gradient data packet, it updates the remaining traffic of each training task recorded in the task total traffic table according to the corresponding field and broadcasts it to each worker node, so that the worker node updates the remaining traffic of each training task recorded in the task remaining traffic table;

[0011] When network congestion is detected, count the number of training tasks S with non-zero remaining traffic. When S is greater than 1, assign a higher real-time ratio bytes_radio to the training tasks with lower remaining traffic, and dynamically adjust the sending rate of the working nodes corresponding to each training task through the increasing function f(bytes_radio).

[0012] In one embodiment, creating a task remaining traffic table in the working node and a task total traffic table in the collaborative switch specifically includes:

[0013] Create a task remaining traffic table with 1 row and n columns in the working node to record the remaining traffic of each training task in the communication phase of this round of iteration; n is the total number of training tasks;

[0014] Create a task total traffic table with 2 rows and n columns in the collaborative switch to record the total traffic and remaining traffic that each training task needs to transmit in the communication phase of this round of iteration;

[0015] Both the task remaining traffic table and the task total traffic table are indexed by the training task number and initialized to 0.

[0016] In one embodiment, when the collaborative switch receives a local gradient data packet, update the total traffic that each training task needs to transmit recorded in the task total traffic table according to the corresponding field, specifically including:

[0017] The local gradient data includes a total traffic field total_bytes, and the collaborative switch updates the total traffic that each training task needs to transmit in the task total traffic table according to the content of the total traffic field total_bytes of the local gradient data packet.

[0018] In one embodiment, when the collaborative switch receives a global gradient data packet, update the remaining traffic of each training task recorded in the task total traffic table, and broadcast it to each working node so that the working node updates the remaining traffic of each training task recorded in the task remaining traffic table, specifically including:

[0019] The global gradient data includes a traffic field psize and a remaining traffic field remain_bytes. The traffic field psize is used to record the size of the global aggregated gradient data carried in the global gradient data packet;

[0020] The collaborative switch calculates the latest remaining traffic of the current training task in the communication phase of this round of iteration through the content of the traffic field psize and the remaining traffic recorded in the task total traffic table, updates the task total traffic table, writes it into the remaining traffic field remain_bytes of the global gradient data packet, and forwards it to the working node.

[0021] In one embodiment, for the real-time ratio bytes_radio where the lower the remaining traffic of a training task, the higher the assigned value, the transmission rate of the working node corresponding to each training task is dynamically adjusted through the increasing function f(bytes_radio), which specifically includes:

[0022] Sort the remaining traffic of each training task from smallest to largest to obtain the ranking of each training task; calculate the proportion of the remaining traffic of each training task in the total remaining traffic of all training tasks, and sort from largest to smallest to obtain the ranking of the remaining traffic proportion of each training task; use the remaining traffic proportion of the i-th ranking as the real-time ratio bytes_radio of the training task of the i-th ranking.

[0023] Adjust the existing transmission rate cur_rate using the increasing function f and bytes_radio to obtain the new transmission rate cur_rate′.

[0024] cur_rate′ = cur_rate * f(bytes_radio).

[0025] In a second aspect, the present invention provides a computer device, including a memory and a processor, where the memory stores a computer program, and when the processor executes the computer program, the steps of the method according to any one of the embodiments in the first aspect are implemented.

[0026] In a third aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method according to any one of the embodiments in the first aspect are implemented.

[0027] Compared with the prior art, the beneficial technical effects of the present invention are:

[0028] By enabling the working node to perceive the existence of other training tasks, obtain the remaining traffic of other training tasks in the communication stage of the current iteration round, and through an "unfair" dynamic bandwidth allocation strategy, allowing the training tasks with less remaining traffic to obtain a larger bandwidth, thereby achieving the interleaving of the communication stage, improving the link utilization rate, and shortening the overall training time of the task. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of the congestion control method for distributed model training provided by an example of the present invention;

[0030] Figure 2 It is a schematic framework diagram of the working node's collaborative perception of training task information provided by an example of the present invention;

[0031] Figure 3Complete topological schematic diagram of the distributed training network provided by the example of the present invention;

[0032] Figure 4 Schematic diagram of the change of relevant fields of the data packet provided by the example of the present invention. Detailed implementation manners

[0033] The following is a detailed description of a preferred implementation manner of the present invention with reference to the accompanying drawings.

[0034] As Figure 1 shown, a congestion control method for distributed model training in the present invention includes:

[0035] Determine the worker nodes, switches, and parameter servers participating in the distributed model training, and use the programmable switch connected to the parameter server as the collaborative switch; create a task remaining communication volume table in the worker nodes and create a task total communication volume table in the collaborative switch;

[0036] Each worker node sends the local gradient data packet to the parameter server for aggregation, and the parameter server sends the aggregated global gradient data packet to each worker node;

[0037] When the collaborative switch receives the local gradient data packet, it updates the total communication volume that each training task needs to transmit recorded in the task total communication volume table according to the corresponding fields; when the collaborative switch receives the global gradient data packet, it updates the remaining communication volume of each training task recorded in the task total communication volume table according to the corresponding fields, and broadcasts it to each worker node, so that the worker node updates the remaining communication volume of each training task recorded in the task remaining communication volume table;

[0038] When network congestion is detected, count the number S of training tasks with non-zero remaining communication volume. When S is greater than 1, assign a higher real-time ratio bytes_radio to the training task with lower remaining communication volume, and dynamically adjust the sending rate of the worker node corresponding to each training task through the increasing function f(bytes_radio).

[0039] In one embodiment, the creating a task remaining communication volume table in the worker nodes and creating a task total communication volume table in the collaborative switch specifically includes:

[0040] Create a 1-row and n-column task remaining communication volume table in the worker nodes to record the remaining communication volume of each training task in this round of iterative communication stage; n is the total number of training tasks;

[0041] Create a 2-row and n-column task total communication volume table in the collaborative switch to record the total communication volume and remaining communication volume that each training task needs to transmit in this round of iterative communication stage;

[0042] The task remaining communication volume table and the task total communication volume table are both indexed by the training task number and initialized to 0.

[0043] In one embodiment, when the collaborative switch receives a local gradient data packet, it updates the total communication volume that each training task needs to transmit recorded in the task total communication volume table according to the corresponding field, specifically including:

[0044] The local gradient data includes a total communication volume field total_bytes, and the collaborative switch updates the total communication volume that each training task needs to transmit in the task total communication volume table according to the content of the total communication volume field total_bytes of the local gradient data packet.

[0045] In a traditional data center, it is infeasible to accurately estimate the size of each flow. However, within the life cycle of most deep neural network training tasks, as long as the hyperparameters remain unchanged, the total number of bytes in the training iteration is a predictable and consistent value.

[0046] The structure of the local gradient data packet of the present invention is shown in Table 1.

[0047] Table 1

[0048]

[0049] In Table 1, jobid is the training task number field and is the unique identifier of the distributed training task. Islocal is an identification bit used to identify the scope of the gradient data (global aggregated gradient, local computing node gradient). Total_bytes is the total communication volume field. Other fields are common fields for those skilled in the art and will not be introduced in detail here.

[0050] In one embodiment, when the collaborative switch receives a global gradient data packet, it updates the remaining communication volume of each training task recorded in the task total communication volume table according to the corresponding field, and broadcasts it to each worker node so that the worker node updates the remaining communication volume of each training task recorded in the task remaining communication volume table, specifically including:

[0051] The global gradient data includes a communication volume field psize and a remaining communication volume field remain_bytes. The communication volume field psize is used to record the size of the global aggregated gradient data carried in the global gradient data packet;

[0052] The collaborative switch calculates the latest remaining communication volume in the communication stage of the current training task iteration through the content of the communication volume field psize and the remaining communication volume recorded in the task total communication volume table, updates the task total communication volume table, writes it into the remaining communication volume field remain_bytes of the global gradient data packet, and forwards it to the worker node.

[0053] The structure of the global gradient data packet of the present invention is shown in Table 2.

[0054] Table 2

[0055]

[0056] In Table 2, the meanings of the jobid field and the islocal field are the same as those in Table 1. The psize is the traffic volume field, which is used to record the size of the global aggregated gradient data carried in the global gradient data packet. The remain_bytes is the remaining traffic volume field, which is used to record the remaining traffic volume in the current communication stage of the current training task iteration. Other fields are common fields for those skilled in the art and will not be introduced in detail here.

[0057] In one embodiment, a real-time ratio bytes_radio with a higher value assigned to a training task with a lower remaining traffic volume is used to dynamically adjust the sending rate of the working nodes corresponding to each training task through an increasing function f(bytes_radio), specifically including:

[0058] Sort the remaining traffic volumes of each training task from smallest to largest to obtain the ranking of each training task; calculate the proportion of the remaining traffic volume of each training task in the total remaining traffic volume of all training tasks and sort from largest to smallest to obtain the ranking of the remaining traffic volume proportion of each training task; use the remaining traffic volume proportion of the i-th ranking as the real-time ratio bytes_radio of the training task of the i-th ranking;

[0059] Use the increasing function f and bytes_radio to adjust the existing sending rate cur_rate to obtain a new sending rate cur_rate';

[0060] cur_rate' = cur_rate * f(bytes_radio).

[0061] Specifically, after the working node receives the congestion notification packet (CNP), it indicates that network congestion has occurred and the sending rate needs to be adjusted. The identification of network congestion, congestion feedback, and adjustment of the sending rate in the present invention can all be executed according to the DCQCN algorithm.

[0062] In the present invention, tasks with less remaining volume are given higher priority: the less the remaining volume of a task, the more forward its ranking in the sorting from smallest to largest. After mapping to the proportion sequence sorted from largest to smallest, the proportion corresponding to the task with less remaining volume may be higher.

[0063] In one embodiment, the distributed model training based on data parallelism of the present invention adopts a centralized architecture of a parameter server. As Figure 3As shown, there are two training clusters, each consisting of 5 servers as worker nodes and 1 parameter server connected through a spine-leaf network architecture. Among them, Cluster 1 participates in Training Task 1 (job1), and Cluster 2 participates in Training Task 2 (job2). Here, the programmable switch connecting the parameter server is selected as the collaborative switch.

[0064] First, determine the worker nodes, switches, and parameter servers participating in the distributed model training, and allocate relevant resources to each. In this example, since there are 2 training tasks, in each worker node, a 1-row 2-column task remaining communication volume table needs to be allocated to store the remaining communication volume in the communication stage and initialized to 0, and looked up according to the jobid; in the switch, a 2-row 2-column task total communication volume table needs to be allocated to store the total communication volume in the communication stage of each task and initialized to 0, and looked up according to the jobid.

[0065] The worker nodes participating in the training of job1 fill the jobid field with 1, fill the islocal field with 1, fill the total_bytes field with 10000000, and fill the remaining fields in sequence to form a complete data packet. This indicates that this is the local gradient data packet of job1, and the total data volume in this round of iterative communication stage is 10MB. Similarly, the worker nodes participating in the training of job2 fill the jobid, islocal, and total_bytes fields with 2, 1, and 5000000 respectively. This indicates that this is the local gradient data packet of job2, and the total data volume in this round of iterative communication stage is 5MB.

[0066] Suppose at a certain moment during the training process, the task information table in the switch is as Figure 4 , indicating that the remaining communication volume of job1 is 6MB and the remaining communication volume of job2 is 4MB. The switch receives the local gradient data packets from job1 and job2, parses fields such as jobid, islocal, and total_bytes, and updates the task information table. The switch receives the aggregated result data packet from job1, parses the psize field as 100000. At this time, to calculate the remaining communication volume, subtract 1000000 from 6000000 to get 5000000 and fill it into the remain_bytes field and broadcast it to the worker nodes. According to this step, each worker node will obtain the remaining communication volume of the other training tasks.

[0067] At this time, congestion occurred. The working node 1 received the congestion notification packet CNP and calculated bytes_radio. The remaining communication volume table of the working node 1 is 5000000 and 4000000, indicating that the remaining communication volumes of job1 and job2 are 5MB and 4MB respectively. Calculating bytes_radio gives 4 / 9. The increasing function selected is f(x) = 1.2x + 0.05. From this, the adjusted rate can be calculated.

[0068] It should be understood that although the steps in the flowchart of the accompanying drawings of the specification are shown in sequence according to the indication of the arrows, these steps do not necessarily have to be executed in the sequence indicated by the arrows. Unless there is a clear indication in this article, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings of the specification may include multiple steps or multiple stages. These steps or stages do not necessarily have to be executed at the same moment, but can be executed at different moments. The execution order of these steps or stages does not necessarily have to be sequential, but can be executed alternately or in turn with at least a part of the steps or stages in other steps or other steps.

[0069] In one embodiment, the present invention provides a computer device, which may be a server. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data used in the above method. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the above method is implemented.

[0070] In one embodiment, the present invention provides a computer-readable storage medium including instructions, such as a memory including instructions. The above instructions can be executed by a processor to complete the above method. The storage medium may be a computer-readable storage medium. For example, the computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0071] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0072] It is obvious to those skilled in the art that the present invention is not limited to the details of the above-described exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, in any regard, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Accordingly, all changes that fall within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present invention, and any reference signs in the claims should not be construed as limiting the claims involved.

Claims

1. A congestion control method for distributed model training, characterized in that, Including: Determine the worker nodes, switches, and parameter servers participating in the distributed model training, and use the programmable switch connected to the parameter server as the collaborative switch; Create a task remaining communication volume table in the worker nodes and create a task total communication volume table in the collaborative switch; Each worker node sends the local gradient data packet to the parameter server for aggregation, and the parameter server sends the globally aggregated gradient data packet obtained by aggregation to each worker node; When the collaborative switch receives the local gradient data packet, it updates the total communication volume that each training task needs to transmit recorded in the task total communication volume table according to the corresponding field; when the collaborative switch receives the global gradient data packet, it updates the remaining communication volume of each training task recorded in the task total communication volume table according to the corresponding field, and broadcasts it to each worker node, so that the worker node updates the remaining communication volume of each training task recorded in the task remaining communication volume table; When network congestion is detected, count the number S of training tasks with non-zero remaining communication volume. When S is greater than 1, assign a higher real-time ratio bytes_radio to the training task with lower remaining communication volume, and dynamically adjust the sending rate of the worker node corresponding to each training task through the increasing function f(bytes_radio).

2. The congestion control method for distributed model training according to claim 1, wherein The creating a task remaining communication volume table in the worker nodes and creating a task total communication volume table in the collaborative switch specifically includes: Create a 1-row and n-column task remaining communication volume table in the worker nodes to record the remaining communication volume of each training task in the current round of iterative communication phase; n is the total number of training tasks; Create a 2-row and n-column task total communication volume table in the collaborative switch to record the total communication volume and the remaining communication volume that each training task needs to transmit in the current round of iterative communication phase; Both the task remaining communication volume table and the task total communication volume table are indexed by the training task number and initialized to 0.

3. The congestion control method for distributed model training according to claim 1, characterized in that, When the collaborative switch receives the local gradient data packet, the updating the total communication volume that each training task needs to transmit recorded in the task total communication volume table according to the corresponding field specifically includes: The local gradient data includes a total communication volume field total_bytes, and the collaborative switch updates the total communication volume that each training task needs to transmit in the task total communication volume table according to the content of the total communication volume field total_bytes of the local gradient data packet.

4. The congestion control method for distributed model training according to claim 1, characterized in that When the collaborative switch receives the global gradient data packet, the updating the remaining communication volume of each training task recorded in the task total communication volume table according to the corresponding field, and broadcasting it to each worker node, so that the worker node updates the remaining communication volume of each training task recorded in the task remaining communication volume table specifically includes: The global gradient data includes a communication volume field psize and a remaining communication volume field remain_bytes. The communication volume field psize is used to record the size of the globally aggregated gradient data carried in the global gradient data packet; The collaborative switch calculates the latest remaining traffic volume in the current communication phase of the current training task through the content of the traffic volume field psize and the remaining traffic volume recorded in the total task traffic volume table, updates the total task traffic volume table, and writes it into the remaining traffic volume field remain_bytes of the global gradient data packet, and then forwards it to the worker node.

5. The congestion control method for distributed model training according to claim 1, wherein A real-time ratio bytes_radio with a higher value is assigned to a training task with a lower remaining traffic volume, and the sending rate of the worker node corresponding to each training task is dynamically adjusted through an increasing function f(bytes_radio), specifically including: Sort the remaining traffic volumes of each training task from smallest to largest to obtain the ranking of each training task; calculate the proportion of the remaining traffic volume of each training task in the total remaining traffic volume of all training tasks, and sort from largest to smallest to obtain the remaining traffic volume ratio ranking of each training task; use the remaining traffic volume ratio of the i-th ranking as the real-time ratio bytes_radio of the training task of the i-th ranking; Use the increasing function f and bytes_radio to adjust the existing sending rate cur_rate to obtain a new sending rate cur_rate'; cur_rate' = cur_rate * f(bytes_radio).

6. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 5.

7. A computer-readable storage medium, on which a computer program is stored, characterized in that, When the computer program is executed by the processor, it implements the steps of the method described in any one of claims 1 to 5.

Images