Load balancing method and system for distributed deep learning training

Abstract

The invention relates to a load balancing method and system for distributed deep learning training, and the method comprises the steps: analyzing the head information of a data packet in a switch, extracting a gradient index corresponding to the data packet, and dividing the data packets of the same gradient index from different training nodes into the same batch according to the gradient index; constructing a candidate path table based on the real-time queue length of the switch; distributing the same transmission path for the data packets in the same batch based on the Hash operation according to the candidate path table; the data packets in the same batch are continuously transmitted to the receiving end through the same transmission path; and carrying out instant aggregation on the received data packets through a receiving end according to the batch identifiers and the gradient indexes. The same batch of gradient data packets are ensured to continuously arrive at a receiving end through path selection based on gradient indexes and a transmission mechanism irrelevant to a sequence, so that instant aggregation and rapid synchronization of the gradient data packets are realized, and meanwhile, the utilization rate of network resources is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of communication networks, and particularly relates to a load balancing method and system for distributed deep learning training. Background Art

[0002] Distributed deep neural network training has become an important basic technology for machine learning applications, and is widely used in fields such as edge computing, computer vision, and recommendation systems. With the rapid improvement of the computing capabilities of hardware such as GPUs and TPUs, the demand for multi-node parallel training of large-scale models continues to grow. However, distributed training relies on collective communication operations to synchronize intermediate results, and such operations need to be repeatedly executed in hundreds of training iterations, which is extremely likely to cause network congestion and long-tail latency, seriously restricting the improvement of training efficiency.

[0003] To optimize the utilization rate of network resources, various load balancing mechanisms have been proposed in traditional data center networks. The flow-granularity-based scheme fixes the transmission path through hash mapping, but it is prone to load imbalance due to hash conflicts when the traffic distribution is skewed; the packet-granularity-based scheme can achieve approximately optimal traffic allocation, but it causes serious out-of-order problems due to fine-grained scheduling; the packet-cluster-granularity-based scheme is limited by the RDMA network traffic shaping mechanism and it is difficult to accurately identify the inter-packet gap to form packet clusters. If the above load balancing schemes are directly applied, although the completion time of a single flow can be optimized, the timing constraint relationship of synchronous communication in the field of distributed training is ignored. Specifically, the worker nodes training the same task pack gradients in the same format, and each data packet contains gradients corresponding to an index. We define all data packets with the same index, training the same model, and coming from different training nodes as a data packet batch. To accelerate the training process, we hope that the data packets in the same batch can reach the aggregator synchronously so that gradient aggregation can be performed in a timely manner. When the data packets within the same batch are rerouted to paths with different bandwidths or congestion levels, the arrival time difference will cause the aggregation process to be blocked by the slowest data packet, significantly increasing the training duration.

[0004] To reduce communication overhead, various solutions have been proposed in existing research. Some solutions focus on scheduling and accelerate the communication pipeline by overlapping the communication and computing phases. Gradient compression schemes utilize the sparsity of gradient data to reduce the amount of transmitted data through quantization or sparsification. In-network aggregation offloads part of the computing tasks to programmable switches. Although these solutions can generally shorten the overall training time, they are vulnerable to network congestion caused by the bursty traffic patterns of distributed training, resulting in poor training performance. In addition, existing solutions usually assume a symmetric network topology. In a heterogeneous cluster, low-speed links may become bottlenecks, and the difference in bandwidth will further increase the aggregation latency.

[0005] Therefore, in the scenario of distributed deep neural network training, how to design a load balancing mechanism to sense the batch synchronization requirements and avoid the tailing situation caused by path bandwidth differences, and balance fine-grained load balancing and out-of-order control, so as to ensure the timeliness and reliability of synchronous communication, is a key issue worthy of research. Summary of the Invention

[0006] To solve the above technical problems, the present invention provides a load balancing method and system for distributed deep learning training. Through gradient index-based path selection and order-independent transmission mechanisms, it ensures that gradient data packets of the same batch arrive at the receiving end continuously, thereby realizing instant aggregation and fast synchronization of gradient data packets, while improving the utilization rate of network resources.

[0007] The technical solution adopted by the present invention is as follows:

[0008] In a first aspect, a load balancing method for distributed deep learning training is provided, including:

[0009] Analyze the header information of the data packets in the switch, extract the gradient index corresponding to the data packets, and divide the data packets with the same gradient index from different training nodes into the same batch;

[0010] Dynamically construct a candidate path table based on the real-time queue length of the switch. The candidate path table includes multiple transmission paths with the lowest load rankings;

[0011] According to the candidate path table, allocate the same transmission path for the data packets within the same batch based on hash operation, so that the data packets within the same batch are continuously transmitted to the receiving end through the same transmission path;

[0012] Instantly aggregate the received data packets at the receiving end according to the batch identifier and gradient index.

[0013] Further, dynamically constructing a candidate path table based on the real-time queue length of the switch includes:

[0014] Every fixed detection period, detect the real-time queue length of each port of the switch. Under dynamic network conditions, rank according to the order of the real-time queue length, and select the top n transmission paths with the smallest real-time queue length as candidate paths;

[0015] Form a candidate path table according to the n candidate paths.

[0016] Further, after forming a candidate path table according to the n candidate paths, it further includes:

[0017] When the number of concurrent training nodes increases, expand the number n of candidate paths to disperse the traffic;

[0018] When the network traffic fluctuates or the path load is unbalanced, reduce the number n of candidate paths, so as to concentrate on selecting better paths.

[0019] Furthermore, the receiving end immediately aggregates the received data packets according to the batch identifier and the gradient index, including:

[0020] The receiving end creates an independent buffer for each data packet batch;

[0021] The independent buffer records the number of data packets that have arrived with the same gradient index;

[0022] When all the data packets with the same batch identifier have arrived, immediately trigger the aggregation operation for immediate aggregation;

[0023] When not all the data packets with the same batch identifier have arrived, and the arrival interval time between adjacent data packets within the same batch exceeds the timeout threshold, send a selective retransmission request.

[0024] Furthermore, the method further includes:

[0025] The receiving end dynamically sets the timeout threshold, and the specific process is as follows:

[0026] The receiving end obtains the time sensitivity of the specific deep learning training task, as well as the path delay of the path selected by the data packets in the same batch, and sets the timeout threshold based on the path delay.

[0027] Furthermore, according to the candidate path table, the same transmission path is allocated to the data packets within the same batch based on the hash operation, including:

[0028] Parse the packet header of the data packets within the same batch to obtain the gradient index;

[0029] Use a register to record the path selection result of the first data packet within the batch corresponding to each gradient index, determine the target transmission path, and subsequent data packets directly reuse the same target transmission path;

[0030] Synchronize the queue status of the target transmission path to the entry pipeline through a loopback operation to support real-time path status update.

[0031] Furthermore, the loopback operation includes:

[0032] Periodically generate probe data packets, carrying the queue length and bandwidth utilization rate of the current port;

[0033] After the probe data packets are processed by the exit pipeline, re-enter the entry pipeline to update the status information of the transmission path in the candidate path table.

[0034] In a second aspect, a load balancing system for distributed deep learning training is provided, including:

[0035] An index marking module, configured to parse the header information of data packets in a switch, extract the gradient index corresponding to the data packets, and divide the data packets with the same gradient index from different training nodes into the same batch according to the gradient index;

[0036] A path table construction module, configured to dynamically construct a candidate path table based on the real-time queue length of the switch, where the candidate path table includes multiple transmission paths with the lowest load ranking;

[0037] A path decision module, configured to allocate the same transmission path to the data packets within the same batch based on the candidate path table and based on hash operation, so that the data packets within the same batch are continuously transmitted to the receiving end through the same transmission path;

[0038] An aggregation execution module, configured to perform immediate aggregation on the received data packets by the receiving end according to the batch identifier and the gradient index.

[0039] Furthermore, the path table construction module is specifically configured to detect the real-time queue length of each port of the switch every fixed detection period, rank in the order of the real-time queue length under dynamic network conditions, and select the top n transmission paths with the smallest real-time queue length as candidate paths; form a candidate path table according to the n candidate paths.

[0040] Furthermore, the aggregation execution module is specifically configured to create an independent buffer for each data packet batch by the receiving end; record the number of data packets that have arrived with the same gradient index through the independent buffer; immediately trigger an aggregation operation for immediate aggregation when all the data packets with the same batch identifier have arrived; send a selective retransmission request when all the data packets with the same batch identifier have not arrived and the arrival interval time between adjacent data packets within the same batch exceeds the timeout threshold.

[0041] The beneficial effects achieved by the present invention:

[0042] Parse the header information of data packets in a switch, extract the gradient index corresponding to the data packets, and divide the data packets with the same gradient index from different training nodes into the same batch according to the gradient index; dynamically construct a candidate path table based on the real-time queue length of the switch, where the candidate path table includes multiple transmission paths with the lowest load ranking; allocate the same transmission path to the data packets within the same batch based on the candidate path table and based on hash operation; make the data packets within the same batch be continuously transmitted to the receiving end through the same transmission path; perform immediate aggregation on the received data packets by the receiving end according to the batch identifier and the gradient index. Through the gradient index-based path selection and order-independent transmission mechanism, it is ensured that the gradient data packets of the same batch continuously arrive at the receiving end, thereby realizing the immediate aggregation and fast synchronization of the gradient data packets, and at the same time improving the utilization rate of network resources. Description of the Drawings

[0043] Figure 1 It is a flowchart of the load balancing method for distributed deep learning training according to the present invention;

[0044] Figure 2 It is a flowchart of path selection based on gradient index according to the present invention;

[0045] Figure 3 It is a structural diagram of the load balancing system for distributed deep learning training according to the present invention. Detailed Embodiments

[0046] The present invention will be further described below with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and cannot be used to limit the protection scope of the present invention.

[0047] As Figure 1 shown, an embodiment of the present invention provides a load balancing method for distributed deep learning training, including:

[0048] 101. Analyze the header information of the data packets in the switch, extract the gradient index corresponding to the data packets, and divide the data packets with the same gradient index from different training nodes into the same batch;

[0049] In the data packets of the switch, first analyze the header information of the data packets. There is a gradient index in the header information. Divide the data packets with the same gradient index from different training nodes into the same batch.

[0050] 102. Dynamically construct a candidate path table based on the real-time queue length of the switch. The candidate path table includes multiple transmission paths with the lowest load rankings;

[0051] Every fixed detection period, detect the real-time queue length of each port of the switch. Under dynamic network conditions, rank according to the order of the real-time queue length, and select the top n transmission paths with the smallest real-time queue length as candidate paths; form a candidate path table according to the n candidate paths; this mechanism filters paths through the local information of the switch (such as the queue length and arrival traffic rate of each port) to ensure efficient adaptation to bandwidth heterogeneity under dynamic network conditions; each time it is updated, the candidate paths are ranked according to the order of the queuing length, and the top n paths with the lightest load are selected;

[0052] The number of paths n in the candidate path table needs to be dynamically adjusted according to the number of concurrent training nodes, network traffic, and path load. Specifically:

[0053] When the number of concurrent training nodes increases, expand the number of candidate paths n to disperse the traffic;

[0054] When the network traffic fluctuates or the path load is unbalanced, reduce the number n of candidate paths, so as to concentrate on selecting better paths.

[0055] 103. According to the candidate path table, based on the hash operation, allocate the same transmission path for the data packets within the same batch, so that the data packets within the same batch are continuously transmitted to the receiving end through the same transmission path;

[0056] Through the synchronous batch path binding mechanism, hash the gradient index of each batch of data packets to ensure that all data packets belonging to the same batch are routed to the same path, thereby reducing the tail effect caused by path asymmetry; in this mechanism, the transmission path of the data packets is dynamically selected through hash calculation at the Ingress stage of the switch, and the update of the path selection is based on the queue length information of the link; specifically, the path selection is dynamically adjusted according to the current load condition and link state of each path, thereby avoiding the performance bottleneck caused by path imbalance in the traditional load balancing scheme;

[0057] Parse the packet headers of the data packets within the same batch to obtain the gradient index; use a register to record the path selection result of the first data packet within the batch corresponding to each gradient index to determine the target transmission path; subsequent data packets directly reuse the same target transmission path; synchronize the queue state of the target transmission path to the ingress pipeline through a loopback operation to support real-time path decision-making;

[0058] The specific content of the loopback operation is as follows:

[0059] Periodically generate probe data packets, carrying the queue length and bandwidth utilization rate of the current port;

[0060] After the probe data packets are processed by the egress pipeline, re-enter the ingress pipeline to update the status information of the transmission path in the candidate path table.

[0061] During the implementation process, the path selection and traffic scheduling are implemented on the switch using the P4 programming language; specifically, the action matching table of the P4 switch is used to process the hash calculation and path selection of the gradient index to ensure that the gradient data packets belonging to the same batch always follow the same transmission path; in the P4 switch, in order to ensure the consistency of the hash selection, registers are used to record the path selection information and dynamically update the path according to different gradient indexes; the aggregation dependency relationship of the data packets is accurately recorded in these registers, thereby avoiding hash collisions and traffic congestion in the traditional scheme;

[0062] Data packets within the same batch are continuously transmitted to the receiving end via the same transmission path, adopting a transmission mechanism independent of sequence numbers. This mechanism can effectively handle the problem of out-of-order data packets. In traditional transmission schemes, data packets need to arrive in order before the gradient aggregation process can be carried out, and asynchronous arrivals will cause additional delays. By shielding the receiving end's inspection of data packet sequence numbers, the receiving end can ignore the arrival order of data packets during aggregation, ensuring the efficiency of the training process. This design effectively solves the problem of out-of-order data packets caused by the randomness of the hash routing method and improves the stability of data transmission.

[0063] 104. Immediately aggregate the data packets received by the receiving end according to the batch identifier and gradient index.

[0064] Create an independent buffer for each data packet batch by the receiving end; record the number of data packets that have arrived with the same gradient index through the independent buffer; when all data packets with the same batch identifier have arrived, immediately trigger the aggregation operation for immediate aggregation; when not all data packets with the same batch identifier have arrived and the arrival interval between adjacent data packets within the same batch exceeds the timeout threshold, send a selective retransmission request; based on the timeout-based packet loss detection mechanism, packet loss can be detected in a timely manner and retransmission can be executed; when the receiving end detects that a data packet in a certain batch is lost, it will immediately request the sending end to retransmit the lost data packet through the selective retransmission protocol, thus ensuring the integrity of data during the training process; to improve the efficiency of packet loss detection, a dynamic timeout threshold is adopted, which is flexibly adjusted according to the network state and the specific requirements of the training task.

[0065] The setting of the timeout threshold is obtained through experience. Specifically, based on the time sensitivity of the task in deep learning training and the path delay of the paths selected by data packets in the same batch, the timeout threshold is set based on the path delay to ensure that packet loss events can be detected in a timely manner, thereby effectively triggering selective retransmission. In addition, the detection mechanism will send periodic probe signals to ensure that the packet order between layers of the deep learning model is not affected and to accelerate pipeline processing.

[0066] The aggregation operation at the receiving end is compatible with existing transmission protocols. Specifically:

[0067] Bypass the default sequence number check in the Transmission Control Protocol (TCP) or Remote Direct Memory Access (RDMA) protocol stack, directly extract the gradient index and store it in the buffer; through the custom Acknowledge character (ACK) mechanism, after the receiving end completes the aggregation operation, notify the sending end to complete the aggregation to avoid triggering redundant retransmissions.

[0068] Advantages achieved by the embodiments of the present invention:

[0069] In the load balancing method for distributed deep learning training in the present invention, a combination of path selection, a transmission mechanism independent of sequence numbers, and a packet loss detection and retransmission mechanism is designed to achieve efficient gradient aggregation and fast training; by avoiding the negative impacts brought by path conflicts and out-of-order, efficient communication between nodes during the training process is ensured, thereby significantly improving the overall training performance.

[0070] It should be noted that before path selection and traffic scheduling, an initialization operation is also included, and the initialization operation includes clearing the register recording path selection, specifically:

[0071] Set the values of all path selection registers to empty to ensure the initialization state of the path selection table.

[0072] In some embodiments, a query operation is also included, specifically:

[0073] Query the selection record of the current path according to the gradient index and batch identifier; if the data packet has selected a path, query its path record in the path selection register, and if it exists, return the corresponding path; otherwise, reselect the optimal path through the path selection algorithm and return it.

[0074] Combined with Figure 1 the embodiments shown, the following is a specific implementation to illustrate the working flowchart of path selection based on the gradient index when performing the update operation as shown in Figure 2 shown, including:

[0075] 201, Update the candidate path table according to the real-time queue length of the switch;

[0076] 202, Receive the data packet, and obtain the training task number (job_ID) and gradient index sequence number (index) of the data packet;

[0077] 203, Calculate the hash value according to the gradient index, and map the data packet to a path in the candidate path set;

[0078] 204, Perform routing and forwarding according to the hash decision;

[0079] Judge whether the data packet is the first data packet in the batch; if so, select the current optimal path and record it in the corresponding register, and after sending the data packet, execute step 205; if it is not the first data packet, forward the data packet to the same path according to the previously recorded path information, and then return to step 201;

[0080] 205, The receiving end receives the data packets in an unordered manner;

[0081] 206, Determine whether the interval between data packets in the same batch is greater than the timeout threshold;

[0082] If timeout occurs within the batch, execute step 207; if all data packets in the current batch have arrived, execute step 208;

[0083] 207, Selective retransmission;

[0084] 208, Perform gradient aggregation.

[0085] Combined with the load balancing method for distributed deep learning training described in the above embodiments, a load balancing system for distributed deep learning training will be described below through embodiments.

[0086] As Figure 3 shown, the embodiment of the present invention provides a load balancing system for distributed deep learning training, including:

[0087] Index marking module 301, used to parse the header information of data packets in the switch, extract the gradient index corresponding to the data packet, and divide the data packets with the same gradient index from different training nodes into the same batch according to the gradient index;

[0088] Path table construction module 302, used to dynamically construct a candidate path table based on the real-time queue length of the switch, and the candidate path table includes multiple transmission paths with the lowest load ranking;

[0089] Path decision module 303, used to allocate the same transmission path for the data packets in the same batch based on the candidate path table through hash operation, so that the data packets in the same batch are continuously transmitted to the receiving end through the same transmission path;

[0090] Aggregation execution module 304, used to immediately aggregate the received data packets at the receiving end according to the batch identifier and gradient index.

[0091] Preferably, the path table construction module 302 is specifically used to detect the queue length of each port of the switch every fixed detection period, rank in the order of the queue length under dynamic network conditions, and select the top n transmission paths with the smallest queue length as candidate paths; form a candidate path table according to the n candidate paths.

[0092] Preferably, the aggregation execution module 304 is specifically configured to create an independent buffer for each data packet batch through the receiving end; record the number of data packets that have arrived with the same gradient index through the independent buffer; immediately trigger an aggregation operation for immediate aggregation when all the data packets with the same batch identifier have arrived; and send a selective retransmission request when not all the data packets with the same batch identifier have arrived and the arrival interval time between adjacent data packets within the same batch exceeds the timeout threshold.

[0093] The beneficial effects achieved by the embodiments of the present invention are as follows:

[0094] The index marking module 301 is used to parse the header information of the data packets in the switch, extract the gradient index corresponding to the data packets, and divide the data packets with the same gradient index from different training nodes into the same batch according to the gradient index; the path table construction module 302 is used to dynamically construct a candidate path table based on the real-time queue length of the switch, and the candidate path table includes multiple transmission paths with the lowest load ranking; the path decision module 303 is used to allocate the same transmission path for the data packets within the same batch based on the candidate path table through hash operation, so that the data packets within the same batch are continuously transmitted to the receiving end through the same transmission path; the aggregation execution module 304 is used to perform immediate aggregation on the received data packets according to the batch identifier and the gradient index through the receiving end. By combining the path selection, the transmission mechanism without irrelevant sequence numbers, and the packet loss detection and retransmission mechanism, efficient gradient aggregation and fast training are realized; by avoiding the negative impacts brought by path conflicts and out-of-order, the efficient communication between each node during the training process is ensured, thereby significantly improving the overall training performance.

[0095] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as the combination of flows and / or blocks in the flowcharts and / or block diagrams, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate for realizing the process Figure 1one process or multiple processes and / or boxes Figure 1 means for the functions specified in one box or multiple boxes.

[0097] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured article including an instruction device, and the instruction device implements the process Figure 1 one process or multiple processes and / or boxes Figure 1 the functions specified in one box or multiple boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for implementing the process Figure 1 one process or multiple processes and / or boxes Figure 1 the steps of the functions specified in one box or multiple boxes.

[0099] The above are only embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included within the scope of the claims of the present invention pending approval of the application.

Claims

1. A load balancing method for distributed deep learning training, characterized in that Including: Analyze the header information of the data packets in the switch, extract the gradient index corresponding to the data packets, and divide the data packets with the same gradient index from different training nodes into the same batch according to the gradient index; Dynamically construct a candidate path table based on the real-time queue length of the switch, where the candidate path table contains multiple transmission paths with the lowest load rankings; According to the candidate path table, allocate the same transmission path for the data packets within the same batch based on hash operation, so that the data packets within the same batch are continuously transmitted to the receiving end through the same transmission path; Instantly aggregate the received data packets at the receiving end according to the batch identifier and the gradient index.

2. The load balancing method for distributed deep learning training according to claim 1, characterized in that The dynamically constructing a candidate path table based on the real-time queue length of the switch includes: Every fixed detection period, detect the real-time queue length of each port of the switch, rank them in the order of the real-time queue length under dynamic network conditions, and select the top n transmission paths with the smallest real-time queue length as candidate paths; Form a candidate path table according to the n candidate paths.

3. The load balancing method for distributed deep learning training according to claim 2, wherein After forming the candidate path table according to the n candidate paths, it further includes: When the number of concurrent training nodes increases, expand the number n of candidate paths to disperse the traffic; When the network traffic fluctuates or the path load is unbalanced, reduce the number n of candidate paths to centrally select better paths.

4. The load balancing method for distributed deep learning training according to claim 1, wherein The instantly aggregating the received data packets at the receiving end according to the batch identifier and the gradient index includes: Create an independent buffer for each data packet batch at the receiving end; Record the number of data packets that have arrived with the same gradient index through the independent buffer; When all the data packets with the same batch identifier have arrived, immediately trigger the aggregation operation for instant aggregation; When not all the data packets with the same batch identifier have arrived, and the arrival interval time between adjacent data packets within the same batch exceeds the timeout threshold, send a selective retransmission request.

5. The load balancing method for distributed deep learning training according to claim 4, wherein The method further includes: The receiving end dynamically sets the timeout threshold, and the specific process is: The receiving end obtains the time sensitivity of the specific deep learning training task, as well as the path delay of the selected path of the data packets in the same batch, and sets the timeout threshold based on the path delay.

6. The load balancing method for distributed deep learning training according to claim 1, characterized in that, The allocating the same transmission path for the data packets within the same batch based on hash operation according to the candidate path table includes: Analyze the packet header of the data packets within the same batch to obtain the gradient index; Use a register to record the path selection result of the first data packet within the batch corresponding to each gradient index, determine the target transmission path, and subsequent data packets directly reuse the same target transmission path; Synchronize the queue state of the target transmission path to the entry pipeline through a loopback operation to support real-time path state update.

7. The load balancing method for distributed deep learning training according to claim 6, wherein The loopback operation includes: Periodically generate probe data packets, carrying the queue length and bandwidth utilization rate of the current port; After the probe data packets are processed by the exit pipeline, re-enter the entry pipeline to update the status information of the transmission paths in the candidate path table.

8. A load balancing system for distributed deep learning training, characterized in that, Including: An index marking module, which is used to parse the header information of data packets in a switch, extract the gradient index corresponding to the data packets, and divide the data packets with the same gradient index from different training nodes into the same batch according to the gradient index; A path table construction module, which is used to dynamically construct a candidate path table based on the real-time queue length of the switch, and the candidate path table includes multiple transmission paths with the lowest load ranking; A path decision module, which is used to allocate the same transmission path for the data packets in the same batch based on hash operation according to the candidate path table, so that the data packets in the same batch are continuously transmitted to the receiving end through the same transmission path; An aggregation execution module, which is used to immediately aggregate the received data packets by the receiving end according to the batch identifier and the gradient index.

9. The load balancing system for distributed deep learning training according to claim 8, wherein the path table construction module is specifically configured to detect the real-time queue length of each port of the switch every fixed detection period, rank in the order of the real-time queue length under dynamic network conditions, select the first n transmission paths with the smallest real-time queue length as candidate paths, and form a candidate path table according to the n candidate paths.

10. The load balancing system for distributed deep learning training according to claim 8, wherein the aggregation execution module is specifically configured to create an independent buffer for each data packet batch through the receiving end; record the number of data packets with the same gradient index that have arrived through the independent buffer; immediately trigger an aggregation operation for immediate aggregation when all the data packets with the same batch identifier have arrived; and send a selective retransmission request when all the data packets with the same batch identifier have not arrived and the arrival interval time between adjacent data packets in the same batch exceeds the timeout threshold.

Images