An adaptive data transmission method for distributed machine learning

Abstract

The application discloses a kind of adaptive data transmission methods for distributed machine learning, comprising: sending end constructs gradient block tensor, sets index and importance label for gradient block, constructs gradient block importance bitmap, initiates data transmission request to receiving end;Receiving end feeds back batch size N to sending end;Sending end uses non-blocking sliding window to order sliding on gradient block tensor, every time the gradient block in window and its index are packed into data packet and sent to receiving end, until all are sent;Receiving end feeds back second ACK information (first index bitmap+target index) to sending end after receiving and confirming N data packets;Sending end updates starting sliding position according to target index, and retransmits gradient block according to gradient block importance bitmap and first index bitmap.The application completely solves the long tail delay problem in distributed machine learning, and significantly reduces communication time.

Description

Technical Field

[0001] This invention belongs to the field of machine learning, specifically relating to an adaptive data transmission method for distributed machine learning. Background Technology

[0002] With the rapid increase in the scale of datasets and the complexity of neural networks in the field of machine learning (ML), distributed machine learning (DML) has become the mainstream model training method in the industry. In the training process of distributed machine learning, the Bulk Synchronous Parallel (BSP) algorithm is widely used due to its excellent convergence performance. BSP requires all nodes participating in the training to synchronize parameter updates after each iteration to ensure that the model parameters on each node remain consistent, thereby achieving common convergence of the model. In order to achieve efficient parameter synchronization between training nodes, the industry has proposed distributed machine learning communication architectures such as Parameter Server (PS[2]) architecture and Ring All-Reduce[3] architecture, which have significantly improved communication efficiency.

[0003] Ideally, distributed machine learning can achieve speedup proportional to cluster size. However, typically, training nodes complete each iteration almost simultaneously and enter the parameter synchronization phase, leading to a surge in instantaneous communication traffic. This causes massive network congestion, resulting in a sharp increase in packet loss. The endpoints then have to frequently retransmit, thus extending the tail stream completion time. This creates severe long-tail latency, a communication bottleneck, and significantly hinders overall training efficiency.

[0004] To improve communication efficiency in distributed machine learning, the industry has proposed many solutions, such as gradient compression, in-network aggregation, and computation-communication overlap scheduling. Below is a brief introduction to these three technologies:

[0005] 1. Gradient compression: To reduce the proportion of communication time, a direct and effective way is to compress the gradient to be transmitted, thereby reducing the amount of communication. Currently, there are two main schemes for gradient compression: quantization and sparsification.

[0006] a) Quantization Compression: This scheme discretizes the continuous gradient values ​​trained during iterations and maps them to integers within different ranges, converting them into low-bit representations. However, because the difference between the quantized gradient and the original gradient is too large, the quantized gradient is usually a biased estimate of the original gradient, which makes the model convergence slower and results in a significant loss of accuracy. To address this issue, error feedback techniques are typically used to correct the impact of this loss on model convergence. Common compensation methods include gradient accumulation and momentum correction.

[0007] b) Sparsity Compression: The basic idea of ​​this approach is to only transmit gradient values ​​that play a crucial role in model updates, while ignoring those close to zero and unimportant gradient values. Compared to quantization techniques, sparsity has the potential to achieve higher compression ratios. To ensure model convergence, sparsity typically operates on gradient residuals (locally accumulated historical gradients) rather than directly modifying the current gradient. Improvements such as momentum correction, local gradient clipping, and pre-training can further enhance the accuracy of sparsity compression.

[0008] In summary, gradient compression can significantly reduce communication volume, but since it only improves the application layer, it is difficult to solve the long-tail latency problem caused by network congestion, and there are many problems with training accuracy. Different correction strategies are required for different models, resulting in poor general applicability.

[0009] 2. On-Network Computing: This method primarily offloads the computational load to the network, utilizing programmable data plane switches within the network to perform computational tasks such as gradient aggregation, thereby reducing end-to-end communication and alleviating network congestion. Specifically, this scheme utilizes the built-in data plane processing unit of the programmable switch to perform pipelined aggregation of the gradient data carried by data packets as they pass through the switch. Because modern switches typically employ highly optimized dedicated hardware circuitry for their data planes, switches can complete simple arithmetic operations with significantly lower latency compared to CPUs on the endpoints.

[0010] In recent years, with the increasing power of programmable network hardware, in-network computing technology has gained widespread attention from academia and industry. For example, SwitchML implemented gradient aggregation with a 4-layer binary tree structure using Tofino switches, reducing aggregation latency from milliseconds to microseconds. ATP further improves the efficiency of in-network aggregation by jointly optimizing topology and routing, minimizing parameter synchronization latency.

[0011] While on-network computing can significantly reduce end-to-end communication latency, it also has some limitations. First, programmable switches have limited computing and storage resources, making it difficult to support large-scale parameter synchronization. Second, the data plane of switches typically uses fixed-point numbers rather than floating-point numbers for operations, which may introduce additional precision loss. Finally, distributed machine learning frameworks for on-network computing are still immature, and adapting them to existing frameworks is quite difficult.

[0012] 3. Computation-Communication Overlap Scheduling: During backpropagation, gradients are generated sequentially from the last layer to the first layer. Therefore, there is no need to wait for the computation of the previous layer to complete before transmitting the gradients of subsequent layers. In other words, the computation of the earlier layers and the communication of the later layers are independent of each other, and the parameter updates of the later layers are also unrelated to the earlier layers. Therefore, while calculating the gradient of the (L-1)th layer, the values ​​of the Lth layer can be transmitted simultaneously. In reality, computation-communication overlap does not actually shorten communication time, but rather parallelizes the computation and communication processes as much as possible. Based on this, the academic community has proposed a scheduling algorithm called Wait-Free Backpropagation (WFBP). WFBP enables each layer to start communicating immediately after its gradient is calculated during backpropagation, thereby maximizing the overlap between computation and communication and significantly improving parallel efficiency. However, computation-communication overlap scheduling also has some shortcomings. First, this algorithm is often optimized for specific network architectures (such as structures with parameterized servers), limiting its general applicability. Second, this algorithm is still optimized at the application layer, making it difficult to alleviate the long-tail latency problem caused by network congestion.

[0013] Furthermore, existing technologies include a differential gradient transmission mechanism for distributed machine learning, which aims to optimize communication performance by dynamically adjusting transmission reliability. This scheme uses TCP for important gradients and UDP for unimportant gradients. However, this strategy does not depart from the "all or nothing" semantics of traditional transport layer protocols, and therefore, long-tail latency issues may still occur for important gradients. In addition, this scheme is based on a parameter server (PS) network architecture, and the "incast" traffic pattern of this architecture can cause severe network congestion on the parameter server, leading to communication bottlenecks and making it unsuitable for large-scale deployments, thus exhibiting weak versatility.

[0014] In summary, how to fundamentally solve the long-tail latency problem, thereby reducing communication time and enabling the training model to achieve rapid convergence in fewer training rounds, is a technical problem that urgently needs to be solved. Summary of the Invention

[0015] To address the aforementioned problems in the existing technology, this invention provides an adaptive data transmission method and system for distributed machine learning.

[0016] The technical problem to be solved by this invention is achieved through the following technical solution:

[0017] An adaptive data transmission method for distributed machine learning is characterized in that, during the global gradient reduction process performed by distributed training nodes through the All-Reduce operation, any two training nodes act as communicating parties and use the adaptive data transmission method for data transmission; the communicating parties include a sender and a receiver; the adaptive data transmission method includes:

[0018] The originating end vectorizes the gradients generated during the backpropagation of the neural network model to obtain a gradient tensor; it divides the gradient tensor into blocks, constructs a gradient block tensor from the block results, and sets an index and importance label for each gradient block in the gradient block tensor; it constructs a gradient block importance bitmap based on the importance labels of each gradient block; and it initiates a data transmission request to the receiving end.

[0019] In response to the data transmission request, the receiving end sends a first ACK message back to the sending end; the first ACK message carries the batch size N of the data packets that the receiving end expects to receive and acknowledge in batches.

[0020] After receiving the batch size, the sending end uses a non-blocking sliding window to slide orderly on the gradient block tensor. After each slide, the gradient block and its index in the current non-blocking sliding window are packaged into a data packet and sent to the receiving end until the entire gradient block tensor has been sent. The number of gradient blocks in the non-blocking sliding window is N.

[0021] After receiving and acknowledging N data packets, the receiving end sends a second ACK message back to the sending end. The second ACK message carries a first index bitmap of the gradient blocks that the receiving end currently needs to receive but has not yet received, as well as a target index. The target index is used to indicate the cutoff position of a group of gradient blocks that the receiving end has received consecutively from the N data packets in the gradient block tensor.

[0022] After receiving the second ACK information, the sending end updates the starting sliding position of the non-blocking sliding window according to the target index, and performs gradient block retransmission according to the gradient block importance bitmap and the first index bitmap.

[0023] Optionally, the starting point, in the manner of setting importance labels for each gradient block, includes:

[0024] The average gradient of each gradient block in the gradient block tensor is computed in parallel.

[0025] Based on the average gradient, the TopK algorithm is used to sort the gradient blocks in the gradient block tensor to obtain the sorting result;

[0026] Based on the sorting results, determine whether each gradient block is important; if important, set a first label value as an importance label for the gradient block; if not important, set a second label value as an importance label for the gradient block.

[0027] Optionally, the index of the gradient block is represented as:<L,O,S> ;

[0028] Where L represents the number of the gradient matrix of the network layer corresponding to the gradient block in the gradient tensor, O is the layer offset, representing the starting position of the gradient block in the gradient matrix, and S represents the size of the gradient block.

[0029] Optionally, the size of the non-blocking sliding window is set according to the bandwidth-delay product.

[0030] Optionally, the sending end performs gradient block retransmission based on the gradient block importance bitmap and the first index bitmap, including:

[0031] The important gradient ratio is calculated based on the first index bitmap; the important gradient ratio is the proportion of important gradient blocks in the gradient blocks that need to be retransmitted this time.

[0032] The importance threshold for this retransmission is determined based on the important gradient ratio, the current iteration number of the neural network model, and the loss value of the previous iteration; the importance threshold is a threshold related to the important gradient ratio.

[0033] If the proportion of important gradients exceeds the importance threshold, then according to the gradient block importance bitmap, all gradient blocks that the receiving end currently needs to receive but has not yet received are retransmitted.

[0034] If the proportion of important gradients does not exceed the importance threshold and is higher than the preset lower limit, then according to the gradient block importance bitmap, all important gradient blocks that the receiving end currently needs to receive but has not received will be retransmitted.

[0035] If the significant gradient ratio is not higher than the lower limit of the ratio, then the gradient block retransmission is abandoned.

[0036] Optionally, the initiator determines the importance threshold for this retransmission based on the important gradient ratio, the current iteration number of the neural network model, and the loss value of the previous iteration, including:

[0037] Based on the importance gradient ratio, the current iteration number of the neural network model, and the loss value of the previous iteration, the importance threshold of the previous retransmission is dynamically adjusted using a preset threshold adjustment formula to obtain the importance threshold of the current retransmission; wherein, the importance threshold of the initial retransmission is a preset value.

[0038] The threshold adjustment formula is as follows:

[0039] RT(τ)=α·lastLoss+β·(1-PLR)+γ·τ+δ·RT(τ-1);

[0040] Where lastLoss represents the loss value of the previous iteration, PLR represents the important gradient ratio, τ represents the current iteration number, α is the first weight factor corresponding to lastLoss, β is the second weight factor corresponding to PLR, γ is the third weight factor corresponding to τ, δ is the fourth weight factor corresponding to RT(τ-1), RT(τ-1) is the importance threshold of the previous retransmission, and RT(τ) is the importance threshold of the current retransmission.

[0041] Optionally, the adaptive data transmission method further includes:

[0042] After completing the retransmission operation, the sending end sends a stop signal to the receiving end;

[0043] The receiving end responds to the stop signal to start timing and sends a third ACK message back to the transmitting end; the third ACK message carries a second index bitmap of the gradient block that the receiving end currently needs to receive but has not yet received, and also carries the expected timing end time.

[0044] After receiving the third ACK information, the transmitting end tries its best to retransmit the gradient block indicated by the second index bitmap to the receiving end, referring to the timeout end time.

[0045] Optionally, when the sending end retransmits the gradient block indicated by the second index bitmap, it prioritizes retransmitting the important gradient blocks indicated by the second index bitmap according to the gradient block importance bitmap.

[0046] Optionally, the data transmission request, the first ACK information, the second ACK information, the blocking signal, and the third ACK information are all carried in the data packet for transmission; wherein, the header of the data packet is provided with multiple flag bits to distinguish different ACK information.

[0047] Optionally, the batch size is preset based on network conditions and training tasks.

[0048] This invention proposes an adaptive data transmission method for distributed machine learning. From the perspective of the transport layer, considering that machine learning tasks already employ fault-tolerant algorithms like SGD (Stochastic Gradient Descent), the communication process can tolerate a certain degree of data loss, making completely reliable transmission unnecessary. Therefore, by re-examining the necessity of reliable transmission semantics, the packet loss retransmission strategy is improved, moving away from the "all or nothing" semantics of traditional transport layer protocols. This completely solves the long-tail latency problem in distributed machine learning, significantly reducing communication time. Furthermore, this invention fully utilizes the characteristics of gradients in data-parallel training, where the model parameter tensors of each training node have the same shape but different training data. It designs out-of-order sending and receiving functions, further reducing communication time. Therefore, using the adaptive data transmission method proposed in this invention, the training model can achieve rapid convergence in fewer training epochs.

[0049] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0050] Figure 1 The program modules involved in the adaptive data transmission method proposed in the embodiments of the present invention are shown;

[0051] Figure 2 This is a flowchart illustrating an adaptive data transmission method for distributed machine learning provided in an embodiment of the present invention.

[0052] Figure 3 This is a schematic diagram of the gradient block tensor and gradient block importance bitmap in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of using a non-blocking sliding window to slide on the gradient block tensor in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the first index bitmap and the first index bitmap in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the data packets used in the embodiments of the present invention;

[0056] Figure 7 and Figure 8 These are all schematic diagrams illustrating the interaction between the sending and receiving ends via data packets in embodiments of the present invention;

[0057] Figure 9 This is an example of implementing an in-network gradient sparse protocol stack in the connection dimension of the embodiments of the invention;

[0058] Figure 10This is an example of implementing an in-network gradient sparse protocol stack at the single-packet dimension in the embodiments of the invention. Detailed Implementation

[0059] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0060] In current distributed machine learning frameworks, the default transport layer protocol only supports "all or nothing" semantics, meaning it guarantees data arrival or does not guarantee delivery, representing two types of protocols: Reliable Data Protocol (RDP) and Unreliable Data Transfer (UDT). The inventors, taking a transport layer perspective, considered that machine learning tasks already employ fault-tolerant algorithms like Stochastic Gradient Descent (SGD), allowing for a certain degree of data loss during communication. Therefore, completely reliable transmission is not necessary. By re-examining the necessity of reliable transmission semantics, the inventors improved the packet retransmission strategy, moving away from the traditional "all or nothing" semantics of transport layer protocols. This completely solves the long-tail latency problem in distributed machine learning and significantly reduces communication time.

[0061] Specifically, this invention proposes an adaptive data transmission method for distributed machine learning. The network architecture based on this method does not include a parameter server; instead, distributed training nodes communicate with each other to perform data transmission and global gradient reduction, thereby achieving distributed learning. The specific learning process can be implemented based on the PyTorch deep learning framework. During the global gradient reduction process performed by the distributed training nodes through the All-Reduce operation, any two training nodes act as communicating parties, using the adaptive data transmission method proposed in this invention to transmit data; the communicating parties include a sender and a receiver.

[0062] The adaptive data transmission method proposed in this invention combines the idea of ​​gradient sparsity compression with elastic reliable semantics and retransmission strategies to achieve in-network gradient sparsity. This method can be implemented through several modular program modules, all of which are deployed on the training nodes. Figure 1The diagram illustrates the program modules involved in the adaptive data transmission method proposed in this embodiment of the invention, mainly including a gradient evaluation module, a communication control module, and an on-network gradient sparse protocol stack. The gradient evaluation module is responsible for gradient vectorization, gradient block partitioning, and gradient block sorting; the communication control module is responsible for training group initialization and the implementation of the All-Reduce communication architecture; and the on-network gradient sparse protocol stack is responsible for implementing the resilient and reliable protocol and retransmission strategy. Subsequent descriptions of the steps of the adaptive data transmission method proposed in this embodiment will use examples of these program modules.

[0063] Before training the neural network model, the communication control module is responsible for initializing the training group, implementing the All-Reduce communication architecture during training, and performing gradient reduction and gradient update during iteration. The specific implementation is as follows:

[0064] i. Training Group Initialization: Using the distributed communication library provided by the deep learning framework, the interface for initializing the process group is called, passing in initialization parameters, including the communication backend, the number of training nodes, and the current process ID. Synchronous training is achieved by blocking the entire process group and waiting for all training processes to establish connections.

[0065] ii. All-Reduce Communication Architecture: To efficiently reduce gradients between nodes, an All-Reduce communication architecture based on a ring topology is adopted. Specifically, assuming there are N training nodes, each process on each node holds a complete copy of the model. In each iteration, each node computes its own local gradient in parallel, and then the All-Reduce operation reduces the local gradients of each node to obtain the global gradient, which is then used to update the local model parameters. The All-Reduce implementation uses a ring algorithm, and the specific steps are as follows:

[0066] 1. Arrange N nodes into a logical ring.

[0067] 2. Divide the local gradient of each node into N parts.

[0068] 3. The i-th node sends its i-th gradient to the (i+1)%N-th node, and at the same time receives its (i-1+N)%N-th gradient from the (i-1+N)%N-th node.

[0069] 4. Each node accumulates the received gradient with its local corresponding gradient.

[0070] 5. Repeat steps 3-4 for a total of N-1 times, until each node obtains the global gradient.

[0071] In addition, to prevent process communication from blocking, the communication module also needs to determine the parity of the nodes. For odd-numbered training nodes, the gradient is sent first and then received, while for even-numbered training nodes, the gradient is received first and then sent to achieve synchronous communication.

[0072] The adaptive data transmission method proposed in this invention will be described in detail below with reference to specific steps. See also... Figure 2 The adaptive data transmission method proposed in this embodiment of the invention includes the following steps:

[0073] S10. The sending end vectorizes the gradients generated during the backpropagation of the neural network model to obtain a gradient tensor; it divides the gradient tensor into blocks, constructs the block results into gradient block tensors, and sets an index and importance label for each gradient block in the gradient block tensor; it constructs a gradient block importance bitmap based on the importance labels of each gradient block; and it initiates a data transmission request to the receiving end.

[0074] Specifically, the process begins by iterating through each parameter in the neural network model and using hook functions to monitor the parameter training status. Once the gradient of a parameter is calculated, it is immediately vectorized, thereby achieving an overlap with the backpropagation process and shortening the gradient vectorization time.

[0075] In this model, each network layer corresponds to a matrix in the gradient tensor. Multiple matrices can be ordered to form a gradient tensor. The global gradient is constructed as a gradient tensor. By dividing it into blocks, a gradient block tensor can be formed.

[0076] Since each training node uses the same algorithm to divide the gradient vector into blocks, the positions of gradient blocks in the gradient vector are one-to-one between different nodes. Leveraging this characteristic, a globally unique index can be assigned to each gradient block in the intermediate layer as its mapping identifier in the original gradient vector. This allows for the design of a more efficient communication mechanism, enabling out-of-order transmission and reception, and further reducing communication time.

[0077] See Figure 1 At the start of each iteration, the application layer sends metadata such as the gradient tensor shape and block partitioning scheme to the middleware layer via control messages. The middleware layer then generates an index array and sends it to the receiving middleware layer, thus synchronizing the gradient index mapping between the sending and receiving ends. Subsequently, when sending gradient blocks via data packets, the sending middleware layer appends the index corresponding to the gradient block to the packet header, allowing the receiving middleware layer to directly map the data to the correct location based on the index in the packet header. A single data packet may carry multiple gradient blocks, therefore multiple gradient indices may be appended to the packet header.

[0078] The gradient block index is represented as:<L,O,S> ;

[0079] Where L represents the number of the gradient matrix of the network layer corresponding to the gradient block in the gradient tensor, O is the in-layer offset, representing the starting position of the gradient block in the gradient matrix, and S represents the size of the gradient block.

[0080] In one implementation, the starting point sets an importance label for each gradient block, including:

[0081] (1) Calculate the average gradient of each gradient block in the gradient block tensor in parallel;

[0082] (2) Based on the average gradient, the TopK algorithm is used to sort the gradient blocks in the gradient block tensor to obtain the sorting result;

[0083] (3) Determine whether each gradient block is important based on the sorting results; if important, set a first label value as the importance label for the gradient block; if not important, set a second label value as the importance label for the gradient block. For example, the first label value can be 1 and the second label value can be 0, and it is not limited to this.

[0084] In this embodiment of the invention, the structures of the gradient block tensor and the gradient block importance bitmap can be found in [reference needed]. Figure 3 In the gradient block tensor, the letters and their indices represent the sorting numbers of the gradient blocks within the tensor, and the numbers in the gradient block importance bitmap represent the importance labels of the gradients.

[0085] Figure 1 In this model, a key function of the gradient evaluation module is to implement gradient vectorization and inverse vectorization. Inverse vectorization restores the updated gradient block tensor to its original form, facilitating gradient updates. Furthermore, the gradient evaluation module can clean up previously stored, unused gradients, preventing the model from exceeding the GPU memory limits of training nodes due to the continuous creation of vectorized gradients, thus avoiding training termination.

[0086] Specifically, when partitioning the gradient tensor into blocks, a tensor slicing method can be used for synchronous cutting instead of traversing the gradient, thus obtaining gradient block tensors. Secondly, during TopK sorting, the partitioned tensors are stacked into a single tensor using a tensor stacking method. Leveraging the parallel computing capabilities of the GPU, the average gradient is calculated simultaneously, thereby reducing the algorithm complexity to O(mlogn). Furthermore, the TopK algorithm selects the top p% of the gradient blocks in the sorting result as important gradient blocks, and the remaining blocks are considered unimportant, thus forming a gradient block importance bitmap. This bitmap and the gradient block tensor are then passed to the communication control module.

[0087] In order to achieve information sharing between the application layer, protocol stack, and middleware layer, the communication control module, in addition to performing the training process control function, also needs to undertake the task of information transmission, becoming the link between the application layer and the underlying layer.

[0088] Specifically, during each training iteration, the communication control module can obtain two parts of information from the gradient evaluation module: first, the vectorized global gradient after update and reduction; and second, the gradient block tensor and gradient block importance bitmap formed after block evaluation. Then, the communication module processes these two parts of information separately.

[0089] 1. Vectorized Global Gradient Propagation: The reduced vectorized global gradient is passed to the intermediate layer to guide packet loss decisions in the reliable out-of-order delivery mechanism. This allows the intermediate layer to fill in non-critical gradient blocks from lost packets based on gradient information from previous iterations, reducing the impact of gradient sparsity on accuracy. Since the global gradient aggregates gradient information from all nodes, reflecting the overall update direction of the current model, sharing it with the intermediate layer allows for a better assessment of the impact of dropping certain data packets on model convergence.

[0090] 2. Block Gradient Transmission: The gradient block tensor and gradient block importance bitmap formed after block evaluation are passed to the on-network gradient sparse protocol stack to guide the transmission of gradient blocks.

[0091] In practical applications, the communication control module can employ a message queue mechanism to acquire the aforementioned two parts of information. Specifically, the communication control module maintains two message queues: one for caching the vectorized global gradients generated in each iteration; and the other for caching the gradient block tensors and gradient block importance bitmaps formed after block evaluation. In each iteration, the corresponding gradient information is encapsulated into a message and delivered to the corresponding queue. The middleware layer and protocol stack can obtain the required information in a timely manner by subscribing to these two message queues, achieving end-side information sharing. Furthermore, considering that the message delivery and processing rates may not match, leading to queue accumulation, a periodic cleanup mechanism can be introduced to discard timed-out messages to avoid affecting subsequent iterations.

[0092] In this embodiment of the invention, the process of evaluating the importance of gradient blocks and setting importance labels is similar to gradient sparsity compression in existing technologies. The difference lies in that existing gradient sparsity compression sets unimportant gradients to zero to reduce communication overhead and requires multiple methods to correct accuracy. In this embodiment, however, after evaluation, gradient blocks are transmitted completely regardless of their importance. Furthermore, this embodiment uses precise retransmission strategies and flexible semantics to handle packet loss, thereby achieving efficient on-network gradient sparsity and significantly reducing long-tail latency while maintaining training accuracy. Specifically, the sending end initiates a data transmission request to the receiving end, which triggers... Figure 1The relevant functions of the in-network gradient sparsity protocol stack are described. This protocol stack mainly implements a custom transport layer protocol and uses mechanisms such as elastic reliable semantics and gradient retransmission strategy to achieve gradient sparsity in the network. This protocol can be named the Elastic Reliable Data Protocol (ERDP). Its main implementation method is to bypass the kernel protocol stack, which greatly improves communication efficiency, reduces latency, and achieves high-performance data reception and transmission, providing efficient data path support for in-network gradient sparsity.

[0093] S20. The receiving end responds to the data transmission request and sends the first ACK information back to the sending end; the first ACK information carries the batch size N of the data packets that the receiving end expects to receive and acknowledge.

[0094] The batch size can be preset according to the network conditions and training tasks.

[0095] S30. After receiving the batch size, the sending end uses a non-blocking sliding window to slide orderly on the gradient block tensor. After each slide, the gradient block and its index in the current non-blocking sliding window are packaged into a data packet and sent to the receiving end until all gradient block tensors are sent. The number of gradient blocks in the non-blocking sliding window is N.

[0096] For an example of the effect of using a non-blocking sliding window to slide over the gradient block tensor, see [link / reference]. Figure 4 .

[0097] In this embodiment of the invention, when the transmitting end uses a non-blocking sliding window to slide on the gradient block tensor, it does not require the receiving end to acknowledge the previously sent data packet before continuing to slide; instead, it can slide continuously, thus making full use of the bandwidth. The size of the non-blocking sliding window can be set according to the bandwidth-delay product, ensuring that the transmitting end's sending capacity matches the network's carrying capacity.

[0098] S40. After receiving and acknowledging N data packets, the receiving end sends a second ACK message back to the sending end. The second ACK message carries a first index bitmap of the gradient blocks that the receiving end currently needs to receive but has not yet received, and also carries a target index. The target index is used to indicate the cutoff position of a set of gradient blocks that the receiving end has received consecutively from N data packets in the gradient block tensor.

[0099] The first index bitmap is as follows: Figure 5 As shown, each bit corresponds to the reception state of a gradient block. Initially, all bits are 0, indicating that no gradient blocks have been received. When a gradient block is successfully received by the receiver, the corresponding bit is set to 1.

[0100] In distributed machine learning, the simultaneous generation of large bursts of traffic from various training nodes can lead to severe network congestion and transmission errors. This invention proposes a differentiated retransmission strategy based on receiver feedback using an ARQ (Automatic Repeat Request) mechanism to achieve gradient sparsity in the network. Specifically, the receiver acknowledges all N received data packets within that group. This reduces computational overhead at the receiving end, ensures timely retransmission triggering, and aligns with the training iteration rhythm, synchronizing communication convergence with model convergence. This fully leverages the bounded loss tolerance characteristic of gradients.

[0101] S50. After receiving the second ACK information, the sending end updates the starting sliding position of the non-blocking sliding window according to the target index, and performs gradient block retransmission according to the gradient block importance bitmap and the first index bitmap.

[0102] Specifically, after receiving the second ACK message, the sending end aligns the starting sliding position of the non-blocking sliding window to the target index, adds the gradient block indicated by the first index bitmap to the retransmission queue, and retransmits it at an opportune time.

[0103] The sending end performs gradient block retransmission based on the gradient block importance bitmap and the first index bitmap, including:

[0104] (1) Calculate the proportion of important gradients based on the first index bitmap; the proportion of important gradients is the proportion of important gradient blocks in the gradient blocks that need to be retransmitted this time.

[0105] (2) Determine the importance threshold for this retransmission based on the proportion of important gradients, the current iteration number of the neural network model, and the loss value of the previous iteration; the importance threshold is a threshold related to the proportion of important gradients.

[0106] (3-a) If the proportion of important gradients exceeds the importance threshold, then according to the gradient block importance bitmap, all gradient blocks that the receiving end needs to receive but has not yet received are retransmitted, referred to as all retransmissions.

[0107] (3-b) If the proportion of important gradients does not exceed the importance threshold and is higher than the preset lower limit, then according to the gradient block importance bitmap, all important gradient blocks that the receiving end needs to receive but has not received are retransmitted, referred to as partial retransmission.

[0108] (3-c) If the important gradient ratio is not higher than the lower limit of the ratio, then the gradient block is abandoned and retransmitted, which is called abandoning retransmission.

[0109] In step (2), the sending end determines the importance threshold for this retransmission based on the important gradient ratio, the current iteration number of the neural network model, and the loss value of the previous iteration. This includes: dynamically adjusting the importance threshold of the previous retransmission using a preset threshold adjustment formula based on the important gradient ratio, the current iteration number of the neural network model, and the loss value of the previous iteration to obtain the importance threshold for this retransmission; wherein the importance threshold for the initial retransmission is a preset value.

[0110] The above threshold adjustment formula is:

[0111] RT(τ)=α·lastLoss+β·(1-PLR)+γ·τ+δ·RT(τ-1);

[0112] Where lastLoss represents the loss value of the previous iteration, PLR represents the proportion of important gradients, τ represents the current iteration number, α is the first weight factor corresponding to lastLoss, β is the second weight factor corresponding to PLR, γ is the third weight factor corresponding to τ, δ is the fourth weight factor corresponding to RT(τ-1), RT(τ-1) is the importance threshold of the previous retransmission, and RT(τ) is the importance threshold of the current retransmission.

[0113] Understandably, when both lastLoss and τ are large, it indicates that the model has not yet converged after multiple iterations. In this case, the retransmission requirement can be relaxed appropriately, and the importance threshold can be increased to improve model accuracy. Conversely, when lastLoss is small, it indicates that the model is close to convergence. In this case, the retransmission requirement should be increased appropriately, and the importance threshold should be decreased to ensure reliable delivery of important gradients, thereby reducing retransmission overhead and accelerating training.

[0114] In actual training, a situation may arise where certain parameters converge rapidly within one or two iterations, causing their gradients to be deemed important in one iteration but unimportant in the next. This can lead to the loss of the parameter's gradient across multiple iterations, thus affecting its convergence performance. To address this issue, this embodiment of the invention considers lastLoss when dynamically adjusting the importance threshold. When a parameter's gradient is lost across multiple iterations, lastLoss increases, and the dynamically adjusted importance threshold decreases accordingly. This encourages the sender to choose a higher proportion of retransmissions, thereby resolving the previous problem.

[0115] In one embodiment, the adaptive data transmission method provided by this invention may further include the following steps:

[0116] (1) After the sending end completes the retransmission operation, it sends a stop signal to the receiving end;

[0117] (2) The receiving end responds to the stop signal to start timing and sends the third ACK information back to the transmitting end; the third ACK information carries the second index bitmap of the important gradient block that the receiving end needs to receive but has not yet received, and also carries the expected timing end time.

[0118] (3) After receiving the third ACK message, the transmitter tries its best to retransmit the gradient block indicated by the second index bitmap to the receiver, referring to the end time of the timing.

[0119] When the originating end retransmits the gradient block indicated by the second index bitmap, it prioritizes retransmitting the important gradient blocks indicated by the second index bitmap according to the gradient block importance bitmap.

[0120] The adaptive data transmission method for distributed machine learning proposed in this invention takes a transport layer perspective. Considering that machine learning tasks already employ fault-tolerant algorithms like SGD (Stochastic Gradient Descent), the communication process can tolerate a certain degree of data loss, making completely reliable transmission unnecessary. Therefore, by re-examining the necessity of reliable transmission semantics, the packet loss retransmission strategy is improved, moving away from the "all or nothing" semantics of traditional transport layer protocols. This completely solves the long-tail latency problem in distributed machine learning, significantly reducing communication time. Furthermore, this invention fully utilizes the characteristics of gradients in data-parallel training, where the model parameter tensors of each training node have the same shape but different training data. It designs out-of-order sending and receiving functions, further reducing communication time. Therefore, using the adaptive data transmission method proposed in this invention, the training model can achieve rapid convergence in fewer training rounds.

[0121] Furthermore, compared with the differential gradient transfer mechanism mentioned in the background art, the advantages of the embodiments of the present invention are mainly reflected in the following aspects:

[0122] 1. It fundamentally solves the long-tail latency problem. Differential gradient transmission mechanisms essentially alleviate network congestion by reducing communication volume, but they still rely on reliable transmission for critical gradients, failing to avoid long-tail latency. This invention, however, addresses the issue at the transport layer by introducing resilient reliable semantics and differentiated retransmission strategies, allowing less important gradients to be sparsified, thereby significantly reducing end-to-end latency and improving communication efficiency.

[0123] 2. Achieves more refined gradient importance determination. The differential gradient transfer mechanism simply classifies importance based on gradient magnitude, which is insufficient to handle sharp gradient changes in the later stages of training. However, this invention, through edge information sharing, adaptively adjusts the importance threshold, enabling more accurate and dynamic determination of critical gradients.

[0124] 3. A packet ACK mechanism is introduced. Unlike the packet-by-packet acknowledgment of the differential gradient transmission mechanism, this embodiment of the invention adopts a packet ACK strategy. The receiving end receives a group of N data packets before generating a second ACK message, which reduces the number of acknowledgments, lowers the processing overhead of the receiving end, and improves communication performance.

[0125] 4. It breaks the strict sequential delivery semantics. Differential gradient transmission mechanisms require strict sequential delivery to ensure data integrity, leading to frequent waiting for out-of-order packets at the receiving end and causing head-of-line (HOL) blocking. However, in this embodiment of the invention, since each gradient block has its own index, it supports out-of-order sending and receiving, thus introducing an out-of-order delivery mechanism based on gradient semantics. This allows out-of-order packets to be buffered first, reducing unnecessary waiting and resolving the HOL problem.

[0126] 5. End-to-end collaborative communication control is achieved. Differential gradient transmission mechanisms primarily focus on application layer optimization, failing to fully utilize the capabilities of the transport and intermediate layers. However, this invention, through end-side information sharing, enables the application layer, transport layer, and intermediate layer to exchange key information such as gradient information and retransmission strategies, forming end-to-end collaborative adaptive retransmission and delivery control, significantly improving communication performance.

[0127] 6. A high-performance user-space protocol stack has been implemented. Based on the DPDK (Data Plane Development Kit) framework, the transmission protocol in this embodiment of the invention bypasses the kernel protocol stack, significantly reducing packet processing latency and providing efficient data path support.

[0128] In summary, compared with the differential gradient transmission mechanism in existing technologies, the embodiments of this invention have achieved substantial improvements and enhancements in multiple aspects, including long-tail latency, discrimination accuracy, ACK overhead, sequential semantics, end-to-end collaboration, and high-performance data paths. Through innovative mechanisms such as network gradient sparsity and reliable out-of-order delivery, the convergence speed of distributed machine learning can be significantly accelerated, training time shortened, and communication performance maximized while ensuring model accuracy. This effectively balances model convergence speed and communication performance, providing strong support for large-scale distributed training.

[0129] In this embodiment of the invention, the data transmission request, the first ACK information, the second ACK information, the blocking signal, and the third ACK information are all carried in the data packet for transmission; wherein, the header of the data packet is provided with multiple flag bits to distinguish different ACK information.

[0130] Specifically, this embodiment of the invention incorporates a resilient and reliable data protocol into the transport layer, i.e., within the data packet. To ensure data transmission efficiency, the data packet header should be as concise as possible, thereby allowing for greater payload capacity. See also... Figure 6The receiving end distinguishes connections using the five-tuple: source IP, source port, destination IP, destination port, and flow ID (equal to τ).

[0131] Furthermore, six flag bits (REG, ACK, NACK, EACK, FIN, DATA) are used to identify different types of data packets, enabling the regulation of the protocol transmission process and connection control. Next, we will combine... Figure 6 Detailed descriptions of the features and payloads for each different data packet:

[0132] 1. REGIST packet: The flag REG=1 indicates that the data packet is a data packet initiating a data transmission request from the sending end to the receiving end.

[0133] 2. REGIST+ACK packet: With REG=1 and ACK=1, it is a REGIST data packet in which the receiving end acknowledges receipt of the data transmission request. The option field in its header carries the batch size N, which is used to negotiate the window size for group ACK when sending data packets.

[0134] 3. ACK packet: The flag ACK=1 indicates that the sending end has acknowledged receiving the REGIST+ACK packet sent by the receiving end, indicating that both the sending and receiving ends have entered the connection establishment (ESTABLISH) state and can carry out gradient data transmission.

[0135] 4. DATA Packet: With the DATA flag set to 1, this is a data packet used to transmit gradient blocks. Gradient blocks have already been segmented at the application layer, so multiple gradient blocks can be carried within a single data packet. An index field needs to be added before each gradient block to ensure reliable out-of-order delivery at the receiving end. The size of each gradient block must be less than the MTU (Maximum Transmission Unit).

[0136] 5. NACK packet: With the NACK flag set to 1, this is a data packet used by the receiving end to request retransmission from the sending end. This packet carries the destination index and a first index bitmap. The destination index is filled into the options field of the packet header, while the first index bitmap is carried in the packet payload.

[0137] 6. FIN Packet: With the FIN flag set to 1, this packet is used by the sending end to send a stop signal to the receiving end, and it is also a data packet in which the sending end requests connection release. After the sending end has sent all the gradient blocks it needs to send, it will send a FIN packet to notify the receiving end, requesting to close the connection. Upon receiving the FIN packet, the receiving end enters the SELECTIVE STOP state: it starts a timer, determines the expected timeout time according to the predetermined duration, and begins generating EACK packets. The sending of the FIN packet marks the beginning of the connection release process.

[0138] 7. EACK Packet: With the EACK flag set to 1, this is the data packet sent by the receiver to the sender after the sender has sent the stop signal. The packet contains two parts: (1) the second index bitmap; and (2) the estimated timeout end time. Exceeding this time will result in a timeout. These two parts will be filled into the options field and the payload, respectively. After receiving the EACK packet, the sender will prioritize retransmitting the important gradient blocks marked in the second index bitmap, and then retransmit the non-important gradient blocks as appropriate, depending on the timeout period. This ensures reliable delivery of important gradients while also giving non-important gradients a certain retransmission opportunity.

[0139] 8. FIN+ACK Packet: With FIN=1 and ACK=1, this packet is used by the receiving end to reply to the FIN packet, indicating agreement to release the connection. After the receiving end sends all EACK packets and the timer expires, it will reply with a FIN+ACK packet to the sending end, notifying it that the connection can be completely closed. Both ends then enter the CLOSED state. The sending of the FIN+ACK packet marks the end of data transmission within the data transmission window.

[0140] Based on the aforementioned resilient and reliable data protocol, see [link / reference] Figure 7 and Figure 8 The specific implementation process of this invention can be referred to as follows:

[0141] 1. Connection establishment phase:

[0142] Specifically, before establishing a connection, both the sending and receiving ends are in the CLOSE state, indicating that the connection has not been established. Then, the sending and receiving ends establish a connection through a three-way handshake mechanism, including:

[0143] a) The sending end sends a REGIST packet to the receiving end, requesting the transmission of data.

[0144] (b) Upon receiving the REGIST packet, if the receiving end agrees to establish a connection, it replies with a REGIST+ACK packet. This packet carries the batch size N of the received end's expected batch of received and acknowledgment data packets. After sending this packet, the receiving end enters the REGIST-RCVD state, indicating that the request has been received.

[0145] c) After receiving the REGIST+ACK packet from the receiver, the sending end replies with an ACK confirmation packet, indicating that the connection has been successfully established. Both parties then enter the ESTABLISHED state and can begin transmitting data.

[0146] 2. Data transmission stage:

[0147] After the connection is established, both communicating parties can begin transmitting data packets. Utilizing application-layer end-side information sharing, multiple gradient blocks within the data packet can be globally indexed, facilitating reliable out-of-order reception at the receiving end.

[0148] 3. Error control stage:

[0149] The receiving end sends a NACK packet to the sending end, requesting the sending end to retransmit gradient blocks. After receiving the NACK packet, the sending end updates the starting sliding position of the non-blocking sliding window according to the target index in the NACK packet, and marks the lost gradient blocks according to the first index bitmap. Then, the sending end calculates the proportion of important gradients according to the first index bitmap to determine the importance threshold of this retransmission, and then selects to retransmit all, partially, or abandon the retransmission based on this importance threshold.

[0150] 4. Connection Release Phase:

[0151] See Figure 8 After the sending end completes the first retransmission, it sends a FIN packet to the receiving end. Upon receiving the FIN packet, the receiving end starts timing. During the timing period, it iterates through the gradient blocks sent by the sending end, counts the gradient blocks that have not been received, and sends back an EACK packet. Then it enters the SELECTIVE STOP state. The EACK packet carries the second index bitmap of the important gradient blocks that the receiving end needs to receive but has not yet received, as well as the estimated timing end time.

[0152] After receiving the EACK packet, the transmitting end begins the final retransmission. First, it retransmits the packets containing the important gradient blocks marked in the second index bitmap. Only after all packets containing important gradient blocks have been acknowledged by the receiving end does it consider retransmitting packets containing less important gradient blocks, until the timer expires. When the timer expires, the receiving end sends a FIN-ACK packet to the transmitting end; then, the transmitting end closes the connection with the receiving end, and both return to the CLOSE state.

[0153] 6. Enter the next data transmission window: After the connection is closed, the sender removes the retransmitted gradient blocks from the gradient block tensor. At this time, the sender can adjust the batch size according to the network conditions and continue to send the remaining gradient blocks in the gradient block tensor.

[0154] In this embodiment of the invention, the specific design and implementation of the protocol stack can be described from two perspectives: single-packet dimension and connection dimension. The single-packet dimension is responsible for the encapsulation, parsing, and distribution of data packets, while the connection dimension is responsible for the establishment, maintenance, and release of connections.

[0155] See Figure 10 In a single packet dimension, the protocol stack can be divided into three functional layers: environment abstraction layer, frame encapsulation processing layer, and ERDP protocol layer.

[0156] 1. Environment Abstraction Layer: The main purpose of the environment abstraction layer is to shield the underlying hardware differences and provide a unified and efficient application programming interface (API) for the upper layers. This layer uses polling receive and send components and leverages GPU affinity technology to bind specific GPU logical cores (Lcores) to polling threads, thereby efficiently processing data packets received by the network card. Received data packets are stored in a mempool using an mbuf data structure. To reduce the data copying overhead between system memory and GPU memory, a new mempool is created within the GPU memory, enabling direct data transfer between the network card and the GPU, further improving performance.

[0157] 2. Frame Encapsulation Processing Layer: The main function of the frame encapsulation processing layer is to encapsulate and parse data packets at the IP and Ethernet layers. When the Environment Abstraction Layer (EAB) receives a data packet, it queues it in the receive message queue. The frame parsing module retrieves the data packet from the message queue, extracts relevant Ethernet and IP layer information, and generates the corresponding Address Resolution Protocol (ARP) entries. After the transport layer encapsulation is completed at the ERDP protocol layer, the destination IP address is passed to the frame encapsulation processing module. This module looks up the ARP table based on the IP address to obtain information such as the destination MAC address and performs the Ethernet and IP layer encapsulation process.

[0158] 3. EDDP Protocol Layer: The EDDP protocol layer is the core of the resilient and reliable data protocol stack. It is primarily responsible for the encapsulation and parsing of EDDP protocol headers and works in conjunction with connection-level modules to complete data packet encapsulation and distribution. This layer includes the following key modules:

[0159] a) Multiplexing Component: After the frame encapsulation processing layer passes the received data packet to this component, it parses the Type field in the ERDP header to identify the data packet type (e.g., REGIST, DATA, NACK). Then, it extracts the values ​​of the corresponding fields and constructs the corresponding Protocol Data Unit (PDU) object. Finally, based on the five-tuple information (source IP, source port, destination IP, destination port, and flow ID), it distributes the PDU object to the corresponding connection queue.

[0160] b) Header Encapsulation Component: This module receives data to be sent from the connection maintenance module. Depending on the PDU type, it fills in the corresponding fields in the ERDP protocol header and encapsulates the application data into the data portion of the PDU, constructing a complete ERDP PDU. Then, the encapsulated PDU is handed over to the frame encapsulation processing layer, where Ethernet and IP headers are added, and finally, it is sent out by the polling thread of the environment abstraction layer.

[0161] c) Co-connection Maintenance Component: This component assists in the establishment, maintenance, and release of ERDP connections. It receives information from the multiplexing component and interacts with the connection-level modules to refine the packet-level processing flow. When it receives the packet information and type parsed by the multiplexing component, it converts it into a PDU structure and passes it to the upper-layer connection-level modules.

[0162] See Figure 9 In the connection-level design, a modular and decoupled architecture can be adopted, dividing it into three major subsystems: connection maintenance, receive control, and transmit control. The connection maintenance subsystem manages the state machines of multiple concurrent connections; the receive control subsystem ensures reliable, out-of-order data reception; and the transmit control subsystem introduces an adaptive differentiated retransmission mechanism. This layered asynchronous processing model improves concurrency while also enhancing the system's flexibility and scalability.

[0163] 1. Connection Maintenance Area: An event-driven approach based on the Reactor pattern is adopted. Specifically, the connection control unit uses I / O multiplexing mechanisms such as epoll to listen for events from multiple modules, including the application layer, receive control, and transmit control, and performs efficient event distribution and processing. For each ERDP connection, this embodiment defines a Connection class, which records key information such as the connection state (e.g., CLOSED, ESTABLISHED), connection socket, and receive / transmit buffers. The state control logic is abstracted as a finite state machine (FSM). Based on the received PDU type and timer events, the corresponding state transition function is called to implement connection management functions. For example, upon receiving a REGIST message, the FSM will transition to the ESTABLISHED state and start connection maintenance timers such as fast retransmission and selection acknowledgment. Simultaneously, to support high concurrency, the connection control unit also introduces a connection pool mechanism, pre-creating and initializing a certain number of Connection objects to avoid frequent memory allocation and release.

[0164] 2. Receive Control System: A layered architecture can be used to achieve out-of-order reliable delivery. The interaction between the receive queue and the middle layer is accomplished through callback functions: when the receive queue receives a new PDU, it triggers a callback function registered by the middle layer, passing the PDU and its metadata (such as connection number, sequence number, gradient index, etc.) as parameters to the middle layer. The middle layer then handles the detailed reliable delivery mechanism and error control functions.

[0165] 3. Transmission control system: Adopt a sliding window flow control mechanism based on packet ACK. Specifically, the sender maintains a sending window sendWindow with a fixed size, which is N maximum segment sizes (Maximum Segment Size, MSS), and the size of MSS corresponds to a non-blocking sliding window. At the same time, a window size groupAckSize = N for packet ACK is defined, indicating that every N MSSs form an ACK group. The sender advances N MSSs forward each time in units of ACK groups.

[0166] Among them, a circular buffer sendBuf can be used to implement the sending window, and its size is N MSSs. The left and right boundaries of the sending window are represented by two pointers sndLeft and sndRight respectively. sndLeft points to the starting position of the first MSS that has been sent but not yet acknowledged, and sndRight points to the starting position of the next MSS to be sent. Obviously, sndRight - sndLeft <= N.

[0167] After the communication control module receives the gradient block to be sent from the application layer, it first encapsulates it into several MSSs and stores them in sendBuf. Then, it checks whether the remaining space in the sending window is sufficient, that is, sndRight - sndLeft < N. If the window is full, the sending is suspended until a new ACK group is received; otherwise, the MSSs within the range [sndRight, sndRight + groupAckSize) in sendBuf are continuously sent out, and sndRight is advanced by groupAckSize MSSs.

[0168] Under the packet ACK mechanism at the receiver, the sender does not receive an acknowledgment every time it sends an MSS, but has to wait until the last MSS of an ACK group is sent before receiving the ACK (the second ACK information) of this group. Therefore, the sender needs to record the starting sequence number groupSndLeft of each ACK group so that when receiving the aggregated ACK, sndLeft can be updated to groupSndLeft + groupAckSize * NumOfGroupsAcked, where NumOfGroupsAcked represents the number of ACK groups acknowledged by this aggregated ACK.

[0169] In summary, the in-network gradient sparsification retransmission system proposed in the embodiments of the present invention, through end-to-end collaborative communication control, adaptive differential retransmission strategies, and an efficient and scalable elastic reliable transport protocol stack, while improving the communication performance of distributed machine learning, well balances the model convergence speed and accuracy.

[0170] Building upon this foundation, this invention further introduces a reliable out-of-order delivery mechanism based on gradient information to optimize overall system performance. Traditional reliable transport protocols, such as TCP, employ strict sequential delivery semantics to achieve fully reliable delivery of byte streams. However, this strict sequential delivery often requires waiting for the lost segments to be retransmitted and delivered before subsequent segments can be delivered to the application layer when dealing with out-of-order packet loss. This is the well-known "head-of-line blocking" problem. Head-of-line blocking causes the receiving end to frequently wait for and buffer out-of-order data, increasing end-to-end long-tail latency.

[0171] In distributed machine learning, since the gradient blocks generated by each training node are semantically independent, strict sequential delivery is not necessary. Instead, timely submission of out-of-order gradient blocks to the upper-layer model for aggregation and parameter updates can accelerate model convergence. Therefore, this invention proposes a reliable out-of-order delivery mechanism based on gradient information, introducing gradient indexing and delivery processing logic in the intermediate layer, breaking the constraints of traditional sequential delivery. This will be further described below:

[0172] This invention implements a reliable out-of-order delivery mechanism based on gradient semantics in the intermediate layer. Its core idea is to fully utilize the positional correspondence of distributed gradients across different training nodes, relaxing the sequential delivery constraints of traditional reliable transmission protocols. This allows out-of-order gradient blocks to be cached first, and then delivered to the application layer as a whole after the entire gradient vector is reassembled. Simultaneously, it introduces receive state management based on gradient indexes and bitmaps, along with reliability assurance measures such as packet ACK and timeout retransmission, to ensure the final complete delivery of gradient data.

[0173] Furthermore, if certain critical gradient blocks are delayed in being retransmitted due to packet loss, a missing block filling mechanism can be introduced to dynamically assess the reception progress and fill in the timed-out blocks with alternative values ​​to maintain the overall delivery rhythm.

[0174] Specifically, the intermediate layer maintains a backup of the complete gradient vector from the previous iteration in the Gradient Backup Buffer (GBB). It also sets a receive ratio threshold (e.g., 0.99) and a receive timeout threshold (e.g., 1.5 iteration cycles). When one of the following two conditions is met, the missing gradient blocks in the current iteration's Gradient Receiving Buffer (GRB) are filled with the old values ​​from the GBB, and the filled complete gradient vector is pushed to the application layer.

[0175] Furthermore, considering that the size of the receiving buffer is always limited, long-accumulated old gradient data may exhaust storage resources, so it can be cleaned up periodically. The triggering condition for cleaning up is:

[0176] Once the gradient block of a certain iteration has been completely pushed to the application layer, it can be completely cleared and the space can be reclaimed.

[0177] When the storage pressure reaches a certain threshold (such as 80% capacity), the oldest batch of gradient data is selected for cleaning until the overall storage usage of the buffer drops below the threshold.

[0178] It is worth mentioning that since out-of-order delivery is performed on the basis of the entire gradient vector, the granularity of buffer cleanup is also enlarged accordingly. This reduces the frequency of cleanup operations to some extent and improves the utilization efficiency of the buffer.

[0179] Overall, the reliable out-of-order delivery mechanism in the middle layer plays a crucial role in end-to-end communication control. On one hand, it performs semantically aware caching and reassembly of out-of-order gradient blocks received from the ERDP protocol stack. On the other hand, it pushes complete gradient vectors in batches in a form suitable for application layer use. Through flexible state management, group ACK, and approximate recovery, it largely mitigates the long-tail latency problem, maximizing the performance of parallel training while tolerating some packet loss and latency. Furthermore, the overall delivery semantics align with the iterative characteristics of the application layer, achieving decoupling and coordination between communication and computation.

[0180] In summary, the key innovation of the adaptive data transmission method for distributed machine learning provided in this invention lies in:

[0181] a) It breaks the strict sequential delivery semantics of traditional protocols, allowing gradient blocks that arrive out of order to be cached first, and then delivered to the application layer as a whole after all of them have arrived, thus avoiding head-of-line blocking.

[0182] b) Introduce a receive status management system based on gradient index and bitmap to accurately track the arrival status of each gradient block and notify the sender to retransmit in a targeted manner.

[0183] c) To address the issue of individual key gradient blocks failing to arrive within a timeout period, a missing block filling mechanism is proposed, which fills the incomplete gradients with the old values ​​from the previous iteration to maintain the overall delivery rhythm.

[0184] d) Adopt a batch delivery strategy, that is, after collecting a sufficient number of complete gradient vectors, push them to the application layer as a whole to reduce the overhead of frequent communication synchronization.

[0185] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0186] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. In addition, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0187] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the description of this invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0188] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An adaptive data transmission method for distributed machine learning, characterized in that, During the global gradient reduction process performed by distributed training nodes through the All-Reduce operation, any two training nodes act as communicating parties and use the adaptive data transmission method for data transmission; the communicating parties include a sender and a receiver; the adaptive data transmission method includes: The originating end vectorizes the gradients generated during the backpropagation of the neural network model to obtain a gradient tensor; it divides the gradient tensor into blocks, constructs a gradient block tensor from the block results, and sets an index and importance label for each gradient block in the gradient block tensor; it constructs a gradient block importance bitmap based on the importance labels of each gradient block; and it initiates a data transmission request to the receiving end. In response to the data transmission request, the receiving end sends a first ACK message back to the sending end; the first ACK message carries the batch size N of the data packets that the receiving end expects to receive and acknowledge in batches. After receiving the batch size, the sending end uses a non-blocking sliding window to slide orderly on the gradient block tensor. After each slide, the gradient block and its index in the current non-blocking sliding window are packaged into a data packet and sent to the receiving end until the entire gradient block tensor has been sent. The number of gradient blocks in the non-blocking sliding window is N. After receiving and acknowledging N data packets, the receiving end sends a second ACK message back to the sending end. The second ACK message carries a first index bitmap of the gradient blocks that the receiving end currently needs to receive but has not yet received, as well as a target index. The target index is used to indicate the cutoff position of a group of gradient blocks that the receiving end has received consecutively from the N data packets in the gradient block tensor. After receiving the second ACK information, the sending end updates the starting sliding position of the non-blocking sliding window according to the target index, and performs gradient block retransmission according to the gradient block importance bitmap and the first index bitmap. The step of retransmitting gradient blocks based on the gradient block importance bitmap and the first index bitmap includes: The important gradient ratio is calculated based on the first index bitmap; the important gradient ratio is the proportion of important gradient blocks in the gradient blocks that need to be retransmitted this time. The importance threshold for this retransmission is determined based on the important gradient ratio, the current iteration number of the neural network model, and the loss value of the previous iteration; the importance threshold is a threshold related to the important gradient ratio. If the proportion of important gradients exceeds the importance threshold, then according to the gradient block importance bitmap, all gradient blocks that the receiving end currently needs to receive but has not yet received are retransmitted. If the proportion of important gradients does not exceed the importance threshold and is higher than the preset lower limit, then according to the gradient block importance bitmap, all important gradient blocks that the receiving end currently needs to receive but has not received will be retransmitted. If the significant gradient ratio is not higher than the lower limit of the ratio, then the gradient block retransmission is abandoned.

2. The adaptive data transmission method according to claim 1, characterized in that, The starting point, the method for setting importance labels for each gradient block, includes: The average gradient of each gradient block in the gradient block tensor is computed in parallel. Based on the average gradient, the TopK algorithm is used to sort the gradient blocks in the gradient block tensor to obtain the sorting result; Based on the sorting results, determine whether each gradient block is important; if important, set a first label value as an importance label for the gradient block; if not important, set a second label value as an importance label for the gradient block.

3. The adaptive data transmission method according to claim 1, characterized in that, The gradient block index is represented as: ; in, The number in the gradient tensor represents the gradient matrix of the network layer corresponding to the gradient block. The offset within the layer indicates the starting position of the gradient block within the gradient matrix. This indicates the size of the gradient block.

4. The adaptive data transmission method according to claim 1, characterized in that, The size of the non-blocking sliding window is set according to the bandwidth-delay product.

5. The adaptive data transmission method according to claim 1, characterized in that, The originating point determines the importance threshold for this retransmission based on the important gradient ratio, the current iteration number of the neural network model, and the loss value of the previous iteration, including: Based on the importance gradient ratio, the current iteration number of the neural network model, and the loss value of the previous iteration, the importance threshold of the previous retransmission is dynamically adjusted using a preset threshold adjustment formula to obtain the importance threshold of the current retransmission; wherein, the importance threshold of the initial retransmission is a preset value. The threshold adjustment formula is as follows: ； in, This represents the loss value from the previous iteration. Indicates the proportion of the important gradient. Indicates the current iteration round number. yes The corresponding first weighting factor yes The corresponding second weighting factor, yes The corresponding third weighting factor, yes The corresponding fourth weighting factor, It is the importance threshold of the last retransmission. This is the importance threshold for this retransmission.

6. The adaptive data transmission method according to claim 1, characterized in that, Also includes: After completing the retransmission operation, the sending end sends a stop signal to the receiving end; The receiving end responds to the stop signal to start timing and sends a third ACK message back to the transmitting end; the third ACK message carries a second index bitmap of the gradient block that the receiving end currently needs to receive but has not yet received, and also carries the expected timing end time. After receiving the third ACK information, the transmitting end tries its best to retransmit the gradient block indicated by the second index bitmap to the receiving end, referring to the timeout end time.

7. The adaptive data transmission method according to claim 6, characterized in that, When the transmitting end retransmits the gradient block indicated by the second index bitmap, it prioritizes retransmitting the important gradient blocks indicated by the second index bitmap according to the gradient block importance bitmap.

8. The adaptive data transmission method according to claim 6, characterized in that, The data transmission request, the first ACK information, the second ACK information, the blocking signal, and the third ACK information are all carried in the data packet for transmission; wherein, the header of the data packet is provided with multiple flag bits to distinguish different ACK information.

9. The adaptive data transmission method according to claim 1, characterized in that, The batch size is preset based on network conditions and training tasks.

Images