A cross-data center distributed machine learning communication method and system

Abstract

The present invention discloses a cross-data center distributed machine learning communication method and system. After a training node sends multiple gradient data packets to a parameter server that establishes an RUDP connection with itself according to a sending rate determined according to an estimated bandwidth, the parameter server repairs high-priority gradient data encoded by FEC, and fills the lost low-priority gradient data by using a Bubble-Filling mechanism when the actual packet loss rate is higher than a Bubble-Filling packet loss rate threshold, and performs gradient data aggregation on the repaired and filled gradient data to obtain aggregated gradient data, which is sent to the training node; the training node updates model parameters according to the aggregated gradient data, and continues training until the training is completed, thereby significantly improving the training efficiency and improving the reliability of the cross-data center distributed machine learning communication method.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a cross-data center distributed machine learning communication method and system. Background Art

[0002] With the rapid development of artificial intelligence, machine learning, especially deep learning, has made breakthrough progress in image recognition, speech processing, natural language understanding and other fields. However, training large deep learning models often requires massive data samples and computing resources. In order to speed up the model training process, distributed machine learning came into being. It splits the training tasks into multiple training nodes for parallel execution, and then synchronizes the parameters between training nodes through the PS (Parameter Server) architecture, thereby greatly improving the training efficiency.

[0003] However, during the distributed training process, the PS architecture faces a serious long-tail latency problem, which restricts the scalability of the system. The fundamental reason is that as the number of training nodes increases, a large number of training nodes communicate with the PS almost simultaneously at the end of each iteration, resulting in a many-to-one "incast" traffic pattern, which leads to fierce competition for network bandwidth and a sharp increase in packet loss rate. Some data streams have to be retransmitted frequently, which significantly prolongs the completion time of the tail stream. And because of the use of the Bulk Synchronous Parallel (BSP) synchronization model, all training nodes must wait for the slowest training node to complete the parameter update before entering the next iteration. A few nodes with severe delays will greatly reduce the efficiency of the entire training.

[0004] In addition, in cross-data center training scenarios, the communication between the training nodes and the PS needs to go through a wide area network or wireless link. Physical link failures and signal fading often occur, resulting in significant non-congestion packet loss, further exacerbating the long-tail latency problem. The network has become a key bottleneck restricting the scalability of distributed machine learning.

[0005] In response to the above problems, existing solutions mainly start from two aspects: one is to optimize the communication topology, such as Ring-AllReduce (a communication system based on ring topology), hierarchical aggregation parameters of the second-layer parameter server, etc.; the other is to compress the communication volume, such as parameter quantization, pruning, gradient sparsification, etc. However, these methods have not fundamentally solved the network bottleneck problem. Traditional transmission control protocols such as TCP, although they can ensure the reliable delivery of data, cannot tolerate packet loss. Faced with severe packet loss in the incast scenario, frequent retransmissions will worsen the long-tail delay. Although UDP (User Datagram Protocol) tolerates packet loss, excessive packet loss will affect the convergence of model training.

[0006] Therefore, how to provide a reliable and efficient cross-data center distributed machine learning communication method has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] In order to solve the above problems existing in the prior art, the present invention provides a cross-data center distributed machine learning communication method and system.

[0008] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0009] In a first aspect, the present invention provides a cross-data center distributed machine learning communication method, the method comprising:

[0010] The training node encapsulates the gradient data calculated during the training model to obtain multiple gradient data packets;

[0011] The training node estimates the bandwidth and determines the sending rate based on the estimated bandwidth;

[0012] The training node sends the plurality of gradient data packets to a parameter server that establishes an RUDP connection with the training node at the sending rate; the packet header of the gradient data packet carries the priority of the gradient data, and the priority includes a high priority and a low priority; wherein the high priority gradient data is encoded using FEC;

[0013] The parameter server repairs the high-priority gradient data and uses the Bubble-Filling mechanism to fill the lost low-priority gradient data when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold;

[0014] The parameter server aggregates the repaired and filled gradient data to obtain aggregated gradient data, and sends the aggregated gradient data to the training node;

[0015] The training node updates the model parameters according to the aggregated gradient data and continues training until the training is completed.

[0016] Optionally, the training node performs bandwidth estimation, including:

[0017] The current bandwidth is measured by adjusting the congestion window size during each round trip delay, triggering the accumulation and depletion of the network queue.

[0018] Optionally, the training node determines the sending rate according to the estimated bandwidth, including:

[0019] At the beginning of each data packet transmission window, the initial sending rate is calculated based on the estimated bandwidth;

[0020] During each packet transmission window, monitor the loss of gradient packets; when there is no packet loss within multiple consecutive round-trip delays, increase the sending rate; when packet loss is detected, adjust the sending rate according to the following process:

[0021] If the actual packet loss rate is lower than the target packet loss rate of congestion control, increasing the sending rate;

[0022] If the actual packet loss rate is higher than the target packet loss rate and lower than the Bubble-Filling packet loss rate threshold, the current sending rate is maintained.

[0023] If the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, the current sending rate is reduced to half.

[0024] Optionally, the target packet loss rate is calculated by:

[0025] The training node uses the least squares method to fit a first function of the average throughput with respect to the packet loss rate and a second function of the average delay with respect to the packet loss rate according to the historical average throughput and the historical average delay; and calculates the current target packet loss rate based on the first function and the second function.

[0026] Optionally, the cross-data center distributed machine learning communication method further includes:

[0027] When the proportion of high-priority gradient data packets in the gradient data packets received by the parameter server reaches a preset proportion, feedback is sent to the training node to notify the training node to enter the next data packet transmission window period.

[0028] Optionally, the gradient data packet carries meta information of a gradient tensor; the gradient tensor is a tensor formed by gradient data contained in the gradient data packet; the meta information includes at least sparsity of high-priority gradient data in the gradient data packet;

[0029] The parameter server repairs high-priority gradient data, including:

[0030] The parameter server repairs the high-priority gradient data according to the meta-information.

[0031] In a second aspect, the present invention provides a cross-data center distributed machine learning communication system, the system comprising a plurality of training nodes and at least one parameter server:

[0032] The training node is used to encapsulate the gradient data calculated during the training model, and after obtaining multiple gradient data packets, perform bandwidth estimation, and determine the sending rate according to the estimated bandwidth; send the multiple gradient data packets to the parameter server that establishes an RUDP connection with itself according to the sending rate; the packet header of the gradient data packet carries the priority of the gradient data, and the priority includes a high priority and a low priority; wherein the high priority gradient data is encoded using FEC;

[0033] The parameter server is used to repair the high-priority gradient data and fill the lost low-priority gradient data using the Bubble-Filling mechanism when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold; perform gradient data aggregation on the repaired and filled gradient data to obtain aggregated gradient data, and send the aggregated gradient data to the training node;

[0034] The training node is used to update the model parameters according to the aggregated gradient data and continue training until the training is completed.

[0035] Optionally, training a node to perform bandwidth estimation includes:

[0036] The current bandwidth is measured by adjusting the congestion window size during each round trip delay, triggering the accumulation and depletion of the network queue.

[0037] Optionally, the training node determines a sending rate according to the estimated bandwidth, including:

[0038] At the beginning of each data packet transmission window, the initial sending rate is calculated based on the estimated bandwidth;

[0039] During each packet transmission window, monitor the loss of gradient packets; when there is no packet loss within multiple consecutive round-trip delays, increase the sending rate; when packet loss is detected, adjust the sending rate according to the following process:

[0040] If the actual packet loss rate is lower than the target packet loss rate of congestion control, increasing the sending rate;

[0041] If the actual packet loss rate is higher than the target packet loss rate and lower than the Bubble-Filling packet loss rate threshold, the current sending rate is maintained.

[0042] If the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, the current sending rate is reduced to half.

[0043] Optionally, the target packet loss rate is calculated by:

[0044] The training node uses the least squares method to fit a first function of the average throughput with respect to the packet loss rate and a second function of the average delay with respect to the packet loss rate according to the historical average throughput and the historical average delay; and calculates the current target packet loss rate based on the first function and the second function.

[0045] Optionally, the parameter server is also used to send feedback to the training node to notify the training node to enter the next data packet transmission window when the proportion of high-priority gradient data packets in the received gradient data packets reaches a preset proportion.

[0046] Optionally, the gradient data packet carries meta information of a gradient tensor; the gradient tensor is a tensor formed by gradient data contained in the gradient data packet; the meta information includes at least sparsity of high-priority gradient data in the gradient data packet;

[0047] The parameter server repairs high-priority gradient data, including:

[0048] The parameter server repairs the high-priority gradient data according to the meta-information.

[0049] In a third aspect, the present invention provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0050] Memory, used to store computer programs;

[0051] The processor is used to implement the method steps described in any of the above-mentioned cross-data center distributed machine learning communication methods when executing the program stored in the memory.

[0052] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any of the above-mentioned cross-data center distributed machine learning communication methods are implemented.

[0053] In another aspect of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute the method steps described in any of the above-mentioned cross-data center distributed machine learning communication methods.

[0054] The present invention provides a cross-data center distributed machine learning communication method, which encapsulates and processes the gradient data calculated during the training model through a training node to obtain a plurality of gradient data packets; the training node estimates the bandwidth and determines the sending rate according to the estimated bandwidth; the training node sends the plurality of gradient data packets to a parameter server that establishes an RUDP connection with itself according to the sending rate; the packet header of the gradient data packet carries the priority of the gradient data, and the priority includes a high priority and a low priority; wherein the high priority gradient data is encoded using FEC; the parameter server repairs the high priority gradient data, and when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, fills the lost low priority gradient data using a Bubble-Filling mechanism; the parameter server aggregates the repaired and filled gradient data to obtain aggregated gradient data, and sends the aggregated gradient data to the training node; the training node updates the model parameters according to the aggregated gradient data, and continues training until the training is completed.

[0055] By prioritizing the gradient data and using FEC encoding for high-priority gradient data, differentiated reliability guarantees are provided to avoid the loss of high-priority gradient data. For the lost low-priority gradient data, the Bubble-Filling mechanism is used to fill it only when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, which tolerates the loss of a certain amount of low-priority gradient data, significantly improves training efficiency, and also avoids the situation where poor training results occur due to excessive loss of low-priority gradient data.

[0056] In addition, by using the estimated bandwidth to determine the sending rate of the gradient data packet, the utilization of network resources is fully guaranteed.

[0057] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A cross-data center distributed machine learning communication method provided by an embodiment of the present invention;

[0059] Figure 2 It is a flowchart of another cross-data center distributed machine learning communication method provided by an embodiment of the present invention;

[0060] Figure 3 is a schematic diagram of the structure of an exemplary cross-data center distributed machine learning communication system provided by an embodiment of the present invention;

[0061] Figure 4 It is a structural diagram of an electronic device. DETAILED DESCRIPTION

[0062] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0063] In order to solve the problems of poor reliability and low efficiency of existing cross-data center distributed machine learning communication methods, an embodiment of the present invention provides a cross-data center distributed machine learning communication method and system.

[0064] In order to facilitate the understanding of the present invention, the distributed cluster is first described. A distributed cluster is a system composed of multiple computer nodes, and the nodes communicate and cooperate with each other through a network to jointly complete a task or provide a service. In this embodiment, the distributed cluster includes multiple training nodes and at least one parameter server.

[0065] Distributed machine learning training uses data parallelism to divide the training data set into multiple non-overlapping subsets, and then assigns these subsets to multiple training nodes. Each training node can be trained based on the training data and model parameters in the subset, and then regularly synchronized and aggregated with other training nodes through the parameter server to update the model parameters. Multiple nodes collaborate to optimize the global model, thereby accelerating the training process.

[0066] The following is a detailed description of a cross-data center distributed machine learning communication method provided by an embodiment of the present invention. Figure 1 , Figure 1 A cross-data center distributed machine learning communication method provided by an embodiment of the present invention specifically includes the following steps:

[0067] Step S101: The training node encapsulates the gradient data calculated during the training model to obtain a plurality of gradient data packets.

[0068] In this embodiment, each training node can use the model parameters and the training data allocated to itself to calculate the gradient data of the training node, and then encapsulate the gradient data to obtain multiple gradient data packets of the training node.

[0069] The specific encapsulation step is that the training node submits the gradient data to the RUDP protocol stack. RUDP divides the gradient data into fixed-size blocks, adds header information such as sequence number (SN), data type (DATA / ACK / FEC), and encapsulates it into one or more RUDP data packets and stores them in the send buffer.

[0070] Step S102: The training node estimates the bandwidth and determines the sending rate according to the estimated bandwidth.

[0071] In this embodiment, the training node performs bandwidth estimation in order to estimate the maximum available bandwidth of the current network, provide a basis for the sending rate of the gradient data packet, and thus ensure full utilization of the current network resources.

[0072] In step S103, the training node sends multiple gradient data packets to the parameter server that has established an RUDP connection with itself according to the sending rate; the header of the gradient data packet carries the priority of the gradient data, which includes high priority and low priority; among them, the high-priority gradient data is encoded using FEC.

[0073] In this embodiment, before the training node transmits the gradient data packet to the parameter server, it is necessary to first establish an RUDP (Reliable UDP, Reliable User Datagram Protocol) session between the parameter server and the training node. The specific steps of establishing the session are as follows:

[0074] The training node sends a SYN (Synchronize Sequence Numbers) packet to the parameter server. The SYN packet contains parameters such as the initial sequence number (ISN_c), receive window size (RWND_c) and maximum packet size (MSS_c) of the training node.

[0075] After receiving the SYN packet, if the parameter server agrees to establish a connection, it will reply a SYN-ACK packet to the parameter server. The SYN-ACK packet contains parameters such as the initial sequence number (ISN_s), receive window size (RWND_s) and maximum packet size (MSS_s) of the parameter server, and returns the parameters of the training node as is as confirmation.

[0076] After receiving the SYN-ACK packet, the training node checks whether the parameters in the SYN-ACK packet returned by the parameter server are consistent with those sent by itself. If they are consistent, it replies with an ACK packet to formally establish a connection with the parameter server; otherwise, it sends a RST packet to terminate the connection.

[0077] In this embodiment, the RUDP communication method is adopted. Compared with the three-way handshake connection of TCP, the connection establishment process of RUDP only exchanges necessary connection parameters and does not transmit any business data. Therefore, one RTT (Round-Trip Time) can be saved, which speeds up the connection establishment speed between the training node and the parameter server.

[0078] In this embodiment, when there is only one parameter server, all training nodes establish an RUDP connection with one parameter server; when there are multiple parameter servers, it can be specified as needed which training nodes and which parameter server to establish an RUDP connection with.

[0079] In this embodiment, the header of the gradient data packet carries the priority of the gradient data, which includes high priority and low priority. The priority of the gradient data carried in the header can be used to determine the gradient data packet with high priority and the gradient data packet with low priority.

[0080] In this embodiment, high-priority gradient data is encoded using FEC (Forward Error Correction). FEC encoding is a data transmission technology whose core idea is to use error correction codes to redundantly encode gradient data at training nodes. This technology uses data to transmit redundant information, allowing the parameter server to reconstruct data when errors occur in gradient data.

[0081] Specifically, RS erasure coding is performed on high-priority gradient data to generate a certain proportion (e.g., 10%) of redundant packets. These redundant packets are marked as FEC type and sent together with the original gradient data packets. In contrast, low-priority data packets are not FEC-encoded to save computing and transmission overhead.

[0082] In step S104, the parameter server repairs the high-priority gradient data and fills the lost low-priority gradient data using the Bubble-Filling mechanism when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold.

[0083] In this embodiment, since the high-priority gradient data is encoded by FEC, when the high-priority gradient data is lost, FEC can be used to correct it. In addition, when the cumulative received FEC redundant packets reach a certain proportion (such as 50%), the parameter server can send a retransmission request to the training node so that the training node resends the lost high-priority gradient data, thereby avoiding unnecessary retransmission.

[0084] In this embodiment, the bubble-filling packet loss rate threshold is a preset maximum tolerable packet loss rate threshold, and the lost low-priority gradient data within this threshold can be appropriately ignored. When the real-time packet loss rate exceeds this threshold, the lost low-priority gradient data needs to be filled.

[0085] In this embodiment, Bubble-Filling is the final mechanism for RUDP to deal with severe packet loss. It is triggered when the parameter server detects that the packet loss rate is too high. Its function is to fill the lost low-priority gradient data in situ with easy-to-calculate substitute values, thereby avoiding the subsequent blocking caused by the difficulty of Shape alignment when the parameter server aggregates the gradient data, and improving the aggregation efficiency.

[0086] For lost low-priority gradient data, the parameter server can infer the sequence number and size of the lost gradient data based on the RUDP header information, insert a special placeholder (Bubble) at the corresponding position, and identify and skip the placeholder when processing gradient data. This ensures the integrity of the data stream and avoids unnecessary waiting.

[0087] In this embodiment, a specific repair method can be to fill the lost low-priority gradient data with a preset replacement value. For example, for the lost low-priority sparse gradient data, since it usually contains a large number of 0 elements, the loss of a small number of non-0 elements has little effect on the update of model parameters, so it can be uniformly filled with 0. For the lost low-priority dense gradient data, the lost gradient data can be uniformly filled with a preset constant C. The value of this constant C can be set by experience. It should be noted that the selection of this value is usually much smaller than the mean of the gradient data to avoid introducing large deviations.

[0088] In addition, in order to ensure the unbiasedness of the filling value and the true gradient as much as possible, the statistical meta-information attached to the gradient data can be further used to fine-tune the filling value. For example, the constant C is replaced with the mean of the gradient data, and the filled gradient vector is multiplied by a correction factor to make its variance equal to the true gradient data.

[0089] Step S105: the parameter server aggregates the repaired and filled gradient data to obtain aggregated gradient data, and sends the aggregated gradient data to the training node.

[0090] In this embodiment, the gradient data used by the parameter server for gradient data aggregation is the repaired and filled gradient data, thereby avoiding the blockage caused by the difficulty of shape alignment, and the repaired and filled gradient data of multiple training nodes are aggregated into global aggregated gradient data in the parameter server. After completing the gradient data aggregation, the parameter server sends the aggregated gradient data to the training node so that the training node can update itself based on the aggregated gradient data.

[0091] Step S106: The training node updates the model parameters according to the aggregated gradient data and continues training until the training is completed.

[0092] In this embodiment, each training node can update the model parameters according to the aggregated gradient data, and continue to calculate the gradient data according to the updated model parameters and the training data assigned to itself. After each training node calculates the new gradient data, it returns to execute the training node to encapsulate the gradient data calculated during the training model to obtain multiple gradient data packets, until the number of iterations reaches a preset number, and the cross-data center distributed machine learning communication is completed.

[0093] In this embodiment, by dividing the priorities of the gradient data and using FEC coding for the high-priority gradient data, differentiated reliability guarantees are provided to avoid the loss of high-priority gradient data. For the lost low-priority gradient data, only when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, the Bubble-Filling mechanism is used to fill it, which tolerates the loss of a certain amount of low-priority gradient data, significantly improves the training efficiency, and also avoids the situation where the training effect is poor due to too much lost low-priority gradient data.

[0094] In addition, by using the estimated bandwidth to determine the sending rate of the gradient data packet, the utilization of network resources is fully guaranteed.

[0095] In the embodiment of the present invention, the training node performs bandwidth estimation, including:

[0096] The current bandwidth is measured by adjusting the congestion window size during each round trip delay, triggering the accumulation and depletion of the network queue.

[0097] In this embodiment, at the beginning of each round-trip delay, the size of the congestion window (cwnd) is set to the product of the estimated current bandwidth (BW_est) and the RTT, that is, the sending rate V is matched to the estimated current bandwidth. At a certain moment in the middle of each round-trip delay, the congestion window is multiplied by a detection factor ∝ (such as 1.5), and the sending rate is temporarily increased to V1, causing the network queue to accumulate. Continue to send while monitoring the round-trip delay of the gradient data packet. When the delay increases significantly (such as more than twice the baseline round-trip delay), it indicates that the network queue is about to be saturated, that is, the network carrying capacity has reached its limit, and the sending rate at this time is recorded as V2. Take the average value of V1 and V2 to update BW_est and obtain the current bandwidth for rate control of the next RTT.

[0098] In this embodiment, the maximum available bandwidth of the current network can be determined by measuring the current bandwidth, and then the sending rate of the gradient data packet can be determined according to the estimated bandwidth in the future, so as to more effectively utilize network resources.

[0099] In an embodiment of the present invention, the manner in which a training node determines a transmission rate according to an estimated bandwidth includes:

[0100] At the initial moment of each data packet transmission window period, calculate an initial transmission rate according to the estimated bandwidth;

[0101] During each data packet transmission window period, monitor the loss situation of gradient data packets; when there is no packet loss within a continuous plurality of round-trip delays, increase the transmission rate; when packet loss is detected, adjust the transmission rate according to the following process:

[0102] If the actual packet loss rate is lower than the target packet loss rate of congestion control, increase the transmission rate;

[0103] If the actual packet loss rate is higher than the target packet loss rate and lower than the Bubble-Filling packet loss rate threshold, maintain the current transmission rate.

[0104] If the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, reduce the current transmission rate to half.

[0105] In this embodiment, at the initial moment of each data packet transmission window period, an initial transmission rate can be calculated based on the estimated bandwidth, so as to balance between the transmission efficiency of gradient data packets and the packet loss tolerance according to this initial transmission rate.

[0106] First, start sending gradient data packets based on the initial transmission rate (V_init), and at the same time detect the loss situation of gradient data packets. When there is no packet loss within a continuous plurality of (such as 3) round-trip delays, increase the transmission rate on the basis of V_init.

[0107] When packet loss is detected, adjust the transmission rate by comparing the size of the real-time packet loss rate p and the target packet loss rate p_tol:

[0108] If p < p_tol, then increase the current transmission rate V.

[0109] If p > p_tol and is lower than the Bubble-Filling packet loss rate threshold (p_bf), then keep V unchanged.

[0110] If p > p_bf, further reduce V to 50% of V_init, and start the Bubble-Filling mechanism to fill the lost low-priority gradient data until p drops below p_tol, and then restore V to V_init.

[0111] In the specific implementation, two key parameters are set in the RUDP transport layer, namely TARGET_LOSS_RATE and LOSS_RATE_THRESHOLD. The former represents the target packet loss rate of the congestion control algorithm, and the latter represents the Bubble-Filling packet loss rate threshold that triggers Bubble-Filling, which is usually 2 to 5 percentage points higher than the former. For example, when TARGET_LOSS_RATE is 10%, LOSS_RATE_THRESHOLD can be set to 12%.

[0112] In this embodiment, when the actual packet loss rate is lower than TARGET_LOSS_RATE, RUDP's congestion control will increase the sending rate; when the actual packet loss rate exceeds TARGET_LOSS_RATE but is lower than LOSS_RATE_THRESHOLD, RUDP maintains the current sending rate; when the actual packet loss rate is higher than LOSS_RATE_THRESHOLD, RUDP slows down the sending rate and turns on the Bubble-Filling mechanism to fill in the lost low-priority gradient data packets on the spot, giving the limited bandwidth to high-priority gradient data packets and retransmitted gradient data packets.

[0113] In this embodiment, the network bandwidth is estimated by round-trip delay detection, and the sending rate is dynamically adjusted in combination with the target packet loss rate and the bubble-filling trigger threshold. This can not only make full use of network resources, but also stop losses in time to prevent excessive packet loss from affecting the training effect.

[0114] In this embodiment, congestion control and packet loss tolerance complement each other. By measuring the current bandwidth, the physical bandwidth upper limit of the network can be found, providing decision space for packet loss tolerance. Packet loss tolerance in turn guides congestion control to trigger bubble-filling at the right time, abandoning some low-priority gradient data packets to preserve the training process.

[0115] In an embodiment of the present invention, the target packet loss rate is calculated by:

[0116] The training node uses the least squares method to fit a first function of the average throughput on the packet loss rate and a second function of the average delay on the packet loss rate according to the historical average throughput and the historical average delay; and calculates the current target packet loss rate based on the first function and the second function.

[0117] In this embodiment, the target packet loss rate needs to be dynamically adjusted according to the network status, and the training node can calculate the current target packet loss rate in real time based on the network status information fed back by the parameter server. The network status information may include the sequence number range of the currently successfully received high-priority gradient data packets, the current packet loss rate, the repetition rate, the mean and variance of the packet arrival interval, the parameter server buffer usage rate, and the gradient data packet ratio of Bubble-Filling, etc.

[0118] The following is a detailed description of how to calculate the target packet loss rate:

[0119] At the beginning of each training iteration, the initial target packet loss rate is set to p0 (such as 5%). The training node determines the sending speed based on the initial target packet loss rate. After receiving the feedback from the parameter server, the historical network status information of the parameter server is extracted, and the historical average throughput and historical average delay are calculated. The first function of the average throughput on the packet loss rate and the second function of the average delay on the packet loss rate are fitted using the least squares method. Under the premise of ensuring that the average delay does not exceed the preset training iteration timeout, the current target packet loss rate is calculated based on the first function and the second function.

[0120] In this embodiment, the current target packet loss rate is calculated as follows:

[0121] p1=argmax(T(p)), stD(p)<D_max;

[0122] Wherein, p1 represents the current target packet loss rate; argmax(·) represents the aggregation function; T(p) represents the first function; D(p) represents the second function; T represents the average throughput; D represents the average delay; p represents the packet loss rate; and D_max represents the difference between the preset training iteration timeout and the current delay.

[0123] In an embodiment of the present invention, RUDP communication adopts an adaptive feedback mechanism, and the timing of sending feedback of the RUDP can be triggered by the parameter server according to a dynamic threshold. Therefore, the cross-data center distributed machine learning communication method also includes:

[0124] When the proportion of high-priority gradient data packets in the gradient data packets received by the parameter server reaches a preset proportion, feedback is sent to the training node to notify the training node to enter the next data packet transmission window period.

[0125] In this embodiment, only when the proportion of high-priority gradient data packets accumulated by the parameter server reaches a preset proportion, feedback is sent to the training node to notify the training node to enter the next data packet transmission window period, thereby avoiding frequent feedback overhead.

[0126] In addition, when there is severe packet loss, the parameter server can proactively delay sending feedback to the training node to give the Bubble-Filling mechanism more time to fill in the lost low-priority gradient data.

[0127] In the embodiment of the present invention, the importance of different gradient data is different. Weight gradient data is gradient data directly related to the weight parameters of the model. Weight gradient data is generally considered to be more important than activation gradient data because it directly affects the learning and parameter adjustment of the model. Activation gradient data is related to the output of the activation function in the model, which affects the nonlinear characteristics of the model and the shape of the decision boundary, and is generally considered to be less important than weight gradient data, so the priority of weight gradient data is higher than that of activation gradient data.

[0128] Sparse gradient data refers to the case where most elements in the gradient matrix or tensor are zero. Sparse gradient data is more efficient in storage and computation because we can focus only on non-zero elements. In contrast, most elements of dense gradient data are non-zero. This type of gradient data usually requires more computing resources and storage space, so sparse gradient data has a higher priority than dense gradient data.

[0129] Larger magnitude gradient data indicates that the absolute value of the gradient data is larger, which may indicate a more obvious error direction or a more urgent need for learning. Smaller magnitude gradient data indicates that the absolute value of these gradient data is smaller, which may indicate that the model is close to the optimal solution for some parameters or is in the fine-tuning stage of the learning process. Therefore, larger magnitude gradient data has a higher priority than smaller magnitude gradient data.

[0130] On the basis of the above, when judging the priority of gradient data, the priority follows the principle of amplitude > sparsity > gradient type (weight and activation).

[0131] Specifically, RUDP's heuristic rules can be used to automatically determine the importance of each gradient data and map it to different priorities. The specific rules are as follows:

[0132] Weight gradient data has higher priority than activation gradient data and is mapped to HIGH.

[0133] For each gradient tensor, flatten it into a one-dimensional vector and take the median of the L1 norm as the sparsity threshold. Gradient data greater than this threshold is mapped to HIGH, otherwise the gradient data is mapped to LOW.

[0134] Finally, the highest value is taken as the comprehensive priority. For example, a sparse activation gradient data has a final priority of HIGH.

[0135] The priority information of each gradient data packet is written into the type field in the header of the gradient data packet. At the same time, in the embodiment of the present invention, the gradient data packet also carries the metadata of the gradient tensor; the gradient tensor is a tensor formed by the gradient data contained in the gradient data packet; the metadata at least includes the sparsity of the high-priority gradient data in the gradient data packet;

[0136] The parameter server repairs high-priority gradient data, including:

[0137] The parameter server repairs high-priority gradient data based on the metadata.

[0138] In this embodiment, the meta information of the gradient tensor may include meta information such as sparsity, mean, and variance, which is attached to the end of the gradient tensor and sent together to assist the parameter server in performing reverse gradient repair. Specifically, the meta information can be used as a basis for the parameter server to repair high-priority gradient data.

[0139] In an embodiment of the present invention, after the parameter server receives the repaired and filled gradient data, the training node and the parameter server release the connection, and the specific steps are as follows:

[0140] a) The training node sends a FIN request to close the connection

[0141] b) After receiving FIN, the parameter server replies FIN-ACK, agrees to close the connection, and enters the semi-closed state. In the semi-closed state, the parameter server can continue to send data to the training node.

[0142] c) The parameter server continues to send the data to be sent in the buffer. After all the data is sent, it also sends a FIN to request to close the connection.

[0143] d) After receiving the FIN from the passive party, the training node replies with FIN-ACK and the connection is completely closed.

[0144] It should be noted that in the semi-closed state, the training node will respond to the data packets sent by the parameter server, but will not send new data. In addition, if the training node does not receive the FIN from the parameter server for a long time, a closing timer will be started. After the timeout, the FIN will be sent again until the FIN-ACK is received or the maximum number of retries is reached, and the connection is completely closed.

[0145] After the parameter server and the training node release the connection, the parameter server aggregates the gradient data after repair and filling to obtain the aggregated gradient data, and then re-establishes the RUDP connection with the training node to send the aggregated gradient data to the training node, so that the training node updates the model parameters according to the aggregated gradient data and continues training until the training is completed. The specific establishment process and transmission rules are as described above and will not be repeated here.

[0146] To further illustrate a cross-data center distributed machine learning communication method provided by an embodiment of the present invention, see Figure 2 , Figure 2 1 is a flow chart of another cross-data center distributed machine learning communication method provided by an embodiment of the present invention, comprising the following steps:

[0147] Step S201, constructing a parameter server architecture training cluster.

[0148] In a distributed machine learning scenario across data centers, deploy one or more parameter servers as management nodes for the global model, as well as multiple training nodes distributed in different data centers. The parameter servers and training nodes are interconnected via a wide area network, using the RUDP protocol for communication and carrying bidirectional model synchronization traffic.

[0149] Step S202: The training node completes local training.

[0150] Each training node loads its local training data and overwrites the local model with the global model parameters of the current iteration. The model is trained using the local training data to calculate the gradient data.

[0151] Step S203: Prioritize the gradient data packets.

[0152] After encapsulating the gradient data into gradient packets, the transport layer of the training node RUDP protocol stack is responsible for data encoding and sending control. It maintains two hierarchical sending queues: high priority queue (HQ) and low priority queue (LQ), and the queue capacity ratio can be 1:3. For each gradient packet, it can be pushed into the corresponding queue according to the priority tag in the packet header, thereby determining the high priority gradient packets and low priority gradient packets.

[0153] Step S204: The training node establishes a connection with the parameter server.

[0154] The training node establishes an RUDP connection with the parameter server. For details, please refer to the description of step S103, which will not be repeated here.

[0155] Step S205, transmitting the gradient data packet.

[0156] The training node sends multiple high-priority gradient data packets and multiple low-priority gradient data packets to the parameter server, so that the parameter server repairs the lost high-priority gradient data packets and fills the lost low-priority gradient data packets. The specific repair and filling process is described in step S103 and step S104, which will not be repeated here.

[0157] Step S206: The training node and the parameter server release the connection.

[0158] When the parameter server completes receiving the gradient data packet sent by the training node, the training node and the parameter server release the connection, that is, the training node closes the RUDP connection with the parameter server. The specific connection release content is as described above and will not be repeated here.

[0159] Step S207: The parameter server returns the aggregation result and starts a new round of iterative training.

[0160] The parameter server aggregates the repaired and filled gradient data to obtain aggregated gradient data, and sends the aggregated gradient data to the training node, so that the training node updates the model parameters according to the aggregated gradient data and continues to start the next round of training until the number of iterations reaches the preset number and the training is completed.

[0161] In this embodiment, by dividing the priorities of gradient data, differentiated reliability guarantees are provided to avoid the loss of high-priority gradient data. For the lost low-priority gradient data, the Bubble-Filling mechanism is used to fill it only when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, which tolerates the loss of a certain amount of low-priority gradient data, significantly improves the training efficiency, and while ensuring the necessary data reliability, reduces the end-to-end delay as much as possible, and improves network utilization.

[0162] In addition, by using the estimated bandwidth to determine the sending rate of the gradient data packet, the utilization of network resources is fully guaranteed.

[0163] Moreover, the cross-data center distributed machine learning communication method provided in this embodiment can be seamlessly integrated with the existing distributed machine learning framework and communication library, and the upper-layer applications can benefit from it without any modification. Compared with the previous gradient compression optimization at the application layer, RUDP achieves precise control at the transport layer, avoiding the redundant computing overhead introduced by the application layer solution, and truly exchanges communication time for computing time.

[0164] Based on the same inventive concept, an embodiment of the present invention further provides a cross-data center distributed machine learning communication system, including multiple training nodes and at least one parameter server:

[0165] The training node is used to encapsulate the gradient data calculated during the training model, and after obtaining multiple gradient data packets, perform bandwidth estimation, and determine the sending rate according to the estimated bandwidth; send the multiple gradient data packets to the parameter server that establishes an RUDP connection with itself according to the sending rate; the packet header of the gradient data packet carries the priority of the gradient data, and the priority includes a high priority and a low priority; wherein the high priority gradient data is encoded using FEC;

[0166] The parameter server is used to repair the high-priority gradient data and fill the lost low-priority gradient data using the Bubble-Filling mechanism when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold; perform gradient data aggregation on the repaired and filled gradient data to obtain aggregated gradient data, and send the aggregated gradient data to the training node;

[0167] The training node is used to update the model parameters according to the aggregated gradient data and continue the training until the training is completed.

[0168] In order to further illustrate a cross-data center distributed machine learning communication system provided by an embodiment of the present invention, taking a parameter server as an example, see Figure 3 , Figure 3 It is a structural diagram of an exemplary cross-data center distributed machine learning communication system provided by an embodiment of the present invention, in which the cross-data center distributed machine learning communication system includes a parameter server 301 and multiple training nodes 302.

[0169] In this embodiment, the training node provides differentiated reliability guarantee by prioritizing the gradient data and adopts FEC encoding for the high-priority gradient data, thus avoiding the loss of high-priority gradient data. The parameter server fills the lost low-priority gradient data with the Bubble-Filling mechanism only when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, thus tolerating the loss of a certain amount of low-priority gradient data, significantly improving the training efficiency, and also avoiding the situation where the training effect is poor due to too much lost low-priority gradient data.

[0170] In addition, the training node determines the sending rate of gradient data packets by using the estimated bandwidth, which fully ensures the utilization of network resources.

[0171] Optionally, training a node to perform bandwidth estimation includes:

[0172] The current bandwidth is measured by adjusting the congestion window size during each round trip delay, triggering the accumulation and depletion of the network queue.

[0173] Optionally, the training node determines a sending rate according to the estimated bandwidth, including:

[0174] At the beginning of each data packet transmission window, the initial sending rate is calculated based on the estimated bandwidth;

[0175] During each packet transmission window, monitor the loss of gradient packets; when there is no packet loss within multiple consecutive round-trip delays, increase the sending rate; when packet loss is detected, adjust the sending rate according to the following process:

[0176] If the actual packet loss rate is lower than the target packet loss rate of congestion control, increasing the sending rate;

[0177] If the actual packet loss rate is higher than the target packet loss rate and lower than the Bubble-Filling packet loss rate threshold, the current sending rate is maintained.

[0178] If the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, the current sending rate is reduced to half.

[0179] Optionally, the target packet loss rate is calculated by:

[0180] The training node uses the least squares method to fit a first function of the average throughput with respect to the packet loss rate and a second function of the average delay with respect to the packet loss rate according to the historical average throughput and the historical average delay; and calculates the current target packet loss rate based on the first function and the second function.

[0181] Optionally, the parameter server is also used to send feedback to the training node to notify the training node to enter the next data packet transmission window when the proportion of high-priority gradient data packets in the received gradient data packets reaches a preset proportion.

[0182] Optionally, the gradient data packet carries meta information of a gradient tensor; the gradient tensor is a tensor formed by gradient data contained in the gradient data packet; the meta information includes at least sparsity of high-priority gradient data in the gradient data packet;

[0183] The parameter server repairs high-priority gradient data, including:

[0184] The parameter server repairs the high-priority gradient data according to the meta-information.

[0185] The embodiment of the present invention further provides an electronic device, such as Figure 4 As shown, Figure 44 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, including a processor 401, a communication interface 402, a memory 403 and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0186] Memory 403, used for storing computer programs;

[0187] Processor 401 is used to implement the method steps described in any of the above-mentioned cross-data center distributed machine learning communication methods when executing the program stored in memory 403.

[0188] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not represent that there is only one bus or one type of bus.

[0189] The communication interface is used for communication between the above electronic device and other devices.

[0190] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0191] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0192] The present invention also provides a computer-readable storage medium. A computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method steps described in any of the above-mentioned cross-data center distributed machine learning communication methods are implemented.

[0193] Optionally, the computer-readable storage medium may be a non-volatile memory (NVM), such as at least one disk memory.

[0194] Optionally, the computer-readable memory may also be at least one storage device located away from the aforementioned processor.

[0195] In another embodiment of the present invention, a computer program product comprising instructions is also provided, which, when executed on a computer, enables the computer to execute the method steps described in any of the above-mentioned cross-data center distributed machine learning communication methods.

[0196] It should be noted that, for the system / electronic device / storage medium / computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0197] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of systems and methods consistent with some aspects of the present invention.

[0198] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification.

[0199] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other changes to the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the term "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and "multiple" means two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0200] The method provided in the embodiment of the present invention can be applied to electronic devices. Specifically, the electronic device can be: a desktop computer, a portable computer, an intelligent mobile terminal, a server, etc. This is not limited here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0201] It should be noted that the system, electronic device and storage medium of the embodiments of the present invention are respectively the system, electronic device and storage medium applying the above-mentioned cross-data center distributed machine learning communication method. All embodiments of the above-mentioned cross-data center distributed machine learning communication method are applicable to the system, electronic device and storage medium, and can achieve the same or similar beneficial effects.

[0202] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A cross-data center distributed machine learning communication method, characterized in that: The method comprises: The training node encapsulates the gradient data calculated during the training model to obtain multiple gradient data packets; The training node estimates the bandwidth and determines the sending rate based on the estimated bandwidth; The training node sends the multiple gradient data packets to the parameter server that establishes an RUDP connection with itself at the sending rate; the packet header of the gradient data packet carries the priority of the gradient data, and the priority includes a high priority and a low priority; wherein the high priority gradient data is encoded using FEC; The parameter server repairs the high-priority gradient data and uses the Bubble-Filling mechanism to fill the lost low-priority gradient data when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold; The parameter server aggregates the repaired and filled gradient data to obtain aggregated gradient data, and sends the aggregated gradient data to the training node; The training node updates the model parameters according to the aggregated gradient data and continues training until the training is completed; The training node determines the sending rate based on the estimated bandwidth, including: At the beginning of each data packet transmission window, the initial sending rate is calculated based on the estimated bandwidth; During each packet transmission window, monitor the loss of gradient packets; when there is no packet loss within multiple consecutive round-trip delays, increase the sending rate; when packet loss is detected, adjust the sending rate according to the following process: If the actual packet loss rate is lower than the target packet loss rate of congestion control, increasing the sending rate; If the actual packet loss rate is higher than the target packet loss rate and lower than the Bubble-Filling packet loss rate threshold, maintain the current sending rate; If the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, the current sending rate is reduced to half.

2. The cross-data center distributed machine learning communication method according to claim 1, characterized in that: The training node performs bandwidth estimation, including: The current bandwidth is measured by adjusting the congestion window size during each round trip delay, triggering the accumulation and depletion of the network queue.

3. The cross-data center distributed machine learning communication method according to claim 1, characterized in that: The target packet loss rate is calculated by: The training node uses the least squares method to fit a first function of the average throughput with respect to the packet loss rate and a second function of the average delay with respect to the packet loss rate according to the historical average throughput and the historical average delay; and calculates the current target packet loss rate based on the first function and the second function.

4. The cross-data center distributed machine learning communication method according to claim 1, characterized in that: The cross-data center distributed machine learning communication method further includes: When the proportion of high-priority gradient data packets in the gradient data packets received by the parameter server reaches a preset proportion, feedback is sent to the training node to notify the training node to enter the next data packet transmission window period.

5. The cross-data center distributed machine learning communication method according to claim 1, characterized in that: The gradient data packet carries the meta information of the gradient tensor; the gradient tensor is a tensor formed by the gradient data contained in the gradient data packet; The meta information at least includes the sparsity of high-priority gradient data in the gradient data packet; The parameter server repairs high-priority gradient data, including: The parameter server repairs the high-priority gradient data according to the meta-information.

6. A cross-data center distributed machine learning communication system, characterized in that: The system includes multiple training nodes and at least one parameter server: The training node is used to encapsulate the gradient data calculated during the training model, and after obtaining multiple gradient data packets, perform bandwidth estimation, and determine the sending rate according to the estimated bandwidth; send the multiple gradient data packets to the parameter server that establishes an RUDP connection with itself according to the sending rate; the packet header of the gradient data packet carries the priority of the gradient data, and the priority includes a high priority and a low priority; wherein the high priority gradient data is encoded using FEC; The parameter server is used to repair the high-priority gradient data and fill the lost low-priority gradient data using the Bubble-Filling mechanism when the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold; perform gradient data aggregation on the repaired and filled gradient data to obtain aggregated gradient data, and send the aggregated gradient data to the training node; A training node, used to update model parameters according to the aggregated gradient data, and continue training until the training is completed; The training node determines the sending rate based on the estimated bandwidth, including: At the beginning of each data packet transmission window, the initial sending rate is calculated based on the estimated bandwidth; During each packet transmission window, monitor the loss of gradient packets; when there is no packet loss within multiple consecutive round-trip delays, increase the sending rate; when packet loss is detected, adjust the sending rate according to the following process: If the actual packet loss rate is lower than the target packet loss rate of congestion control, increasing the sending rate; If the actual packet loss rate is higher than the target packet loss rate and lower than the Bubble-Filling packet loss rate threshold, maintain the current sending rate; If the actual packet loss rate is higher than the Bubble-Filling packet loss rate threshold, the current sending rate is reduced to half.

7. The cross-data center distributed machine learning communication system according to claim 6, characterized in that: Training nodes to estimate bandwidth includes: The current bandwidth is measured by adjusting the congestion window size during each round trip delay, triggering the accumulation and depletion of the network queue.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to implement the steps of the cross-data center distributed machine learning communication method described in any one of claims 1-5 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the cross-data center distributed machine learning communication method described in any one of claims 1-5 are implemented.

Images