Large-scale distributed training system routing scheduling method based on network telemetering
By collecting and analyzing network link indicators at the data center node, predicting path quality, selecting the optimal path for traffic orientation, and combining with the ECMP protocol, the problem of poor load balancing and traffic perception capabilities of data centers is solved, and efficient data transmission is achieved and load balancing reduce deployment costs.
Patent Information
- Application Number
- CN202411249446.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-06
- Publication Date
- 2024-11-26
AI Technical Summary
It is difficult for existing data centers to achieve load balancing in large-scale artificial intelligence model training, traditional static routing strategies cannot meet the needs of efficient transmission, and the existing traffic balancing methods are inefficient in long-term traffic conflict scenarios and are difficult to deploy in production environments.
Using a large-scale distributed training system routing scheduling method based on network telemetry, collects and analyzes network link indicator information at the node end, uses time series to predict path quality, selects the optimal path for traffic orientation, and implements traffic routing in combination with ECMP protocol. Reduce deployment costs.
It improves the load utilization rate of the data center network, improves the transmission rate and training efficiency, simplifies the deployment process, and is suitable for large-scale lossless deployment.
Smart Images

Figure CN119030916A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of data center routing scheduling strategies, and in particular relates to a routing scheduling method for a large-scale distributed training system based on network telemetry. Background Art
[0002] In order to better support the efficient training of today's large-scale artificial intelligence models, high-performance clusters are usually designed with sufficient interconnection bandwidth in mind, so there are multiple end-to-end paths between any two hosts. In such a cluster, artificial intelligence model training tasks can choose better paths for parameter transmission and model synchronization. In the current data center architecture, the equal-cost multi-path routing (ECMP) algorithm based on quintuple hashing is generally used to randomly forward traffic on multiple equal-cost available paths. Generally speaking, this algorithm ensures that traffic is forwarded normally to multiple parallel links. However, when the network scale becomes larger and the amount of traffic increases, it is difficult for ECMP to achieve complete load balancing.
[0003] Furthermore, the existing deployment protocol ECMP and recent related work cannot fully utilize the over-provisioned network bandwidth. Due to the periodic traffic patterns on the host and switch sides, static routing strategies are far from meeting the requirements of AI workload routing. This is because any path that performs well in a static network may perform poorly in subsequent training task iterations, resulting in overall low training efficiency.
[0004] Therefore, for modern large-scale machine learning network data centers, existing research has proposed a host-side AI workload awareness and adaptive load balancing solution, which uses inband network telemetry (INT) to continuously capture the key attributes of the path, and then quantifies whether a path can provide high-quality services throughout the life cycle of the training workload based on a new metric of path time quality, and directs traffic to the path. The advantages of this new transmission strategy are: (1) It can perceive the time series indicators of the path, so as to better measure the performance of the traffic entering the path; (2) It selects the most suitable path for each training task based on the time quality of the path, thereby improving the traffic transmission efficiency of the overall task; (3) It performs path selection with minimal overhead and can be effectively deployed in data centers on a large scale without loss. However, this new transmission strategy also raises new challenges: (1) This process not only requires measuring the time attributes of the path, but also requires predicting future attributes in order to infer the performance of new tasks joining the transmission path and thus route the traffic. (2) The path awareness metric needs to include various factors (such as hidden factors such as PFC (Priority-based Flow Control)) to quantify the time quality of the path, but these factors are difficult to quantify using traditional measurement methods. (3) The node traffic direction method needs to be easy to deploy, low-cost, and compatible with the current switch routing strategy. Traditional traffic direction methods have too much overhead and are difficult to deploy on a large scale in production environments.
[0005] In summary, although the ECMP method itself has a certain balancing ability, it is easy to introduce long-term traffic conflicts in model training scenarios where traffic lasts for a long time and is prone to overlap, which hinders the model from improving training efficiency. In addition, existing traffic balancing methods are generally not designed for training traffic, and most require relatively large modifications to the network or customized functions in the switch, which has limitations in actual deployment. The effect can only be tried on a small scale in test machines, and it is difficult to actually deploy it in a production cluster. Summary of the invention
[0006] The purpose of the present invention is to provide a routing scheduling method for a large-scale distributed training system based on network telemetry. The present invention solves the problems that the existing load balancing implementation method in the distributed deep learning model training has poor effect on the specific traffic transmission distribution of the model on the network and the data center has poor perception of the traffic path. The present invention is an incremental method design for the existing network, which is only deployed on the nodes and does not modify the existing network environment. From the node perspective, the present invention can improve the transmission rate and exchange data faster between different nodes for model training; from the network perspective, the present invention can improve the load utilization of the link and save the cost of network equipment, which provides convenience for further model tuning and network operation and maintenance.
[0007] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0008] The present invention provides a large-scale distributed training system routing scheduling method based on network telemetry, comprising the following steps:
[0009] Step S1: Connect the node and the network device, upload the ECMP hash function to the node through the network device; enable INT to carry information on the network device; collect network link indicator information on the host side, and perform periodic analysis and prediction based on the network link indicator information;
[0010] Step S2: Execute a routing algorithm based on the path quality represented by the time attribute to calculate the optimal path of the task flow;
[0011] Step S3: For the nodes assigned with the training task, after selecting the optimal path with the minimum EFCT, that is, the highest time quality, the forced traffic is directed to be sent through the optimal path.
[0012] Furthermore, when the network device enables INT to carry information, after receiving an INT message sent by a node, the network device encodes its own link utilization and processing delay into the INT header of the data packet and forwards it to the next hop.
[0013] Furthermore, when the host side collects network link indicator information, each server runs INT to collect network status, any server individually selects a server in each rack as the destination server, and detects k random paths of the destination server; the source server sends an INT data packet to the destination server; the destination server receives the INT message and extracts the corresponding INT packet header; the destination server returns information carrying an ACK message; the source node host extracts and stores the INT information carried by the ACK message.
[0014] Furthermore, in step S2, the periodic analysis and prediction of the indicator time series are first performed, and the specific operation process is as follows:
[0015] Through INT-based monitoring, each server can collect the utilization rate and link delay of k links to any destination node over a period of time from the past to the present; let the observed time series be X, that is, x i (0 ≤ i < N), then predicting the subsequent signal requires inferring x N 、x N+1 、...; Among them, the observed time series consists of K periodic subsequences, and each subsequence has its own period T(j), that is:
[0016]
[0017] Use the autocorrelation function to find the period of the time series X; for a time series X, its autocorrelation function at period t is defined as the Pearson correlation coefficient between the time series X and the time series X with a time lag of t, that is:
[0018]
[0019] Then enumerate t from N / 2 to 2, and return the t with the maximum autocorrelation value as the period of the time series X.
[0020] Furthermore, in step S2, the specific operation process of calculating the optimal path of the task flow is as follows:
[0021] Use the expected flow completion time to measure the time quality of the path; the time quality measures the suitability of a path for task traffic transmission over a period of time; assume that the traffic f has N iterative communication phases, each communication phase needs to transmit S volume of data, and the duration of the calculation phase is t off ; If t i represents the completion time of the i-th communication phase transmission, then represents the bandwidth allocated at time t on path p; the completion time of traffic f on path p is calculated through the following formula, denoted as t N :
[0022]
[0023] Therefore, for each traffic f, its theoretical completion time on k potential paths P = {p1, p2,... pk} is obtained;
[0024] Furthermore, define a delay threshold d low to indicate whether the path is a PFC path, and the delay threshold d lowIt is set to 20% to 40% of the one-way basic delay. Paths with high available bandwidth and long delay are marked as PFC paths, and other paths are marked as non-PFC paths. The overall adaptability between flow f and path p is updated by the following formula:
[0025]
[0026] For each task, all observed paths between the source server and the destination server are taken as a set of candidate paths, the EFCT of each candidate path is calculated, and the path with the smallest EFCT is selected.
[0027] Furthermore, the EFCT of the path is recalculated every several training cycles. If the EFCT of a path is found to be smaller, the path is switched to that path. The path is switched only when the EFCT of the new path is smaller than the EFCT of the old path by a threshold and the node is in the calculation stage.
[0028] Furthermore, in step S3, a suitable source port is first selected. For each pair of communication groups of the task, all possible combinations of source ports and target paths are enumerated, and the next hop of each combination at each hop is calculated; if the next hop sequence of a combination is the target path, the source port is assigned to the communication pair; if no combination matches the target path, a source port matching the shortest path is returned.
[0029] Furthermore, in step S3, ECMP uses a hash function on the switch to select the next hop.
[0030] Furthermore, in step S3, the traffic is controlled to pass through a specific next hop by manipulating the source port, thereby obtaining a traffic routing path that complies with the ECMP protocol.
[0031] The beneficial effects of the present invention are:
[0032] The present invention designs a new routing scheduling strategy for the data center network of large-scale machine learning training, that is, combining the default ECMP of the network to build a distributed path selection module for each access host. The routing scheduling method of a large-scale distributed training system based on network telemetry proposed by the present invention can predict the path timing attributes based on INT monitoring information in the scenario of remote direct memory access (RDMA) communication in the data center, effectively solving the problem of poor perception of traffic paths in the data center, and facilitating further work such as load balancing.
[0033] The routing scheduling method of a large-scale distributed training system based on network telemetry proposed in the present invention can quantify the transmission quality of each path in the network in terms of time distribution at each distributed node. Specifically, each host collects network time attributes through INT and runs cycle analysis to expand the collected time domain attributes. Then, the time quality of the candidate path is calculated based on the link utilization and the inferred time series. Finally, the high-quality path is forced to be selected by controlling the traffic quintuple so that the traffic meets the target path in the switch ECMP protocol.
[0034] The routing scheduling method for a large-scale distributed training system based on network telemetry proposed in the present invention can improve transmission performance, increase the maximum concurrent capacity of the network, optimize training efficiency, and reduce the deployment cost of the data center network by analyzing and predicting the path timing quality. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is a flow chart of a routing scheduling method for a large-scale distributed training system based on network telemetry proposed by the present invention.
[0036] Figure 2 This is the experimental topology diagram. DETAILED DESCRIPTION
[0037] The present invention proposes a large-scale distributed training system routing scheduling method based on network telemetry. In terms of architecture, it is a solution that comprehensively utilizes the response information of the network, but the control logic is completely implemented on the host side. By deploying on the host side, no modification is required to the existing switch, so only minimal CPU overhead and memory usage are required. The module is coupled with RoCE communication through shared memory, and can be incrementally deployed in the existing cluster at a low cost without interfering with communication performance.
[0038] In terms of services, the present invention designs a transmission path load balancing strategy for AI workloads and adaptive traffic. It monitors the path status in a time series manner, predicts future properties based on the information obtained to select routes for traffic, selects the best path for each training task based on the time quality of the path, and enforces path selection with minimal overhead based on the existing ECMP.
[0039] A large-scale distributed training system routing scheduling method based on network telemetry of the present invention, such as Figure 1 The specific operation process is as follows:
[0040] Step S1: Collect link information, perform cycle analysis and prediction;
[0041] Step S1.1: deploying algorithm module on the host side;
[0042] Nodes and network devices are connected in a pre-designed way (e.g., fat tree or multi-rail topology), and network devices upload the ECMP hash function to the nodes for subsequent path calculations.
[0043] Step S1.2: The network device enables INT to carry information;
[0044] When receiving an INT message from a node, the network device encodes its own link utilization and processing delay into the INT header of the data packet and forwards it to the next hop;
[0045] Step S1.3: The host side collects network link indicators;
[0046] Each server needs to run INT to collect network status. Any server selects a server in each rack as the destination server and detects k random paths to the destination server. After the task is started, the source server will periodically send INT packets to the destination server. After receiving the INT message, the destination server extracts the corresponding INT header and returns it to the source node host via an ACK message. After receiving the ACK message, the source node host extracts the INT information carried and stores the indicator information of the switch that the ACK message passes through in order, hop by hop, into a file for subsequent routing algorithm analysis.
[0047] Step S2: executing a routing algorithm based on the path quality represented by the time attribute;
[0048] When the node is not assigned a task, the source server will periodically send INT packets to the destination server. When the node is assigned a task, it calculates the routing strategy for the upcoming task, and the node needs to predict the future properties of the link. The present invention estimates the available bandwidth and delay of each path in the future through a predictive analysis method based on the current and historical utilization, delay and other properties of each link in the network, and regards the future available bandwidth and delay of the path as the "time attribute" of the path. Based on these time attributes, the "time quality" indicator of the path can be defined. Specifically, the present invention will calculate the expected flow completion time (EFCT) of a task, and use EFCT as the time quality score of the task on the path.
[0049] Step S2.1: Cycle analysis and forecasting of indicator time series;
[0050] Through INT-based monitoring, each server can collect the utilization and link delay of k links for any destination node from the past to the present. The "periodic analysis" method of signal processing is used here, as follows:
[0051] Let the observed time series be X, that is, x iIf (0 ≤ i < N), then predicting the subsequent signal requires inferring x N , x N+1 ,... Among them, the observed time series consists of K periodic subsequences, and each subsequence has its own period T(j), that is:
[0052]
[0053] Then use the autocorrelation function to find the period of the time series X. For a time series X, its autocorrelation function at period t is defined as the Pearson correlation coefficient between the time series X and the time series X with a time lag of t, that is:
[0054]
[0055] Then enumerate t from N / 2 to 2, and return the t with the maximum autocorrelation value as the period of the time series X. Note that according to the Nyquist-Shannon sampling theorem, for N samples (sampling interval t), only periodic signals with a period not exceeding Nt / 2 can be found. In practice, N and t need to be set so that Nt is at least twice the maximum training round time.
[0056] Step S2.2: Calculate the optimal path of the task flow;
[0057] After completing the timing cycle analysis and prediction of the observation path, the routing algorithm needs to select a path with the highest time quality for the tasks assigned to the nodes. To quantify the adaptability of the timing path, the expected flow completion time (EFCT) is used here to measure the time quality of the path. The time quality measures the suitability of a path for task flow transmission over a period of time. For a specific path p and flow f, the calculation process is divided into two steps:
[0058] (1) Assume that the flow f has N iterative communication phases, and each communication phase needs to transmit S volume of data. The duration of the calculation phase is t off . If t i represents the completion time of the transmission in the i-th communication phase, then we can get where represents the bandwidth allocated at time t on path p. After obtaining the completion time t 1 of the transmission in the first communication phase, and so on, the completion time of the flow f on path p can be calculated through the following formula, denoted as t N :
[0059]
[0060] Therefore, for each flow f, its theoretical completion time on the k potential paths P = {p1, p2,... pk} can be obtained.
[0061] (2) The impact of PFC also needs to be considered when calculating EFCT. For the path where packets are blocked due to PFC pause, the available bandwidth and delay are both high, so a delay threshold d is defined. low To indicate whether the path is a PFC path, where d low Set to 20% to 40% of the one-way basic delay. Paths with high available bandwidth and long delay are marked as PFC paths (if D (p) >d low , then PFC p =1), other paths are marked as non-PFC paths (if D (p) ≤dlow, then PFC p =0). Therefore, the overall adaptability between flow f and path p can be updated by the following formula:
[0062]
[0063] For each task, the present invention takes all observed paths of the source server and the destination server as a set of candidate paths, calculates the EFCT of each candidate path, and selects the path with the minimum EFCT.
[0064] The present invention recalculates the EFCT of a path every several training cycles, and switches to the path if the EFCT of a path is found to be smaller. To avoid frequent path switching, the path is switched only when the EFCT of the new path is less than the EFCT of the old path by a threshold and the node is in the calculation stage.
[0065] Step S3: node forced path selection;
[0066] For nodes assigned with training tasks, after enumerating the time quality scores of all candidate paths and selecting the path with the minimum EFCT, i.e., the highest time quality, the traffic will be forced to be sent through the selected path. The enforcement design of the present invention is compatible with the ECMP protocol, i.e., the traffic is forced to follow the target path by setting the five-tuple field in the message header.
[0067] Step S3.1: Select a suitable source port;
[0068] The INT data packet sent by the server is a UDP data packet, in which a five-tuple containing the IP addresses of the two servers, the source server and the destination server, is used for routing, and the destination port is a special port reserved by INT. The present invention detects k random paths by adjusting the value of the source port, that is, randomly assigning k values. For each pair of communication groups of the task, all possible combinations of source ports and target paths are enumerated, and the next hop of each combination at each hop is calculated. If the next hop sequence of a combination is the target path, the source port is assigned to the communication pair. If no combination matches the target path, a source port matching the shortest path is returned.
[0069] Step S3.2: Forced planning of traffic paths;
[0070] ECMP uses a hash function on the switch to select the next hop, where the input of the hash function is a 5-tuple (source IP, destination IP, source port, destination port, protocol number) and a random number seed. By default, in RoCE communication, the source port is automatically selected by the network card within a port range. In the present invention, by manipulating the source port, the traffic can be controlled to pass through a specific next hop, and a traffic routing path that complies with the ECMP protocol can be obtained.
[0071] The present invention can obtain the matching degree between the path traffic cycle and the RDMA flow of the node by observing the path quality and calculating by analysis method, and at the same time, based on the selection result, select the corresponding source port by ECMP hash function calculation to force the traffic to be directed to the selected path, so as to realize RDMA flow level load balancing.
[0072] The present invention utilizes a network path quality observation tool that combines the end and the network. By regularly constructing and sending INT type detection messages corresponding to different pods on the end side, configuring and enabling the INT function on the network side and collecting, analyzing and storing the returned message information on the end side, a path timing quality observation tool is implemented, which can analyze the adaptability of the path and node traffic from a time perspective.
[0073] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0074] The existing network routing method is ECMP based on quintuple hashing, which randomly selects forwarding among multiple equivalent paths. Assume that the current network distribution topology is as follows Figure 2As shown in the figure, the leaf layer of the switch is sw1, ..., sw4 from left to right, and the spine layer is sw5, sw6 from left to right. There are three tasks in the system waiting to be assigned to the node for execution. The direction and quantity of data flow between hosts are unknown. Assuming that the start time is the unix timestamp 1785083866, each host has only one RDMA network card (Remote Network Interface Card, RNIC), and the corresponding relationship is host1→RNIC1, ..., host8→RNIC8. The current real-time task allocation status is shown in Table 1.
[0075] Table 1 Real-time task status table
[0076]
[0077] Task 1 is assigned to the host group of host1 / host3, task 2 is assigned to the host group of host2 / host4 / host6, and task 3 is assigned to the host group of host5 / host7 / host8. All three tasks use data parallelism to accelerate training (i.e., each host contains a complete backup of the model data, and each round of iteration trains a part of the model separately, which is merged and updated after communication and data exchange).
[0078] Assume that allreduce communication is used between hosts in the cluster and the fixed processing delay of the switch is 60μs. The specific monitoring process is as follows:
[0079] (1) End-side monitoring is based on INT technology. Each host randomly monitors several paths in the network. Assume that each host monitors three paths. Initially, there is no traffic on the paths observed by all nodes, and the path_info record data of each path is empty.
[0080] The RNIC of each host forwards the INT packet to the corresponding selected path according to the program logic and waits for the result to be returned, thereby obtaining the path information. Assuming that each host sends an INT detection packet every 0.1s, and the monitoring program reads the corresponding data of the INT packet returned by the RNIC, then by querying the information at fixed intervals, the traffic size sent by each path in each time period can be obtained, thereby obtaining the path-level throughput information.
[0081] When task1 enters, since no computing tasks have been assigned to any nodes in the cluster before, the network traffic is empty, and the corresponding path information collected is also empty. Therefore, each node follows the algorithm logic and randomly selects a path to send traffic. When task1 enters, the traffic of the training task begins to appear in the network, and the detection program of each node continues to work. If the switch recognizes the INT packet header, it writes the corresponding information (port throughput, packet processing delay, etc.) into the metadata of the INT packet header. After the corresponding destination host receives it, it extracts the metadata and carries it in the ack packet and returns it to the source sending node. The information returned by each host after receiving an INT packet is shown in Table 2.
[0082] Table 2 int_info schematic table
[0083] Hop Count Switch outbound port Throughput (Gbps) Delay(μs) 1 0 22 60 2 1 33 102 3 3 22 62
[0084] (2) After receiving the returned INT packet, the node's network card forwards it to the application for parsing and records the corresponding information path_info in chronological order. Taking the host1 node as an example, the format recorded at a certain moment is shown in Table 3.
[0085] Table 3 path_info schematic table
[0086]
[0087]
[0088] Each host-side device then records and saves the monitoring data on the end-side at a regular interval, and analyzes it when the task is actually assigned, so as to select the appropriate path.
[0089] (3) According to Table 1, when task 2 is assigned to the corresponding node, the node checks the stored monitoring data and performs period calculation and pattern matching on the path through the analysis program to obtain the flow matching degree at the path level. Taking the monitoring data recorded up to the time of task 2 assignment as an example, the analysis results are shown in Table 4.
[0090] Table 4 analysis_info schematic table
[0091]
[0092] Then, each host device selects the optimal path based on the analysis results. That is, by matching the cycle and start time of the traffic passing through the path, if the transmission mode of the task traffic of the current node enters the path, the overlap between the transmission mode of the traffic and the transmission mode of the original traffic is minimal, so that the expected flow completion time is the shortest, then the path is the optimal path, and the traffic is allocated to the path for transmission.
[0093] (4) Initially, the network service system needs to collect the hash mapping relationship function of the switch and upload it to the end-side node storage. After completing the route selection, k source port values will be randomly assigned to the node to detect k random paths. For each pair of communication groups of the task, all possible combinations of source ports and target paths will be enumerated, and the next hop of each combination at each hop will be calculated. If the next hop sequence of a combination is the target path, the source port is assigned to the communication pair. The port-path correspondence table calculated by the node is shown in Table 5.
[0094] Table 5 port_path schematic table
[0095] port Port corresponding path The selected path id The path you choose 4123 sw2->sw6->sw3 2 sw2->sw6->sw4 4623 sw2->sw6->sw3 2 sw2->sw6->sw4 41278 sw2->sw6->sw3 2 sw2->sw6->sw4 21341 sw2->sw6->sw3 2 sw2->sw6->sw4 23423 sw2->sw6->sw3 2 sw2->sw6->sw4 2535 sw2->sw5->sw3 2 sw2->sw5->sw3
[0096] After the calculation is completed, the node selects the port corresponding to the optimal path and assigns the source port number to the flow as the source port number of the data packet of the corresponding flow, so that the path of the corresponding flow is routed according to the selected optimal result.
[0097] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A routing scheduling method for a large-scale distributed training system based on network telemetry, characterized in that: The following steps are involved: Step S1: Connect the node and the network device, and upload the ECMP hash function to the node through the network device; The network device enables INT to carry information; the host side collects network link indicator information and performs periodic analysis and prediction based on the network link indicator information; Step S2: Execute a routing algorithm based on the path quality represented by the time attribute to calculate the optimal path of the task flow; Step S3: For the nodes assigned with the training task, after selecting the optimal path with the minimum EFCT, that is, the highest time quality, the forced traffic is directed to be sent through the optimal path.
2. According to claim 1, a large-scale distributed training system routing scheduling method based on network telemetry is characterized in that: When the network device enables INT to carry information, after receiving an INT message sent by a node, the network device encodes its own link utilization and processing delay into the INT header of the data packet and forwards it to the next hop.
3. According to claim 1, a large-scale distributed training system routing scheduling method based on network telemetry is characterized in that: When the host side collects network link indicator information, each server runs INT to collect network status, and any server selects a server in each rack as the destination server and detects k random paths to the destination server; The source server sends an INT packet to the destination server; The destination server receives the INT message and extracts the corresponding INT packet header; The destination server returns information carrying an ACK message; the source node host extracts and stores the INT information carried in the ACK message.
4. A large-scale distributed training system routing scheduling method based on network telemetry according to claim 1, characterized in that: In step S2, the periodic analysis and prediction of the indicator time series are first performed, and the specific operation process is as follows: Through INT-based monitoring, each server can collect the utilization rate and link delay of k links to any destination node over a period of time from the past to the present; let the observed time series be X, i.e., x i (0 ≤ i < N), then predicting subsequent signals requires inferring x N , x N+1 ,...; Among them, the observed time series consists of K periodic subsequences, each of which has its own period T(j), that is: in Use the autocorrelation function to find the period of time series X; for a time series X, its autocorrelation function at period t is defined as the Pearson correlation coefficient between time series X and time series X lagged at time t, that is: Then enumerate t from N / 2 to 2 and return the t with the maximum autocorrelation value as the period of time series X.
5. According to claim 1, a large-scale distributed training system routing scheduling method based on network telemetry is characterized in that: In step S2, the specific operation flow of calculating the optimal path of the task flow is as follows: The time quality of the path is measured using the expected flow completion time; the time quality measures the suitability of a path for task traffic transmission over a period of time; assuming that the flow f has N iterative communication phases, each communication phase requires the transmission of S volume of data, and the duration of the calculation phase is t off ; If you use t i represents the completion time of the transmission in the i-th communication phase, then represents the bandwidth allocated on path p at time t; The completion time of flow f on path p is calculated by the following formula, denoted as t N : Therefore, for each flow f, its theoretical completion time on k potential paths P = {p1, p2, ... pk} is obtained.
6. A large-scale distributed training system routing scheduling method based on network telemetry according to claim 5, characterized in that: Define the delay threshold d low To indicate whether the path is a PFC path, the delay threshold d low It is set to 20% to 40% of the one-way basic delay. Paths with high available bandwidth and long delay are marked as PFC paths, and other paths are marked as non-PFC paths. The overall adaptability between flow f and path p is updated by the following formula: For each task, all observed paths between the source server and the destination server are taken as a set of candidate paths, the EFCT of each candidate path is calculated, and the path with the smallest EFCT is selected.
7. A large-scale distributed training system routing scheduling method based on network telemetry according to claim 5, characterized in that: The EFCT of the path is recalculated every few training cycles. If the EFCT of a path is found to be smaller, it is switched to that path. The path is switched only when the EFCT of the new path is less than the EFCT of the old path by a threshold and the node is in the calculation stage.
8. A large-scale distributed training system routing scheduling method based on network telemetry according to claim 1, characterized in that: In step S3, firstly, a suitable source port is selected. For each communication pair of the task, all possible combinations of source ports and target paths are enumerated, and the next hop of each combination at each hop is calculated; if the next hop sequence of a combination is the target path, the source port is assigned to the communication pair; If none of the combinations matches the destination path, then a source port that matches the shortest path is returned.
9. A large-scale distributed training system routing scheduling method based on network telemetry according to claim 1, characterized in that: In step S3, ECMP uses a hash function on the switch to select the next hop.
10. A large-scale distributed training system routing scheduling method based on network telemetry according to claim 1, characterized in that: In step S3, the traffic is controlled to pass through a specific next hop by manipulating the source port, thereby obtaining a traffic routing path that complies with the ECMP protocol.
Citation Information
Cited By
Intelligent computing center path distribution method, data transmission method and network system
CN121283926A
