A SLO-driven flow scheduling system and method

Abstract

A kind of SLO driving-based flow scheduling system, including Sketch data structure deployed on network card and dynamic priority scheduling system deployed on switch;Sketch data structure monitors job communication mode, and dynamic priority scheduling system adjusts the priority of the data packet in the same iteration number of current job according to the current job communication mode monitored on network card by Sketch data structure.The SLO driving-based flow scheduling system of the application strictly executes the SLO of tenant, while optimizing resource efficiency;The SLO driving-based flow scheduling system of the application eliminates initialization overhead, provides strong SLO guarantee by accurate priority ordering, and maintains hardware-friendly deployability, while resource occupation is small.

Description

Technical Field

[0001] This invention relates to a flow scheduling system and method, and more particularly to a flow scheduling system and method based on SLO (Streaming Loop) driving. Background Technology

[0002] Network communication has become a core bottleneck in distributed DNN training within shared GPU clusters. Its inherent iterative computation-communication model inevitably leads to overlapping communication phases for concurrent jobs in multi-tenant environments, resulting in severe network contention (Sudarsanan Rajasekaran, Manya Ghobadi, et al. Congestion control in machine learning clusters. In Proceedings of the 21st ACM Workshop on HotTopics in Networks, pages 235–242, 2022.). This not only directly reduces the training speed of individual jobs but also leaves expensive computing resources idle while GPUs wait for network data, significantly reducing the overall utilization of the cluster. Therefore, efficient and intelligent network flow scheduling is a key means to coordinate concurrent communication needs, optimize limited bandwidth allocation, and overcome this bottleneck.

[0003] However, most current scheduling schemes suffer from an objective bias: they overemphasize improving system-level efficiency metrics, such as cluster throughput and GPU utilization, while severely neglecting the core needs of tenants in multi-tenant scenarios, namely, ensuring network service level objectives (SLOs). This bias manifests first in the lack of a clear definition of SLO for DNN training networks, and further leads to two key defects: 1) Violating efficiency-first SLOs: To improve system efficiency, such as GPU utilization, solutions, such as CRUX (Jiamin Cao, Yu Guan, et al. Crux: GPU-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 1–15, 2024.), often actively employ unfair scheduling strategies, such as GPU intensity priority. This systematically degrades the network performance of low-priority jobs, causing their training time to be much longer than expected. 2) Lack of SLO guarantee mechanism or fragile mechanism: Existing mechanisms are difficult to reliably guarantee SLO; efficiency-oriented solutions, such as CRUX and MLTCP (Sudarsanan Rajasekaran, Sanjoli Narang, et al. Mltcp: A distributed technique to approximate centralized flow scheduling for machine learning. In Proceedings of the 23rd ACM Workshop on Hot Topics in Networks, pages 167–176, 2024.), inevitably sacrifice the SLO of some jobs in pursuit of global goals.However, D2TCP (which is not optimized for DNN training clusters) lacks dynamic adaptability, and CASSINI (Sudarsanan Rajasekaran, Manya Ghobadi, et al. Cassini: Network-aware jobscheduling in machine learning clusters. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1403–1420, 2024.) relies too heavily on job compatibility, is susceptible to network jitter, and its SLO guarantee robustness is severely inadequate.

[0004] Ignoring SLO safeguards has serious and widespread consequences: 1) Tenant level: Delayed delivery of training tasks, harming user experience and productivity. In commercial scenarios, this is more likely to trigger Service Level Agreement (SLA) penalties, resulting in economic losses and reputational risks. For example, if a single instance's availability time is less than 95% in a month, the tenant will receive 100% of the monthly service fee as compensation (Tencent Cloud. Tencent cloud virtual machine service level agreement, 2024. Effective from February 1, 2024. Last updated: January 23, 2024.). 2) Short-lived job dilemma: This problem is particularly severe because more than 30% of jobs in the cluster have very short lifecycles (< 10 minutes) (Jiamin Cao, Yu Guan, et al. Crux: GPU-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 1–15, 2024.). For these jobs, a brief SLO violation (e.g., due to unfair scheduling) means that much of their lifespan is delayed, or they may even be terminated prematurely due to "timeout". Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a flow scheduling system and method based on SLO that strictly enforces user SLO while optimizing resource efficiency.

[0006] To address the aforementioned technical problems, the present invention proposes the following technical solution: a flow scheduling system based on SLO (Streaming Loop) driver, comprising a Sketch data structure deployed on a network interface card (NIC) and a dynamic priority scheduling system deployed on a switch; the Sketch data structure monitors job communication modes, and the dynamic priority scheduling system adjusts the priority of data packets within the same iteration number of the current job based on the communication mode of the current job monitored by the Sketch data structure on the NIC.

[0007] In the aforementioned SLO-driven flow scheduling system, preferably, the Sketch data structure includes Size_sketch, Time_sketch, and Count_sketch; Size_sketch adopts a probabilistic data structure CMS, which includes k arrays of length m, each array being associated with a different hash function. When a data packet arrives, it is hashed by the k hash functions respectively, mapped to a certain position in the corresponding array, and the counter at that position is incremented to complete the counting.

[0008] Time_sketch captures the timestamps (i.e., time steps) of arriving packets to infer iteration boundaries;

[0009] When Count_sketch identifies an iteration replacement of a stream, it assigns the total amount of data in size_sketch for one iteration to count_sketch and clears size_sketch to zero.

[0010] In the aforementioned SLO-driven flow scheduling system, preferably, the dynamic priority scheduling system maps the calculation result of priority scheduling to the priority field of all data packet headers in the next iteration according to the urgency level Ui of job i.

[0011] The ;

[0012] y′ represents the number of iterations that the task has completed so far;

[0013] Ai(0, y′) represents the actual completion time of iterations 0-y′;

[0014] Ei(0, y′) represents the expected completion time of iterations 0-y′;

[0015] When Ui > 1, it indicates that an SLO violation has occurred; when Ui ≤ 1, the task is within the allowed SLO range.

[0016] In the aforementioned SLO-driven flow scheduling system, preferably, the dynamic priority scheduling system deploys a RunID for each switch port to record the flow ID sent to that port;

[0017] When a data packet arrives, the RunID is checked. If the RunID is empty, the flow ID and priority of the current data packet are recorded, and the current data packet is transmitted.

[0018] If RunID is not empty, compare the Runflag flag of the highest priority stream on the current port. If the new stream has a higher priority, update both values ​​and transmit the data packets of the new stream.

[0019] If the priority of a new flow is the same as the Runflag of the highest priority flow on the current port, the newly arrived data packet is attenuated to the next priority queue, and the switch port continues to execute the transmission of the previous flow.

[0020] A flow scheduling method based on SLO (Streaming Loop) includes the following steps;

[0021] 1) A Sketch data structure deployed on the network interface card (NIC) and a dynamic priority scheduling system deployed on the switch. The dynamic priority scheduling system assigns a RunID to each switch port to record the flow ID sent to that port. The Sketch data structure includes Size_sketch, Time_sketch, and Count_sketch. Size_sketch uses a probabilistic data structure (CMS), which includes k arrays of length m, each array being associated with a different hash function.

[0022] 2) When a data packet arrives at the network card, the Sketch data structure obtains the size of the data packet and the iteration boundary; the dynamic priority scheduling system determines the priority of the next iteration of the current processing flow;

[0023] 3) Within the same iteration cycle; when a data packet arrives, the dynamic priority scheduling system checks the RunID. If the RunID is empty, it records the flow ID and Runflag of the current data packet and transmits the current data packet.

[0024] If RunID is not empty, compare the Runflag flag of the highest priority stream on the current port. If the new stream has a higher priority, update both values ​​and transmit the data packets of the new stream.

[0025] If the priority of a new flow is the same as the Runflag of the highest priority flow on the current port, the newly arrived data packet is attenuated to the next priority queue, and the switch port continues to execute the transmission of the previous flow.

[0026] In the aforementioned SLO-driven flow scheduling method, preferably, the Sketch data structure includes Size_sketch, Time_sketch, and Count_sketch; Size_sketch adopts a probabilistic data structure CMS, which includes k arrays of length m, each array being associated with a different hash function. When a data packet arrives, it is hashed by the k hash functions respectively, mapped to a certain position in the corresponding array, and the counter at that position is incremented to complete the counting.

[0027] Time_sketch captures the timestamp of the arrival of the data packet, i.e. the time step, to infer the iteration boundary;

[0028] When Count_sketch identifies an iteration replacement of a stream, it assigns the total amount of data in size_sketch for one iteration to count_sketch and clears size_sketch to zero.

[0029] In the above-mentioned SLO-driven flow scheduling method, preferably, the dynamic priority scheduling system maps the calculation result of priority scheduling to the priority field of all data packet headers in the next iteration according to the urgency level Ui of job i.

[0030] The ;

[0031] y′ represents the number of iterations that the task has completed so far;

[0032] Ai(0, y′) represents the actual completion time of iterations 0-y′;

[0033] Ei(0, y′) represents the expected completion time of iterations 0-y′;

[0034] When Ui > 1, it indicates a SLO violation has occurred; the larger the Ui value, the more severe the timeout, and the scheduling system should assign a higher priority to reduce the risk. Conversely, when Ui ≤ 1, the task is within the SLO allowable range, and its priority can be reduced to optimize resource utilization.

[0035] Compared with the prior art, the advantages of the present invention are as follows: the SLO-driven flow scheduling system of the present invention strictly enforces the tenant's SLO while optimizing resource efficiency; the SLO-driven flow scheduling system of the present invention eliminates initialization overhead, provides strong SLO guarantees through precise priority sorting, maintains hardware-friendly deployability, and has low resource consumption. Attached Figure Description

[0036] Figure 1 The impact of SP / WRR / RR scheduling modes on JCT compared to SOLO.

[0037] Figure 2 This is an example of traffic patterns on a bottleneck link.

[0038] Figure 3 This is a flowchart of the SLO-driven flow scheduling system in Example 1.

[0039] Figure 4 This is the priority mapping diagram based on urgency factors in Example 1.

[0040] Figure 5 This is a diagram illustrating how streams with the same priority will decay to the next queue.

[0041] Figure 6 A diagram illustrating how a switch collects information about all DNN flows passing through a local device and aggregates flows with the same next hop.

[0042] Figure 7 The distribution of SLO violation severity under different load conditions and different schemes.

[0043] Figure 8 This represents the distribution of SLO violations under mixed loads across multiple bottleneck links; where M represents the mainstream and A represents all flows.

[0044] Figure 9 A graph showing the impact of job information collection costs on the short job SLO violation rate.

[0045] Figure 10 The average actual / SOLO JCT under load B oversubscription.

[0046] Figure 11 shows the JCT distribution of the DNNS training task with priority decay mechanism enabled and disabled. Detailed Implementation

[0047] To facilitate understanding of the present invention, the present invention will be described more fully and in detail below with reference to preferred embodiments, but the scope of protection of the present invention is not limited to the following specific embodiments.

[0048] It should be noted that when a component is described as being "fixed to, attached to, connected to or connected to" another component, it can be directly fixed to, attached to, connected to or connected to the other component, or it can be indirectly fixed to, attached to, connected to or connected to the other component through other intermediate connectors.

[0049] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the invention.

[0050] This section outlines the essential knowledge, then defines the SLO (Sum of Result) for networks that share a DNN training set to complete the task, and finally identifies the key research gaps that guarantee the SLO.

[0051] A. Multi-tenant GPU clusters and communication contention

[0052] Distributed DNN training has evolved into a key solution for handling large-scale datasets and complex model architectures by processing workloads in parallel across multiple GPU workers. The training process follows an iterative pattern, where each iteration alternates between two distinct phases:

[0053] 1) Computation: Involves forward / backward propagation to compute local gradients on the GPU worker;

[0054] 2) Communication: Network-intensive gradient synchronization between workers is required via full reduction or parameter server mode.

[0055] This periodic alternation generates tightly coupled computational communication cycles, in which models with millions of parameters generate a large number of gradient exchanges during synchronization, making network bandwidth limitations and latency spikes critical bottlenecks.

[0056] The efficiency of these communication phases directly determines the training throughput, because any contention on the shared network link will reduce the overall system performance.

[0057] The proliferation of large-scale DNN models (e.g., GPT-3 with 175 billion parameters [9]) and datasets (e.g., LAION-5B with 5.85 billion image-text pairs

[10] ) has driven the adoption of multi-tenant GPU clusters, where heterogeneous DNN jobs share pooled computational and network resources. This model stems from two technical requirements: 1) the intensive nature of GPU infrastructure, which requires maximizing utilization through resource reuse; and 2) the iterative nature of distributed training, which alternates between computational phases (forward / backward propagation) and communication phases (gradient synchronization via AllReduce).

[0058] B. Definition of Network SLO in Multi-tenant Training

[0059] As mentioned earlier, ensuring network SLO for tenants, especially for short jobs, is crucial for improving cluster efficiency and tenant satisfaction. This invention defines the speedup γi for the completion time of job i from iteration 0 to iteration ni (0 < ni, where ni is the total number of iterations for job i) as:

[0060]

[0061] Where: JCTi(0, ni) represents the total job completion time (JCT) required for job i to actually complete from start to finish in a shared cluster environment. JCTSOLOi(0, ni) represents the baseline total time required for job i to complete from start to finish in an ideal environment with exclusive network resources (no contention).

[0062] Therefore, γi quantifies the increase in job iteration execution time due to network resource contention. γi = 1 indicates no contention effect; γi > 1 indicates latency caused by contention. In shared clusters, completely avoiding contention is usually impractical or uneconomical, so γi = 1. Tenants can accept a certain degree of performance degradation in exchange for the cost benefits of resource sharing.

[0063] ci is the SLO threshold coefficient for job i (or its category), determined through negotiation between the tenant and the cluster provider. Its actual completion time JCTi(0, ni) is at most a multiple of its ideal contention-free time JCTSOLOi(0, ni). That is, γ i ≤c i .

[0064] This invention defines the network SLO metric for job i as: for the entire job iteration interval (0, ni), γ i The coefficient c specified by the tenant i constraint:

[0065]

[0066] This metric is defined as the ratio of the actual completion time of a task to the user's maximum acceptable completion time. Its core value lies in quantifying whether the timeliness of task execution meets the Service Requirement (SLO) requirements. When SLO_ind > 1, it indicates that an SLO violation has occurred (i.e., the actual completion time exceeds the tenant's tolerance threshold); when SLO_ind ≤ 1, it indicates that the JCT is within the service commitment range.

[0067] Controllable resources are reserved for low-priority jobs. This invention uses NS3 to simulate the DNN training process on a bottleneck link. This invention runs two randomly generated DNN training workloads on a 100Gbps bottleneck link. The DNN training workloads include multiple "communication and stop" iterations. This invention uses round-robin scheduling (RR), weighted round-robin scheduling (WRR), and strict priority (SP) to schedule these three jobs, and defines their respective ci coefficients, where c1 = 1.1 and c2 = 1.39. Figure 1 As shown, SP helped high-priority task 1 achieve performance almost equivalent to SOLO, but severely hampered the performance of job 2 (even worse than RR). In contrast, WRR's fixed weight allocation (2:1) cannot dynamically sense and respond to the actual SLO urgency of the job. Job 1 has more stringent SLO requirements, thus requiring relatively more resources than Job 2 to meet its stricter constraints, but the fixed 2:1 weight cannot provide this differentiated guarantee.

[0068] The experiments clearly verified the following: 1) Unfair scheduling (SP) systematically impairs the SLO (Solution Time Limit) of low-priority jobs; 2) Static weighted scheduling (WRR) struggles to accurately match dynamic SLO requirements (ci), which can easily lead to some or all jobs violating their SLOs. This supports the necessity of developing dynamic, SLO-aware scheduling mechanisms.

[0069] However, designing a measurable and meaningful network SLO for DNN training jobs presents a unique challenge: the total number of training iterations is often unknown and dynamically changing, making it difficult to apply the traditional SLO definition based on total JCT.

[0070] Traffic patterns on bottleneck links, such as Figure 2 As shown, the focus is on the traffic switching between two jobs. Since job i is assigned a high priority, the traffic switching mode can be divided into two categories: preemption and alternation. During preemption, the high-priority job occupies the bandwidth of all low-priority jobs. During alternation, the communication phase of the previous job ends, and the bandwidth is either idle or occupied by the next job. At this time, no jobs are waiting.

[0071] In this invention, job j must complete three communications before the deadline to guarantee the Service Level Objective (SLO), as shown by the dashed line in Figure 2. However, because job i has a higher priority and preemption occurs during job j's second communication, job j violates the SLO requirement. Ideally, job j only needs to eliminate one preemption to avoid the SLO violation. Therefore, the network SLO guarantee problem for multi-tenant training jobs under a single bottleneck link is abstracted as a resource contention problem in a single-processor system, which can be solved by the traditional EDF algorithm, which assigns higher priority to jobs closer to the deadline. However, some key issues still need to be addressed.

[0072] The first issue lies in the architecture selection and real-time guarantee mechanism design of the EDF algorithm. To achieve dynamic priority scheduling, it's necessary to continuously collect workload status and perform real-time evaluation, thereby triggering a dynamic priority adjustment mechanism. While a centralized architecture can uniformly manage job queues, the bottleneck effect caused by competition for computing resources can easily lead to the failure of real-time guarantees. A distributed architecture, on the other hand, needs to handle conflicts arising from concurrent preemption of resources with the same priority, which may cause the scheduling mechanism to degenerate into a non-preemptive, fair resource competition mode, weakening the effectiveness of the EDF algorithm's deadline constraints.

[0073] Secondly, the appropriate frequency of priority adjustments needs to be carefully weighed. While high-frequency adjustment strategies can quickly respond to system dynamics, frequent context switching increases system overhead and resource fragmentation. Low-frequency adjustment strategies can reduce the scheduler's computational load, but due to priority update delays, the risk of violating Service Level Objectives (SLOs) is difficult to avoid. Even with the highest priority, deadlines may still be invalid due to decision-making delays.

[0074] C. The burden of collecting scheduling information for short-term training tasks.

[0075] Existing scheduling schemes (e.g., Jiamin Cao, Yu Guan, et al. Crux: GPU-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 1–15, 2024.) rely on offline collection and analysis of prior features of DNN training jobs (e.g., computation / communication phase duration, single iteration throughput), which typically requires monitoring independently running iterations, with computation deployment latency ranging from 30 seconds to several minutes.

[0076] This mechanism is particularly disadvantageous for short-time jobs: in real-world clusters, over 30% of jobs complete within 10 minutes [3, 8], while job collection itself can incur up to 5% additional time cost (for a 10-minute job, a 30-second collection process accounts for 5%). This not only limits the cluster's scalability but also poses a direct challenge to tenants' real-time requirements and economic costs, especially when network SLOs require synchronous monitoring of task progress and deadlines; traditional information collection mechanisms become a heavy burden for short-time jobs.

[0077] Example 1

[0078] like Figure 3 The illustrated SLO-driven flow scheduling system includes a Sketch data structure deployed on the network interface card (NIC) and a dynamic priority scheduling system deployed on the switch. The Sketch data structure monitors job communication patterns, and the dynamic priority scheduling system adjusts the priority of data packets within the same iteration number of the current job based on the communication patterns of the current job monitored by the Sketch data structure on the NIC.

[0079] In this embodiment, the Sketch data structure includes Size_sketch, Time_sketch, and Count_sketch; Size_sketch adopts a probability data structure CMS, which includes k arrays of length m. Each array is associated with a different hash function. When a data packet arrives, it is hashed by the k hash functions respectively, mapped to a certain position in the corresponding array, and the counter at that position is incremented to complete the counting.

[0080] Time_sketch captures the timestamps of arriving packets to infer iteration boundaries;

[0081] When Count_sketch identifies an iteration replacement of a stream, it assigns the total amount of data in size_sketch for one iteration to count_sketch and clears size_sketch to zero.

[0082] In this embodiment, the dynamic priority scheduling system maps the calculation result of priority scheduling to the priority field of all data packet headers in the next iteration based on the urgency level Ui of job i.

[0083] The ;

[0084] y′ represents the number of iterations that the task has completed so far; that is, after each iteration, LongLiu will recalculate Ui and guide the next iteration.

[0085] A i (0, y′) represents the actual completion time of iterations 0-y′;

[0086] E i (0, y′) represents the expected completion time of iterations 0-y′;

[0087] When U i When the value is greater than 1, it indicates an SLO violation has occurred; when U i When ≤ 1, the task is within the range allowed by SLO;

[0088] In this embodiment, E i (0, y′) represents the expected completion time for a number of iterations, which can be expressed as Ei(0,y′) = c i ˙JCTSOLO i (0, y′). For a known training load, E i (0, y′) can be completely calculated. The SOLO time for each iteration can be estimated by dividing the measured amount of iteration data by the link bandwidth and adding the computation time. JCTSOLO i (0, y′) is calculated by multiplying that time by the number of iterations. A i (0, y′) represents the actual completion time of the training task, which is affected by factors such as communication contention. The data exchange of a single iteration of the distributed training workload is statistically stable, and therefore can be used as a quantitative benchmark for job scheduling urgency. In this embodiment, the iteration cycle of the training task is the basic unit for observing job urgency, so as to reduce the overhead caused by frequent priority adjustments.

[0089] U i This quantifies the degree to which a task adheres to the Service Level Objective (SLO) from initiation to the current iteration cycle. An absolute value of 1 is taken as the critical threshold; when U... i A value >1 indicates an SLO violation has occurred. i The larger the value, the more severe the timeout, and the higher the priority the scheduling system should assign to it to reduce the risk. Conversely, when U... i When the priority is ≤ 1, the task is within the SLO limit and can be lowered to optimize resource utilization.

[0090] In this SLO-driven flow scheduling system, the sender maintains the urgency coefficient U through a dynamic update mechanism. iThe real-time performance is ensured. The specific process is as follows: Based on real-time monitoring of the training load traffic characteristics, the sender quantitatively evaluates key performance indicators (e.g., A) in the y-th iteration. i and E i The calculation result is then mapped to the priority field of all data packet headers in the next iteration, with a time complexity of O(1). In other words, the time complexity of this mapping behavior is O(1). At the end of the next iteration, y′ is incremented by 1, and the process restarts. Figure 4 As shown, Ui is discretized into 7 priorities. Among them, when U... i When the value is less than 1, it will be mapped to the two lowest priorities; otherwise, it will be mapped to one of the five higher priorities excluding the highest priority. Furthermore, the switch is used to perform priority-based scheduling decisions; on bottleneck links, the switch schedules jobs according to strict priorities.

[0091] In this embodiment, although the granularity of priority monitoring and adjustment is extended from the data packet level to the job iteration level, global real-time job monitoring based on a centralized architecture still requires real-time collection of job transmission status and identification of iteration patterns to adjust priorities, which will generate significant computational overhead.

[0092] However, due to limited network priorities and the lack of consistency in distributed architectures, scheduling degradation (i.e., degenerating into fair scheduling) may occur when multiple flows compete for shared resources due to priority conflicts. Therefore, in order to guarantee the Service Level Objective (SLO), a trade-off needs to be struck between global priority conflicts and computational overhead.

[0093] In this embodiment, distributed scheduling decision-making is employed to avoid the overhead of centralized control. This is achieved by collecting the job load at the sending end in real time and calculating U using the job transmission interval. i And adjust priorities. However, when deploying measurement functions at the sending end, the limitations of on-chip storage and computing resources must be strictly considered, while avoiding modifications to the application. Therefore, in this invention, the measurement function is integrated into the network interface card (NIC), and a hardware-friendly data structure is used to collect data and identify traffic patterns; in this embodiment, the Sketch data structure is adopted. Furthermore, to avoid degrading scheduling decisions, a small amount of state overhead is added to the switch's priority queue to identify competition between different jobs with the same priority. When identified flows are mapped to the same priority, newly arriving flows will be downgraded to a lower priority.

[0094] In this embodiment, when data packets destined for the same port have the same next hop, it is determined that there is flow contention on the current port. In this embodiment, the dynamic priority scheduling system deploys a RunID for each switch port to record the flow IDs sent to that port;

[0095] When a data packet arrives, the RunID is checked. If the RunID is empty, the flow ID and Runflag of the current data packet are recorded, and the current data packet is transmitted.

[0096] If RunID is not empty, compare the Runflag flag of the highest priority stream on the current port. If the new stream has a higher priority, update both values ​​and transmit the data packets of the new stream.

[0097] If the priority of a new flow is the same as the Runflag of the highest priority flow on the current port, the newly arrived data packet is attenuated to the next priority queue, and the switch port continues to execute the transmission of the previous flow.

[0098] In this embodiment, the worst-case scenario for attenuated packets is that they are placed with the lowest priority. Figure 5 illustrates the attenuation priority process. This embodiment addresses this issue using priority-based flow control (PFC) queue monitoring. PFC is a key network traffic management technology designed to build a lossless Ethernet environment for RDMA, especially RoCE protocols, ensuring zero packet loss and low latency data transmission. When the buffer occupancy rate of a low-priority queue on the switch exceeds a preset waterline, the PFC mechanism is triggered to suspend the upstream queue. In this embodiment, a PFC queue length monitoring mechanism is used, and priority (Runflag) and flow ID (RunID) are maintained using a hierarchical method. When a packet arrives, it first checks if the queue is empty, i.e., checks if the RunID is empty. If the queue is empty, the Runflag and RunID are updated, and the queue then focuses on transmitting packets from the current flow until the queue becomes empty again. If the queue is not empty, the values ​​of the current queue flow's Runflag and RunID are checked. When a new flow arrives on this link, the switch checks its priority against its priority queue. If the queue is empty, the new flow will occupy it. If the queue is already occupied by another flow, the new flow's priority will be downgraded. If the downgraded queue is still occupied, the downgrade will continue until the lowest priority is reached. Therefore, if the previous flow has completed its transmission, subsequent flows will not be affected because the new flow will update its Runflag and RunID and preempt the queue. If the previous flow has not yet completed its transmission, subsequent flows will detect a non-empty queue and a mismatched RunID, thus prioritizing the transmission of higher-priority packets.

[0099] like Figure 6As shown, if two flows communicate with G3 through two paths: 1) G1 → S1 → S2 → S4 → G3; 2) G2 → S1 → S3 → S4 → G3. Flow 1) and Flow 2) are contention on the link between S4 and G3, and this link is simultaneously triggered by Runfalg and RunID updates as well as priority decay.

[0100] In this embodiment, a flow scheduling method based on SLO is also provided, including the following steps;

[0101] 1) A Sketch data structure deployed on the network interface card and a dynamic priority scheduling system deployed on the switch. The dynamic priority scheduling system deploys a RunID for each switch port to record the flow ID sent to that port. The Sketch data structure includes Size_sketch, Time_sketch, and Count_sketch. Size_sketch adopts a probabilistic data structure CMS, which includes k arrays of length m, each array being associated with a different hash function.

[0102] 2) When a data packet arrives at the network card, the Sketch data structure obtains the size of the data packet and the iteration boundary; the dynamic priority scheduling system determines the priority of the next iteration of the current processing flow.

[0103] 3) Within the same iteration cycle; when a data packet arrives, the dynamic priority scheduling system checks the RunID. If the RunID is empty, it records the flow ID and priority of the current data packet and transmits the current data packet.

[0104] If RunID is not empty, compare the Runflag flag of the stream with the highest priority on the current port. If the new stream has a higher priority, update both values ​​and transmit the data packets of the new stream.

[0105] If the priority of a new flow is the same as the Runflag of the highest priority flow on the current port, the newly arrived data packet is attenuated to the next priority queue, and the switch port continues to execute the transmission of the previous flow.

[0106] To test the optimization effect of the SLO-driven flow scheduling and method in this embodiment, the SLO-driven flow scheduling and method in this embodiment are evaluated.

[0107] This embodiment implements an SLO-driven flow scheduling and method evaluation based on the HPCC (Yuliang Li, Rui Miao, et al. Hpcc: High precision congestion control. In Proceedings of the ACM special interest group on data communication, pages 44–58. 2019.) open-source NS3 project. Without modifying the congestion control prototype, application-independent traffic monitoring and dynamic priority adjustment functions are implemented in the hardware packet sending logic, and a priority decay mechanism is implemented in the switching module. The link rate is set to 100 Gbps, and the latency is set to 1 µs. This invention verifies the effectiveness of LongLiu by constructing a bottleneck link in a dumbbell topology, and then verifies the decision uniformity and effectiveness of the scheme in a large cluster in the topology shown in Figure 6. This invention randomly generates DNN training loads to simulate complex scenarios. Each training load contains multiple "communication computation" iterations. The simulation runs on a Dell R720 server running Ubuntu 18.04 and GCC version 5.3.1.

[0108] Scheme Comparison. Since shared DNN training clusters lack network SLO optimization, this invention uses the following schemes for comparison and evaluation:

[0109] Fair: This refers to the fair scheduling strategy, which serves as a benchmark for evaluating various options.

[0110] CRUX: CRUX (Jiamin Cao, Yu Guan, et al. Crux: GPU-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 1–15, 2024.) achieves efficient resource utilization through high-priority tasks with high GPU intensity. However, it does not optimize SLO due to the different scheduling objectives.

[0111] D2TCP: D2TCP (Balajee Vamanan, Jahangir Hasan, and TN Vijaykumar. Deadline-aware datacenter tcp (d2tcp). SIGCOMM Comput. Commun. Rev., 42(4):115–126, August 2012.) employs a similar paradigm in implementing the EDF algorithm in data centers, accelerating more urgent tasks by dynamically adjusting the congestion sensitivity of traffic. However, it lacks traffic characteristics suitable for DNN training. This invention calculates the deadline urgency factor based on the expected deadline of one iteration at the time of implementation.

[0112] MLTCP: MLTCP (Sudarsanan Rajasekaran, Sanjoli Narang, et al. Mltcp: A distributed technique to approximate centralized flow scheduling for machine learning. In Proceedings of the 23rd ACM Workshop on Hot Topics in Networks, pages 167–176, 2024.) accelerates inter-job communication by controlling the congestion window by monitoring the ratio of the number of bytes sent to the total number of bytes sent in each iteration.

[0113] CASSINI (Sudarsanan Rajasekaran, Manya Ghobadi, et al. Cassini: Network-aware job scheduling in machine learning clusters. In 21st USENIX Symposium on Networked Systems Design and Implementation (NSDI 24), pages 1403–1420, 2024.) avoids inter-job communication contention by injecting job time offsets, but degenerates into fair contention when jobs are incompatible. The experiments in this invention ignore job compatibility and only discuss general workloads.

[0114] Workload. This invention first quantitatively analyzes the SLO violation rate in cross-job resource contention scenarios. To establish a job classification benchmark, this invention proposes a Communication Iteration Rate (CIR) metric, defined as the proportion of communication time to total iteration time in the isolated training environment. Based on the CIR threshold, this invention constructs a three-part classification model: light communication jobs (CIR ∈ [20%, 40%]), balanced communication jobs (CIR ∈ [40%, 60%]), and heavy communication jobs (CIR ∈ [60%, 80%]). The position parameter µ is the arithmetic mean of each interval, and the discrete parameter σ is calculated by dividing the interval span by 6, i.e., σ = (up−low) / 6 (where up and low are the upper and lower limits of the interval, respectively). This setting ensures that 99.7% of the generated values ​​are within the original interval.

[0115] Network SLO guarantee under a single bottleneck link. This invention is based on... Figure 6 The network topology was used to construct the bottleneck link S1→S4, and 50 sets of contention traffic samples were generated according to three CIR types. Each set of experimental traffic was tested along two transmission paths, G1→S1→S4→G3 and G2→S1→S4→G4. For the L, B, and H models, the value range of ci was defined as: L[1.1,1.3], B[1.1,1.5], H[1.1,1.8], and random parameter sampling was performed based on a normal distribution probability model. The theoretical basis for designing the parameter range is that the heavier the communication load of a job, the higher the probability of SLO violation; therefore, it is necessary to expand the value range of its contention factor to improve scheduling flexibility.

[0116] from Figure 7The experimental results of the single-bottleneck link DNN training process shown can lead to the following conclusions: L. Load: Under low communication contention, Fair, D2TCP, and CASSINI have SLO violation rates of only 1%, indicating good basic scheduling capabilities. However, due to resource shortages from sporadic low-priority traffic, CRUX's violation rate rises to 6%. In contrast, MLTCP and LongLiu have a violation rate of 0%. B. Load: There are significant differences in rate adjustment capabilities among the schemes. The SLO violation rates of CRUX, D2TCP, MLTCP, CASSINI, and LongLiu decrease from the baseline of 30% (based on Fair) to 20%, 28%, 19%, 23%, and 9%, respectively, with LongLiu showing the largest decrease. H. Load: As the proportion of communication-intensive operations increases, increased contention leads to performance differentiation: the SLO violation reduction rates of MLTCP and D2TCP are similar to those of LongLiu (2.3%, 3%, and 3%, respectively), reflecting their stability under high-pressure environments. CRUX's violation rate decreased by 7%, but the violation rate at the tail (99th quartile) reached 1.74 times the baseline, revealing its lack of fairness. CASSINI only decreased slightly by 0.2%, highlighting its insufficient adaptability to high loads. Tail SLO violations: Under B / H loads, LongLiu's 99th quartile violations were 8% and 26.8%, respectively, better than D2TCP's 13.9% and 35.7%, CASSINI's 10.6% and 33.0%, and MLTCP's 8.6% and 29.0%. CRUX's tail violation rate reached 1.5 times (B load) and 1.74 times (H load) the baseline.

[0117] Network SLO Guarantee under Multiple Bottleneck Links. Consider a more complex scenario: a flow competes with different flows on two separate links. Observe the main stream in Figure 6: G1 → S1 → S3 → S4 → G3, whose transmission flows are S2 → S4 → S3 → G4 and G2 → S1 → S3 respectively (GPUs connected to S2 and S3 are omitted here). The main stream competes with different flows on S1 → S3 and S3 → S4 respectively. This invention randomly generates 30 sets of flows covering various load types to simulate complex scenarios, with each set of flows transmitted according to a preset path.

[0118] Figure 8 illustrates the SLO guarantee performance of a mixed-load DNN training stream under multi-bottleneck link contention. In the experiments, the SLO violation rates for the main stream (M) and all streams (A) were calculated separately. Due to fixed priority allocation, CRUX's SLO violation rate increased by 10% at node M, while the overall violation rate decreased to 3.3%. D2TCP reduced its violation rate by 3.4% and 3.3% at nodes M and A, respectively, while MLTCP's violation rate decreased significantly by 8.2% (node ​​M) and 11.1% (node ​​A). This indicates that under high-pressure contention, MLTCP is more effective at optimizing low-priority streams, but may introduce additional latency due to passive congestion control. CASSINI achieved 100% SLO compliance at node M, thanks to the job time offset strategy based on node M in the experiments, which maximizes resource reservation to guarantee high-priority streams. However, the SLO compliance rate at node A only decreased by 3.3%, reflecting its limitations in global load balancing. Compared to the M algorithm, the LongLiu algorithm reduced SLO violations by 13.4% (second only to CASSINI) and total violations by 10%. This result highlights the robustness of its dynamic priority scheduling and bandwidth allocation algorithm in multi-bottleneck scenarios and effectively coordinates the competition between M and A.

[0119] Figure 9 shows the impact of job information collection costs on short job SLO violation rates: an experimental evaluation based on additional time costs (0%-10%). The 0%, 1%, 3%, 5%, and 10% in the legend represent the percentage of additional time cost caused by job information collection.

[0120] JCT under oversubscription. Figure 10 The chart shows the average ratio of actual flow completion time to independent completion time for strict and lenient SLO flows under oversubscription load B, evaluated across 25 groups of competing flows. In each group, two competing flows were assigned a strict SLO (i.e., a completion time coefficient for each job satisfying 1 < ci < 1.1), indicating that each job expected to be scheduled in a near-exclusive manner. The flow with the lower ci value in each group (i.e., more inclined towards near-exclusive scheduling) was categorized as "strict," and the other as "lenient." It can be observed that under oversubscription conditions, all flows violated their SLOs due to overly strict SLO settings, suggesting that the provider should establish a more reasonable SLO setting mechanism. Furthermore, the completion time of the "strict" flow was relatively faster compared to the "lenient" flow, which aligns with the intended use of the ci configuration. Figure 1 To.

[0121] Memory overhead. As shown in Table 1, the error rate exceeding 95% in the statistics was set to less than 1%. Based on this setting, the CMS width is... = 272, depth is = 3, with each entry stored in 64-bit format. The total overhead is 19.125 KB. Additionally, LongLiu maintains two extra runtime states for each subsequent row, stored in 64-bit and 3-bit formats respectively. For a switch with 256 ports, the overhead for storing these extra states is 2.094 KB.

[0122] Table 1: Theoretical memory overhead for network cards and switches.

[0123]

[0124] Scheduling overhead. LongLiu demonstrated the necessity of this mechanism in optimizing JCT by comparing experiments with the control group and experiments with the priority decay function disabled. Experimental design: This study constructed multiple groups of Class B training loads, with each group of jobs competing for the bottleneck link (S1 → S4). To maximize the utilization of idle time and enhance the competition intensity between jobs, the amount of data in the job load was adjusted to make its distribution more uniform. The experiment was conducted in two groups: with the decay mechanism enabled (labeled as original) and with the decay function disabled (labeled as no decay). The experimental results are shown in Figure 11, with approximately 75.0% of the jobs showing a significant improvement in JCT. Among them, more than 10% of the jobs had a JCT latency exceeding 11%, with the maximum latency reaching 22.53%.

[0125] The SLO-driven flow scheduling system of this invention aims to improve system performance while ensuring user service level objectives (SLOs) and reducing the overhead of prior knowledge collection, thereby achieving user-friendly and efficient communication scheduling. The SLO-driven flow scheduling system and method of this invention include two core innovations: ① a network awareness mechanism, employing hardware-friendly data structures to monitor network operating characteristics and real-time network conditions; ② a dynamic priority reconstruction mechanism and a priority decay mechanism, combined with network SLOs that satisfy tasks, to alleviate DNN training latency caused by communication bottlenecks. Evaluation on the NS3 simulation platform showed that the SLO-driven flow scheduling system and method of this invention achieved significant performance improvements. Compared with CRUX (Jiamin Cao, Yu Guan, et al. Crux: GPU-efficient communication scheduling for deep learning training. In Proceedings of the ACM SIGCOMM 2024 Conference, pages 1–15, 2024.), under different loads, the SLO violation rate was reduced by 6%–11%, and the violation severity was reduced by 9%.

Claims

1. A flow scheduling method based on SLO (Solution-Order-Locked) driver, characterized in that: Includes the following steps; 1) A Sketch data structure deployed on the network interface card (NIC) and a dynamic priority scheduling system deployed on the switch. The dynamic priority scheduling system assigns a RunID to each switch port to record the flow ID sent to that port. The Sketch data structure includes Size_sketch, Time_sketch, and Count_sketch. Size_sketch uses a probabilistic data structure (CMS), which includes k arrays of length m, each array being associated with a different hash function. 2) When a data packet arrives at the network card, the Sketch data structure obtains the size of the data packet and the iteration boundary; the dynamic priority scheduling system determines the priority of the next iteration of the current processing flow; 3) Within the same iteration cycle; When a data packet arrives, the dynamic priority scheduling system checks the RunID. If the RunID is empty, it records the flow ID and Runflag of the current data packet and transmits the current data packet. If RunID is not empty, compare the Runflag flag of the highest priority stream on the current port. If the new stream has a higher priority, update both values ​​and transmit the data packets of the new stream. If the priority of a new flow is the same as the Runflag of the highest priority flow on the current port, the newly arrived data packet is attenuated to the next priority queue, and the port of the switch continues to execute the transmission of the previous flow. The Sketch data structure includes Size_sketch, Time_sketch, and Count_sketch. Size_sketch uses a probability data structure CMS, which includes k arrays of length m. Each array is associated with a different hash function. When a data packet arrives, it is hashed by the k hash functions and mapped to a position in the corresponding array. The counter at that position is incremented to complete the counting. Time_sketch captures the timestamps of arriving packets to infer iteration boundaries; When Count_sketch identifies an iteration replacement of a stream, it assigns the total amount of data in size_sketch for one iteration to count_sketch and clears size_sketch to zero.

2. The SLO-driven flow scheduling method according to claim 1, characterized in that: The dynamic priority scheduling system maps the calculation results of priority scheduling to the priority field of all data packet headers in the next iteration based on the urgency level Ui of job i. The ; y′ represents the number of iterations that the task has completed so far; Ai(0, y′) represents the actual completion time of iterations 0-y′; Ei(0, y′) represents the expected completion time of iterations 0-y′; When Ui > 1, it indicates that an SLO violation has occurred. The larger the Ui value, the more severe the timeout, and the scheduling system should assign a higher priority to reduce the risk. Conversely, when Ui ≤ 1, the task is within the SLO allowable range, and the priority can be reduced to optimize resource utilization.

3. A system for a flow scheduling method based on SLO driving according to claim 1 or claim 2, characterized in that: This includes a Sketch data structure deployed on the network interface card (NIC) and a dynamic priority scheduling system deployed on the switch; the Sketch data structure monitors job communication patterns, and the dynamic priority scheduling system adjusts the priority of data packets within the same iteration number of the current job based on the communication patterns of the current job monitored by the Sketch data structure on the NIC.

4. The SLO-driven flow scheduling system according to claim 3, characterized in that: The dynamic priority scheduling system deploys a RunID for each switch port to record the flow ID sent to that port; When a data packet arrives, the RunID is checked. If the RunID is empty, the flow ID and priority of the current data packet are recorded, and the current data packet is transmitted. If RunID is not empty, compare the Runflag flag of the highest priority stream on the current port. If the new stream has a higher priority, update both values ​​and transmit the data packets of the new stream. If the priority of a new flow is the same as the Runflag of the highest priority flow on the current port, the newly arrived data packet is attenuated to the next priority queue, and the switch port continues to execute the transmission of the previous flow.

Images