Tail delay optimization job scheduling method and system based on heterogeneous GPU cluster

Abstract

The invention relates to the technical field of computers, and discloses a tail delay optimization job scheduling method and system based on a heterogeneous GPU cluster. The method comprises the following steps: constructing a cluster physical topological graph containing link delay, bandwidth and hop count; predicting a job communication demand graph through static code analysis and a graph neural network; executing topology matching scheduling, and preferentially deploying a high communication task pair on a high-speed interconnection link in a node or a low-delay link on the same rack; and tail delay is monitored during operation, and cost-aware local rescheduling is triggered. The system comprises a physical topology modeling module, a communication demand prediction module, an affinity scheduling module and a dynamic rescheduling module. Through active prevention and closed-loop optimization, job tail delay is reduced, and heterogeneous cluster service quality and resource utilization efficiency are improved.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a tail-latency optimized job scheduling method and system based on heterogeneous GPU clusters. Background Technology

[0002] With the rapid development of artificial intelligence, high-performance computing, and large-scale distributed training tasks, heterogeneous GPU clusters have become a core infrastructure supporting complex computing workloads. These clusters typically consist of GPU nodes of various models with different computing power and memory configurations, communicating with each other via high-speed interconnect networks (such as InfiniBand or RoCE). In actual operation, job scheduling systems must comprehensively consider computing resource allocation, data locality, and communication overhead to ensure overall throughput and quality of service.

[0003] Tail latency is a key indicator for measuring system response consistency and user experience. Especially in latency-sensitive scenarios such as online inference and real-time recommendation, its stability directly determines the availability and business value of the service.

[0004] Existing job scheduling methods are mostly based on static resource views for task placement. While this can optimize computational efficiency within a single node, it generally lacks dynamic awareness of the underlying physical network topology. When the cluster is under heavy load, a large number of concurrent jobs trigger cross-node data transmission, and the scheduler often fails to plan communication paths based on link bandwidth, switch level, or node proximity, leading to unnecessary remote jumps and congestion hotspots.

[0005] This topological blind spot forces critical communication flows to traverse high-latency, low-bandwidth shared links, causing communication latency spikes. Consequently, the completion time of some task instances deviates from the average, and tail latency frequently exceeds the Service Level Agreement (SLA) threshold.

[0006] More seriously, traditional scheduling architectures typically require all cross-node communication to be coordinated through a centralized scheduler, which not only introduces additional control overhead but also creates performance bottlenecks, further exacerbating latency jitter.

[0007] Therefore, there is an urgent need for a dynamic scheduling mechanism that can deeply integrate network topology status and job communication mode to fundamentally avoid invalid cross-domain transmission, suppress communication congestion, and thus stabilize tail delay performance. Summary of the Invention

[0008] The technical problem this invention aims to solve is that existing job scheduling methods for heterogeneous graphics processor clusters do not fully consider the physical network topology between computing nodes. This results in critical data transmission paths crossing high-latency network links when scheduling communication-intensive jobs, causing sudden communication congestion and increasing the tail latency of job execution, which exceeds the service quality requirement threshold.

[0009] To address the aforementioned technical problems, this invention provides a tail-latency optimized job scheduling method and system based on heterogeneous GPU clusters. The method and system first proactively discover and construct a cluster physical network topology map containing weighted information such as link bandwidth, basic latency, and physical hop count through network probing protocols and management interfaces. In parallel, by performing static code analysis and historical execution data mining on submitted jobs, a graph neural network model is used to predict the communication traffic and patterns between tasks within the job, generating a job communication requirement graph.

[0010] The core of scheduling lies in the topology matching and affinity calculation process, which maps the job communication demand graph to the physical network topology graph. With the goal of minimizing the predicted weighted total communication cost, the process prioritizes placing the most communication-intensive task pairs together on the graphics processor connected by the node interconnect bus with nanosecond latency, and secondly on the nodes connected by the microsecond latency switch in the same rack, thereby generating the optimal initial task layout scheme.

[0011] During job execution, the system continuously monitors the communication latency of the actual network links. Once it detects that the tail latency percentile value of a specific path continuously exceeds the dynamic baseline, it triggers a cost-aware local rescheduling mechanism, which only migrates the single or a small number of tasks causing the bottleneck to alternative low-load, low-latency links, thereby achieving rapid response to network congestion and closed-loop optimization of job execution performance.

[0012] This invention provides a tail-latency optimized job scheduling method based on heterogeneous GPU clusters, which includes the following steps: The steps for constructing a comprehensive physical topology map of the cluster specifically include: By sending Link Layer Discovery Protocol probe messages to all network switching devices in the cluster, and combining them with Simple Network Management Protocol queries, the physical connection relationships, link speeds, and device model information between switch ports can be obtained. On each computing node, query and record the hardware identifier, bus connection type, driver version, and correspondence between each network interface card and the physical switch port. Inside a node equipped with multiple graphics processors, the direct links between the graphics processors formed by a high-speed interconnect bus are detected. All graphics processors, compute node central processing units, and network switches are abstracted as vertices in a graph, and the detected physical links are abstracted as edges connecting the vertices. Each edge is assigned a multi-dimensional weight vector consisting of basic physical latency, maximum theoretical bandwidth, current load rate, and physical hop count, thereby forming a weighted directed graph that accurately reflects the physical interconnection relationship of the entire cluster.

[0013] The steps of parsing and generating a job communication requirement profile specifically include: The system receives distributed computing jobs submitted by users, scans the source code or intermediate representation of the jobs using static code analysis tools, identifies function calls of inter-process communication libraries, such as sending, receiving, broadcasting, and aggregation operations of message passing interfaces, and thus constructs a preliminary inter-task communication dependency graph. Query the historical job execution database and extract historical execution records that match the current job type, code structure, or user tag; Using historical data such as actual communication traffic matrix, number of tasks, and graphics processor model as input features, a pre-trained graph neural network model is used to process the initial communication dependency graph and output a refined job communication demand graph. The weights of each edge in the graph represent the total predicted data exchange volume between the task pairs per unit time and the burst index of communication traffic.

[0014] Performing topology matching and resource affinity scheduling, the steps specifically include: Receive the cluster physical topology panoramic map and the job communication requirement profile; formalize the scheduling problem into a maximum weight matching problem of a weighted bipartite graph, where one part is all the tasks to be scheduled in the job and the other part is all the available graphics processor resources in the cluster; Define an affinity cost function that calculates the expected communication cost of mapping any communication edge in the job communication demand graph to a physical path in the physical topology graph. This cost is the product of the sum of the basic delays of all links on the physical path and the predicted amount of communication data, plus a penalty term related to the inverse of the maximum bandwidth of the path and the current congestion status. A heuristic greedy algorithm is used to solve the problem. First, the edge with the highest weight in the task communication demand graph is identified, which is the task pair with the most dense communication. Then, a physical path with the lowest communication cost and the graphics processor resources at both ends meet the task computing requirements is found in the physical topology graph. The task pair is then locked to the graphics processors at both ends of the path. Based on this, the task with the second highest communication requirement that is connected to the already deployed task is iteratively selected, and the location of the graphics processor in the physical topology that minimizes the total cost of communication with all deployed neighboring nodes is found for it, until all tasks are deployed, and finally the determined task-to-graphics processor allocation scheme is output.

[0015] Implement communication delay monitoring and adaptive rescheduling. The specific steps include: during job execution, collecting data packet information flowing through each physical link in real time through a flow monitoring protocol such as sFlow or NetFlow deployed on the network switch, and aggregating it to the central monitoring server. For each active communication stream generated by a scheduled job, the monitoring server calculates the 99th percentile of its end-to-end packet transmission delay, i.e., the tail delay, within a continuous time window. The calculated real-time tail delay is compared with a dynamic performance baseline calculated based on the link type and historical concurrent load. When it is detected that the real-time tail delay of a link is greater than a preset multiple of its dynamic performance baseline for multiple consecutive time windows, the link is determined to be congested and rescheduling logic is triggered. The rescheduling logic first identifies the computing task that generates the most communication traffic on the congested link, and then calls the topology matching and resource affinity scheduling module to find a new alternative graphics processor location for the task that can reduce its total communication cost. At the same time, it calculates the overhead required for task migration, including the time for checkpoint establishment, memory data transfer and task restart. A migration operation is performed only when the expected performance gain, i.e., the execution time saved by reducing latency, is greater than the migration overhead, and the state of the task is transferred to a new graphics processor to continue execution.

[0016] This invention also provides a tail-latency optimized job scheduling system based on heterogeneous GPU clusters, comprising: The physical topology awareness and modeling module is used to automatically detect the physical connections between switches, nodes and graphics processors within the cluster through link layer discovery protocol and simple network management protocol, and to construct a cluster physical network topology map with graphics processors as vertices and physical links as weighted edges. The weights of the edges include basic latency, bandwidth and hop count information.

[0017] The job communication demand prediction module is used to process jobs to be scheduled through static code analysis and graph neural network models, accurately predict the amount of communication data and communication patterns between the various computing tasks within the job, and generate a quantified job communication demand graph.

[0018] The topology matching and affinity calculation module has the core function of receiving the physical network topology map and job communication requirement map. By executing a heuristic matching algorithm aimed at minimizing global communication costs, it accurately maps each task in the job to the physical graphics processor. The algorithm prioritizes the use of high-speed interconnect buses within nodes and co-rack networks to generate the optimal initial task layout.

[0019] The real-time communication monitoring and dynamic rescheduling module is used to continuously monitor the tail latency indicators of all critical communication links during operation through network telemetry technology. When a persistent latency anomaly is detected, the bottleneck task is automatically identified and a cost-benefit analysis is performed. If the benefit is greater than the cost, the bottleneck task is relocated online and rescheduled to a graphics processor with a better network environment, thereby achieving closed-loop correction of scheduling decisions.

[0020] Furthermore, the system also includes a heterogeneous resource performance evaluation module. When the cluster is initialized and a new node is added, this module automatically performs benchmark tests on the computing power, memory bandwidth, and actual point-to-point communication performance of each graphics processor and its connected network interfaces. The module quantifies these precise performance parameters and stores them in the resource database, which is used by the topology matching and affinity calculation module as a basis for evaluating whether the resources meet the task requirements when making scheduling decisions.

[0021] Furthermore, the graph neural network model used in the job communication demand prediction module has a network structure including three graph convolutional layers, each followed by a linear rectified activation function, which is used to extract and aggregate the communication features of neighboring tasks layer by layer from the input job features, and finally predict the communication weights between each pair of tasks through regression of the fully connected output layer.

[0022] Furthermore, the affinity cost function used by the topology matching and affinity calculation module when calculating communication costs is defined as: ; in and For communication task pairs, To predict traffic, This represents the total physical path delay. Minimum path bandwidth This represents the current congestion index of the path. , , These are configurable weighting coefficients used to adjust the priority of latency, bandwidth, and congestion in scheduling decisions.

[0023] Furthermore, the cost-benefit analysis model used by the real-time communication monitoring and dynamic rescheduling module to determine whether to perform a migration is: if and only if When [the migration occurs], the migration is executed. For the measured tail delay, To predict tail delay, The remaining execution time of the task. The total cost of migration.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. It has achieved a shift from passive response to proactive prevention. By optimizing task layout based on a deep understanding of both physical topology and task communication requirements before the job is executed, it avoids deploying communication-intensive tasks on nodes that cross network bottlenecks from the root, thereby proactively avoiding the generation of communication latency spikes and effectively controlling tail latency.

[0025] 2. Scheduling decisions have physical-level precision. This invention refines the scheduling unit to a single graphics processor and a single physical link, and quantifies the communication costs of multi-level links, including intra-node high-speed interconnect buses, same-rack switching networks, and cross-rack backbone networks. This enables resource allocation to make the most of high-performance local interconnects and reduce contention for shared core network resources.

[0026] 3. A closed-loop adaptive optimization mechanism is introduced. By continuously monitoring the actual communication latency during job execution and combining it with a cost-aware dynamic rescheduling strategy, the system can cope with the inaccuracy of the prediction model or the dynamic changes in the network environment, ensuring that the scheduling scheme remains in a near-optimal state throughout the entire job lifecycle and guaranteeing the stability of service quality.

[0027] 4. By fully considering the heterogeneity of resources, the performance of different models of graphics processors and network devices is measured and quantified, so that the scheduling decision not only focuses on the network topology, but also takes into account the matching of computing and communication capabilities, ensuring that each task is allocated to the most suitable hardware resources for its operation, thereby maximizing the overall efficiency of the cluster. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall technical architecture of the tail latency optimization job scheduling method and system based on heterogeneous GPU clusters proposed in this invention. Figure 2 This is a schematic diagram of the core principle framework of the topology matching and affinity calculation module in this invention; Figure 3 This is a logical flowchart of the process for constructing a panoramic map of the cluster physical topology in this invention. Figure 4 This is a logical flowchart of the process of parsing and generating a profile of job communication requirements in this invention. Figure 5 This is a logical flowchart of the communication delay monitoring and adaptive rescheduling implemented in this invention; Figure 6 This is a schematic diagram of the multi-level interaction relationship and data flow between the physical topology perception and modeling module, the job communication demand prediction module, the topology matching and affinity calculation module, and the real-time communication monitoring and dynamic rescheduling module in this invention. Detailed Implementation

[0029] Please refer to Figures 1 to 6 This invention provides a tail latency optimization job scheduling method and system based on heterogeneous graphics processor clusters, aiming to solve the problem of tail latency spikes in communication-intensive jobs during execution caused by existing scheduling strategies ignoring the physical network topology. This method constructs an accurate panoramic map of the cluster's physical topology, generates a refined profile of job communication requirements, and performs topology matching and resource affinity scheduling based on this. Finally, it combines real-time communication monitoring and an adaptive rescheduling mechanism to achieve proactive control and closed-loop optimization of job tail latency.

[0030] The method includes the following steps: S1, construct a panoramic map of the cluster's physical topology; S2, parse and generate a profile of job communication requirements; S3 performs topology matching and resource affinity scheduling; S4 implements communication delay monitoring and adaptive rescheduling.

[0031] In step S1, the specific process of constructing the cluster physical topology panorama is as follows: First, the scheduling system periodically sends Link Layer Discovery Protocol (LLP) probe messages to all network switching devices in the cluster. These messages are used to obtain the direct physical connection relationships between switch ports.

[0032] Meanwhile, the scheduling system polls each switching device using the Simple Network Management Protocol (SMAP) to query its port speed, device model, firmware version, and current port status information.

[0033] This information together forms a connection view of the switching network hierarchy.

[0034] Secondly, on each computing node, the scheduling agent calls the hardware interface provided by the operating system to read the physical address, bus identifier, driver version number and the connected switch port number of each network interface card, thereby establishing a one-to-one correspondence between the node network interface and the switch port.

[0035] For computing nodes equipped with multiple graphics processors, the scheduling agent further calls the underlying diagnostic tools provided by the graphics processor manufacturers to detect whether there are direct links between the graphics processors consisting of high-speed interconnect buses such as NVLink or InfinityFabric, and records the theoretical bandwidth and measured latency of each direct link.

[0036] The scheduling system then abstracts each graphics processor, each central processing unit of each computing node, and each network switch in the cluster as a vertex in the graph.

[0037] All detected physical connections, including direct links between graphics processors, intra-node buses, Ethernet or InfiniBand links between nodes and switches, and cascaded links between switches, are abstracted as directed edges connecting the corresponding vertices.

[0038] Each edge is assigned a multidimensional weight vector, which contains four components: the basic physical delay, the value of which comes from actual measurement or equipment specifications; the maximum theoretical bandwidth, in gigabits per second. The current load factor is calculated as the ratio of the actual throughput to the theoretical bandwidth in the most recent sampling period. Physical hop count refers to the number of intermediate switching devices required to travel from the source vertex to the target vertex. Through the above process, the scheduling system constructs a weighted directed graph that accurately reflects the physical interconnection relationships of the entire cluster, i.e., a panoramic view of the cluster's physical topology.

[0039] In step S2, the specific process of parsing and generating a job communication requirement profile is as follows: When a user submits a distributed computing job to the scheduling system, the scheduling system first starts the static code analysis tool.

[0040] This tool performs a deep scan of the executable file or intermediate representation of the job to identify all function calls to inter-process communication libraries, including but not limited to point-to-point send and receive operations in the message passing interface standard, as well as collective communication operations such as broadcast, reduction, and full aggregation.

[0041] Based on the location, parameters, and context of these calls, the static analysis tool constructs a preliminary inter-task communication dependency graph, where vertices represent the various computational tasks in the job, and edges represent potential communication paths between tasks.

[0042] Next, the scheduling system accesses the historical job execution database, which stores metadata, code fingerprints, resource usage, and detailed performance profile data for all previously executed jobs.

[0043] The scheduling system retrieves a set of the most similar historical job records from the database based on the current job's type label, code hash value, or user identity information.

[0044] From these records, the scheduling system extracts key features, including the number of tasks, the combination of graphics processor models used, job runtime, and most importantly, the actual observed communication traffic matrix between tasks.

[0045] This matrix is ​​a two-dimensional array, and its element values ​​represent the total amount of data exchanged by the corresponding task pair per unit time.

[0046] These historical characteristics are related to the initial communication dependency. Figure 1They are all input into a pre-trained graph neural network model.

[0047] The model's structure consists of three graph convolutional layers, each responsible for aggregating the features of each task vertex and its first-order neighbors, and introducing non-linearity through a linear rectified activation function.

[0048] After three convolutional layers, the embedding vector of each vertex has incorporated the communication context information of its second-order and even third-order neighborhoods.

[0049] Finally, the fully connected output layer receives the concatenation result of the embedding vectors of all vertex pairs and regresses to predict the expected amount of communication data between each pair of tasks in this execution and the burst index of communication traffic.

[0050] The burst index is used to quantify whether a communication pattern is stable and continuous or bursty and pulse-like.

[0051] Finally, the scheduling system integrates the prediction results into a refined job communication requirements map, which serves as the core basis for subsequent scheduling decisions.

[0052] In step S3, the specific process of performing topology matching and resource affinity scheduling is as follows: The scheduling core receives the cluster physical topology panorama from step S1 and the job communication requirement profile from step S2.

[0053] The scheduling problem is formalized as a maximum weight matching problem in a weighted bipartite graph, where one part is the set of all unscheduled tasks in the job. The other part is the collection of all currently available graphics processor resources in the cluster. .

[0054] The goal of scheduling is to find the source from arrive The injective mapping minimizes the cost of global communication.

[0055] To address this, the scheduling system defines an affinity cost function to evaluate the allocation of any communication edge in the job communication demand graph. (Connection task) and Mapped to a physical path in the physical topology map The expected communication cost.

[0056] The mathematical expression for this cost function is: ; The task predicted in step S2 and The amount of communication data between them; For physical path The sum of the underlying physical delays of all links; For path The minimum bandwidth of all links; For path The maximum current load rate of all links is used to characterize the congestion situation; , , These are configurable weighting coefficients for system administrators, used to adjust the relative importance of latency, bandwidth, and congestion in scheduling decisions.

[0057] The scheduling system employs a heuristic greedy algorithm to solve this optimization problem.

[0058] The algorithm first identifies the weights in the job communication requirements graph. The largest edge represents the task pair with the most dense communication.

[0059] Then, in the physical topology map, the algorithm traverses all graphics processing units (GPUs) that meet the task's computational requirements (e.g., memory capacity, computing power). Calculation will task and Distributed to and At that time, the communication cost of the shortest physical path between them.

[0060] The algorithm selects the pair of graphics processors with the lowest cost and locks the two tasks onto them. Based on this, the algorithm enters the iterative phase: Among all tasks that have not yet been deployed but have communication edges with deployed tasks, select the task with the highest total communication requirement with deployed neighbors. .

[0061] For the task The algorithm iterates through all remaining available graphics processors. , calculation will Assigned to Then, the total communication cost increment between it and all deployed neighboring tasks.

[0062] The algorithm selects the one that minimizes the total cost increment. As The task allocation target is determined. This process is repeated until all tasks in the job are assigned to specific graphics processors, and finally, a complete task-to-graphics processor allocation scheme is output.

[0063] In step S4, the specific process of implementing communication delay monitoring and adaptive rescheduling is as follows: After the job begins execution according to the allocation scheme in step S3, the monitoring component of the scheduling system is immediately activated.

[0064] This component utilizes a flow monitoring protocol proxy pre-deployed on the network switch to continuously collect data packet sampling information flowing through each physical link.

[0065] This sampling information includes the timestamp of the data packets, source and destination addresses, protocol type, and number of bytes, and is aggregated in real time to the central monitoring server.

[0066] The monitoring server maintains an activity communication flow table, where each entry corresponds to a communication flow generated by a scheduled job, with both the source and destination being the graphics processor.

[0067] For each item in the table, the monitoring server collects all packet samples belonging to that flow within a sliding time window (e.g., 5 seconds) and calculates the 99th percentile value of its end-to-end transmission delay, i.e., the tail delay.

[0068] Meanwhile, the monitoring server maintains a dynamic performance baseline for each type of physical link (e.g., intra-node NVLink links, intra-rack 10 Gigabit Ethernet links, cross-rack 400G backbone links). This baseline is obtained by statistically smoothing the tail latency of this type of link under similar historical load conditions.

[0069] The monitoring server compares the real-time tail latency of each active stream with the dynamic baseline of its link type.

[0070] If the real-time tail delay of a flow is greater than a preset multiple (e.g., 1.5 times) of its dynamic baseline for three consecutive time windows, it is determined that the physical path where the flow is located has persistent congestion, and the rescheduling logic is triggered.

[0071] The rescheduling logic first analyzes all communication flows on the congested path, identifies the computational task that contributes the most traffic, and marks it as the bottleneck task.

[0072] The rescheduling logic then invokes the light quantum set of the topology matching and affinity calculation module in step S3, searching for new, available GPU locations in the physical topology graph specifically for this bottleneck task. The goal of the search is to find new locations such that, after the task is migrated, the predicted total communication cost with all its communicating neighbors is lower than the current cost.

[0073] After determining the candidate new positions, the rescheduling logic performs a cost-benefit analysis. The judgment condition for this analysis model is: If and only if Only then was the relocation approved.

[0074] The measured tail delay of the path where the task was located before the migration; The predicted tail delay is calculated based on the physical topology and the new location; The remaining execution time of the job is estimated based on historical execution data. The total overhead required to perform the task migration includes the total time required to create checkpoints on the original graphics processor, transfer task state data over the network to the new graphics processor, and restore the execution environment on the new graphics processor.

[0075] The scheduling system will only send instructions to the relevant computing nodes to perform task migration operations when the expected execution time savings are greater than the migration overhead, thereby completing the closed-loop correction of the scheduling decision.

[0076] The tail latency optimization job scheduling system based on heterogeneous graphics processor clusters includes a physical topology perception and modeling module, a job communication demand prediction module, a topology matching and affinity calculation module, a real-time communication monitoring and dynamic rescheduling module, and a heterogeneous resource performance evaluation module.

[0077] The physical topology awareness and modeling module is responsible for performing all the functions of step S1 above. This module includes a network probing engine, which periodically sends link-layer discovery protocol messages to switching devices and performs simple network management protocol queries; a node agent, deployed on each computing node, for collecting local hardware connectivity information; and a graph builder, for converting raw probing data into a standardized weighted directed graph data structure.

[0078] The job communication demand prediction module is responsible for performing all the functions of step S2 above. This module includes a static analyzer for parsing job code and building a preliminary communication graph; a historical data query interface for accessing the job execution database; and a graph neural network inference engine for loading pre-trained models, processing input features, and outputting refined communication demand predictions.

[0079] The topology matching and affinity calculation module is responsible for performing all the functions of step S3 above. The core of this module is the scheduling solver, which implements the aforementioned heuristic greedy algorithm and incorporates the calculation logic for the affinity cost function. This module also maintains a resource filter to exclude incompatible resources based on the task's computational requirements (such as required memory size and supported instruction set architecture) when searching for available graphics processors.

[0080] The real-time communication monitoring and dynamic rescheduling module is responsible for performing all the functions of step S4 above. This module includes a telemetry data collector for aggregating flow monitoring data from the switch; a tail delay calculator for performing statistical analysis on each active flow; a congestion detector for performing comparison logic with the dynamic baseline; and a migration decision-maker for performing cost-benefit analysis and triggering migration operations.

[0081] The heterogeneous resource performance evaluation module is activated during cluster initialization or when a new node is added. This module automatically executes a benchmark suite on each graphics processor, measuring its floating-point performance, memory bandwidth, and actual throughput and latency for point-to-point communication with other nodes through different network interfaces. These precise performance parameters are quantified and stored in the resource database, serving as a basis for the topology matching and affinity calculation module to assess whether resources meet task requirements, ensuring that scheduling decisions consider not only network topology but also the matching of computational and communication capabilities.

[0082] Through the collaborative work of the above methods and systems, this invention achieves refined, proactive, and adaptive optimization of job tail latency in heterogeneous graphics processor clusters, effectively solving the communication latency spike problem caused by cross-node data transmission not adapting to the physical topology.

[0083] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0084] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A tail-latency optimized job scheduling method based on heterogeneous GPU clusters, characterized in that, include: Construct a comprehensive physical topology map of the cluster; Analyze and generate a profile of job communication requirements; Perform topology matching and resource affinity scheduling; Implement communication delay monitoring and adaptive rescheduling; Constructing a comprehensive physical topology map of the cluster specifically includes: By sending Link Layer Discovery Protocol probe messages to all network switching devices in the cluster, and combining them with Simple Network Management Protocol queries, the physical connection relationships, link speeds, and device model information between switch ports can be obtained. On each computing node, query and record the hardware identifier, bus connection type, driver version, and correspondence between each network interface card and the physical switch port. Inside a node equipped with multiple graphics processors, the direct links between the graphics processors formed by a high-speed interconnect bus are detected. All graphics processors, compute node central processing units, and network switches are abstracted as vertices in a graph, and the detected physical links are abstracted as edges connecting the vertices. Each edge is assigned a multi-dimensional weight vector consisting of basic physical latency, maximum theoretical bandwidth, current load rate, and physical hop count, thereby forming a weighted directed graph that accurately reflects the physical interconnection relationship of the entire cluster.

2. The tail-latency optimized job scheduling method based on heterogeneous GPU clusters according to claim 1, characterized in that, The specific steps for parsing and generating a job communication requirement profile include: The system receives distributed computing jobs submitted by users, scans the source code or intermediate representation of the jobs using static code analysis tools, identifies function calls of inter-process communication libraries, and thus constructs a preliminary inter-task communication dependency graph. Query the historical job execution database and extract historical execution records that match the current job type, code structure, or user tag; Using historical data such as actual communication traffic matrix, number of tasks, and graphics processor model as input features, a pre-trained graph neural network model is used to process the initial communication dependency graph and output a refined job communication demand graph. The weights of each edge in the graph represent the total predicted data exchange volume between the task pairs per unit time and the burst index of communication traffic.

3. The tail-latency optimized job scheduling method based on heterogeneous GPU clusters according to claim 1, characterized in that, Performing topology matching and resource affinity scheduling specifically includes: Receive the cluster physical topology panoramic map and the job communication requirement profile; The scheduling problem is formalized as a maximum weight matching problem in a weighted bipartite graph, where one part consists of all the tasks to be scheduled in the job and the other part consists of all available graphics processor resources in the cluster. Define an affinity cost function that calculates the expected communication cost of mapping any communication edge in the job communication demand graph to a physical path in the physical topology graph. This cost is the product of the sum of the basic delays of all links on the physical path and the predicted amount of communication data, plus a penalty term related to the inverse of the maximum bandwidth of the path and the current congestion status. A heuristic greedy algorithm is used to solve the problem. First, the edge with the highest weight in the task communication demand graph is identified, which is the task pair with the most dense communication. Then, a physical path with the lowest communication cost and the graphics processor resources at both ends meet the task computing requirements is found in the physical topology graph. The task pair is then locked to the graphics processors at both ends of the path. Based on this, the task with the second highest communication requirement that is connected to the already deployed task is iteratively selected, and the location of the graphics processor in the physical topology that minimizes the total cost of communication with all deployed neighboring nodes is found for it, until all tasks are deployed, and finally the determined task-to-graphics processor allocation scheme is output.

4. The tail-latency optimized job scheduling method based on heterogeneous GPU clusters according to claim 1, characterized in that, The specific steps for implementing communication delay monitoring and adaptive rescheduling include: During the operation, the data packet information flowing through each physical link is collected in real time by the flow monitoring protocol deployed on the network switch and aggregated to the central monitoring server. For each active communication stream generated by a scheduled job, the monitoring server calculates the 99th percentile of its end-to-end packet transmission delay, i.e., the tail delay, within a continuous time window. The calculated real-time tail latency is compared with a dynamic performance baseline calculated based on the link type and historical concurrent load. When the real-time tail delay of a link is detected to be greater than a preset multiple of its dynamic performance baseline for multiple consecutive time windows, the link is determined to be congested and rescheduling logic is triggered. The rescheduling logic first identifies the computing task that generates the most communication traffic on the congested link, and then calls the topology matching and resource affinity scheduling module to find a new alternative graphics processor location for the task that can reduce its total communication cost. At the same time, it calculates the overhead required for task migration, including the time for checkpoint establishment, memory data transfer and task restart. A migration operation is performed only when the expected performance gain, i.e., the execution time saved by reducing latency, is greater than the migration overhead, and the state of the task is transferred to a new graphics processor to continue execution.

5. The tail-latency optimized job scheduling method based on heterogeneous GPU clusters according to claim 1, characterized in that, Construct a comprehensive physical topology map of the cluster, including: Each graphics processor, each central processing unit of each computing node, and each network switch in the cluster is abstracted as a vertex in the graph; The direct links between graphics processors, the internal bus of a node, the Ethernet or InfiniBand links between a node and a switch, and the cascaded links between switches are all abstracted as directed edges connecting the corresponding vertices. Each edge is assigned a multi-dimensional weight vector, which contains four components: basic physical latency, maximum theoretical bandwidth, current load rate, and physical hop count.

6. The tail-latency optimized job scheduling method based on heterogeneous GPU clusters according to claim 2, characterized in that, Parse and generate a profile of job communication requirements, including: Static code analysis tools are used to identify function calls to the inter-process communication library in the job, and a preliminary inter-task communication dependency graph is constructed. Extract historical records similar to the current job from the historical job execution database to obtain the number of tasks, the combination of graphics processor models, and the actual communication traffic matrix between tasks; The initial communication dependency graph and historical features are input into a graph neural network model containing three layers of graph convolutional layers and a linear rectified activation function, and the predicted communication data volume and communication traffic burst index between each pair of tasks are output.

7. The tail-latency optimized job scheduling method based on heterogeneous GPU clusters according to claim 3, characterized in that, Perform topology matching and resource affinity scheduling, including: Define the affinity cost function as follows: ； in and For communication task pairs, To predict traffic, This represents the total physical path delay. Minimum path bandwidth This represents the current congestion index of the path. , , Configurable weighting coefficients; A heuristic greedy algorithm is used to prioritize locking the most communication-intensive task pairs to the graphics processor pairs with the lowest communication cost, and then iteratively expand to the remaining tasks.

8. The tail-latency optimized job scheduling method based on heterogeneous GPU clusters according to claim 4, characterized in that, Implement communication latency monitoring and adaptive rescheduling, including: The packet sampling information of each physical link is collected by the flow monitoring protocol, and the 99th percentile tail delay of each active communication flow within the sliding time window is calculated. Compare real-time tail latency with a dynamic performance baseline based on link type and historical load; When the tail delay of multiple consecutive time windows exceeds the baseline preset multiple, the rescheduling logic is triggered.

9. The tail-latency optimized job scheduling method based on heterogeneous GPU clusters according to claim 8, characterized in that, After the rescheduling logic is triggered, it includes: Identify the computational task that contributes the most traffic on the congested link as the bottleneck task; The light quantum set of the topology matching and resource affinity computation module is invoked to search for new graphics processor locations for this bottleneck task; The cost-benefit analysis is performed, and the judgment criteria are: ； in For the measured tail delay, To predict tail delay, The remaining execution time of the task. Total migration costs; The task migration operation is executed only if the condition is met.

10. A tail-latency optimized job scheduling system based on heterogeneous GPU clusters, characterized in that, Tail-latency optimized job scheduling is implemented using the tail-latency optimized job scheduling method based on heterogeneous GPU clusters as described in any one of claims 1 to 9. The system includes: The physical topology awareness and modeling module is used to automatically detect the physical connections between switches, nodes and graphics processors within the cluster through the link layer discovery protocol and the simple network management protocol, and to construct a cluster physical network topology graph with the graphics processor as the vertex and the physical links as weighted edges. The weights of the edges include basic latency, bandwidth and hop count information. The job communication demand prediction module is used to process jobs to be scheduled through static code analysis and graph neural network models, accurately predict the amount of communication data and communication patterns between the internal computing tasks, and generate a quantitative job communication demand graph. The topology matching and affinity calculation module has the core function of receiving the physical network topology map and job communication requirement map. By executing a heuristic matching algorithm aimed at minimizing global communication cost, it accurately maps each task in the job to the physical graphics processor. The algorithm prioritizes the use of high-speed interconnect buses within nodes and co-rack networks to generate the optimal initial task layout. The real-time communication monitoring and dynamic rescheduling module is used to continuously monitor the tail latency indicators of all critical communication links during operation through network telemetry technology. When a persistent latency anomaly is detected, the bottleneck task is automatically identified and a cost-benefit analysis is performed. If the benefit is greater than the cost, the bottleneck task is relocated online and rescheduled to a graphics processor with a better network environment, thereby achieving closed-loop correction of scheduling decisions.

Images