A topology-aware scheduling method for large model training

Abstract

This invention discloses a topology-aware scheduling method for large-scale model training. The method includes: collecting hardware and network information through probe agents deployed on each intelligent computing node to generate a real-time network state matrix; receiving the parallel strategy of the training task, adaptively determining the affinity score weight coefficients, and constructing a task-aware scheduling graph; performing multi-dimensional hierarchical constraint scheduling, constraining each parallel group within the corresponding hierarchical topology constraints based on the communication sensitivity of each parallel mode, with higher sensitivity constrained within the topology hierarchy with higher communication performance; and employing a multi-objective collaborative optimization algorithm that introduces a fragmentation penalty term to select the optimal node combination through comprehensive evaluation values. This invention effectively solves the communication bottleneck problem caused by the "blindness" of network topology in traditional schedulers, avoids computing power fragmentation while ensuring network affinity, and significantly improves GPU utilization and throughput efficiency for distributed training of large-scale models.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence infrastructure and cloud computing resource scheduling technology, and in particular to a topology-aware scheduling method for large model training. Background Technology

[0002] The core of distributed training of large language models relies on large-scale intelligent computing clusters composed of hundreds or thousands of GPUs. During training, collective communication operations such as All-Reduce are frequently executed to synchronize gradients, and training efficiency is highly dependent on the bandwidth, latency, and stability of the communication links. However, distributed training of large models usually mixes multiple parallel strategies such as data parallelism, tensor parallelism, and pipelined parallelism. Different parallel strategies have significantly different requirements for communication bandwidth and latency. Moreover, in the actual construction of intelligent computing centers, the clusters often mix GPUs and network cards of different generations, and the network connections between nodes are heterogeneous. When traditional container schedulers distribute training tasks, they only match based on scalar resources such as the remaining number of CPUs / GPUs, and are completely unaware of the underlying physical network topology. This easily leads to the dispersal of processes that require high-frequency communication on different racks across core switches, resulting in severe network congestion and packet loss. The synchronization speed of the entire training task is blocked by the slowest link, and expensive GPU computing power is idle for a long time, resulting in low computing power utilization.

[0003] The current mainstream scheduling schemes for distributed training of large models are based on the default scheduler or extended scheduler of container orchestration platforms such as Kubernetes. The principle is to distribute tasks based on the remaining scalar resources such as CPU, GPU, and memory of the nodes through filtering and scoring mechanisms. Some schemes introduce simple affinity / anti-affinity rules or node labels, but they have obvious drawbacks, as follows:

[0004] (1) The physical network topology between nodes is not considered: the existing scheduler lacks the ability to perceive network topology information such as switch level, link bandwidth, and latency, and cannot distinguish the performance difference between communication within the same rack and communication across backbone switches, resulting in "network blindness" in scheduling decisions;

[0005] (2) Unable to perceive the differentiated communication requirements of multidimensional parallel strategies for large models: Tensor parallelism is extremely sensitive to bandwidth and latency, requiring high-speed interconnection within nodes; pipeline parallelism is mainly point-to-point communication, requiring high connection stability; data parallelism is periodic high-throughput communication. Existing schedulers treat the communication requirements of these three parallel modes as equal, and cannot achieve differentiated and accurate mapping;

[0006] (3) Lack of perception and response to dynamic degradation of network status: During the long-term training process, the physical network of the intelligent computing center often degrades due to reasons such as loose optical fiber, overheating and degradation of optical module, and sudden congestion of switch. The existing scheduler schedules based on initial static information. Once the network degrades, the communication efficiency of the running tasks drops sharply but cannot be automatically avoided.

[0007] (4) When pursuing network affinity, computing power fragmentation is likely to occur: If the scheduling strategy is too strict in pursuing the optimal network topology, a large number of scattered GPUs in the cluster will appear that cannot be used by subsequent large tasks, which will aggravate computing power fragmentation, prolong the overall job queuing time, and lack a multi-objective collaborative optimization mechanism between affinity and fragmentation rate.

[0008] In summary, distributed training of large language models has rigid requirements for bandwidth, latency, and stability of communication links. However, existing scheduling schemes based on scalar resources cannot perceive physical network topology, cannot match the differentiated communication needs of multi-dimensional parallel strategies, cannot respond to network dynamic degradation, and cannot balance affinity and resource utilization. This results in low training efficiency and serious waste of GPU computing power, making it difficult to meet the needs of efficient training in large-scale intelligent computing clusters. There is an urgent need for a better scheduling scheme to solve these core pain points. Summary of the Invention

[0009] The purpose of this invention is to provide a topology-aware scheduling method for large model training, thereby solving all or one of the aforementioned problems in the prior art.

[0010] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0011] On the one hand, this invention provides a topology-aware scheduling method for training large models, comprising the following steps:

[0012] Responding to the graph construction phase of the scheduling system:

[0013] A probe agent is deployed on each intelligent computing node to perform intra-node topology probe and inter-node network probe, collect hardware and network information, and generate a real-time network state matrix based on the hardware and network information. The real-time network state matrix includes communication link parameters between node pairs.

[0014] The perceptual matching scheduling phase of the scheduling system:

[0015] The system receives training tasks and parallel strategies submitted by users, including tensor parallelism (TP), pipeline parallelism (PP), and data parallelism (DP).

[0016] Based on the parallel strategy, the weight coefficients of the affinity score formula are determined. The affinity score formula is used to convert the communication parameters of heterogeneous links into standardized scores that can be compared globally.

[0017] Based on the weighting coefficients and the real-time network state matrix, the affinity score between each node pair is calculated, and a task-aware scheduling graph is constructed.

[0018] The hierarchical constraint scheduling is performed based on the parallel strategy, wherein the hierarchical constraint scheduling includes constraining parallel groups with different communication sensitivities to a matching physical topology range;

[0019] A heuristic graph partitioning algorithm is used to extract candidate node clusters from the scheduling graph under the constraints corresponding to the hierarchical constraints. The candidate node clusters are then filtered based on the comprehensive evaluation value to output the optimal node combination to complete the scheduling. The comprehensive evaluation value is determined by the affinity score and the fragmentation cost.

[0020] As an improved approach, the step of collecting hardware and network information and generating a real-time network state matrix based on the hardware and network information further includes: converting the hardware and network information into a weighted undirected graph as a basic topology graph, wherein the weights of the edges are the original link parameters, including at least bandwidth, latency, and hop count information; and generating a real-time network state matrix containing bandwidth, latency, and hop count information between node pairs based on the original link parameters.

[0021] As an improvement, the construction of the task-aware scheduling graph includes replacing the weights of each edge in the basic topology graph with the affinity scores between corresponding node pairs.

[0022] As an improvement, determining the weight coefficients of the affinity score formula based on the parallel strategy further includes: the affinity score formula is: Score = α·Bandwidth - β·Latency - γ·Hop_Count, where Bandwidth is the link bandwidth, Latency is the link latency, Hop_Count is the number of hops across switches, and α, β, and γ are weight coefficients; the values ​​of each weight coefficient are determined based on the proportional relationship between the TP value, PP value, and DP value.

[0023] When TP / (PP+DP) is greater than the first preset threshold, the values ​​of β and γ are increased to prioritize the use of GPUs within the same physical rack and exclude cross-switch deployments.

[0024] When PP / (TP+DP) is greater than the second preset threshold, the value of α is increased and the value of γ is decreased to allow cross-switch deployment but prioritize ensuring uniform and stable bandwidth between nodes.

[0025] When DP / (TP+PP) is greater than the third preset threshold, α is set to the highest value and β is set to the lowest value, so as to prioritize finding the network card combination with the largest total bandwidth.

[0026] As an improved approach, the step of screening candidate node clusters through comprehensive evaluation values ​​further includes:

[0027] Calculate the comprehensive evaluation value of the candidate node cluster: Total_Score = Affinity_Score(S) - λ·Fragmentation_Cost(S);

[0028] Wherein, Affinity_Score(S) is the sum of affinity scores between node pairs within the selected node set S; Fragmentation_Cost(S) is the fragmentation cost, which is calculated as follows: first, the total number of GPUs required for the current task is determined according to the parallel strategy, N_total = TP×PP×DP; then, after selecting the node set S, the number of remaining idle GPUs within the jurisdiction of each physical rack and leaf switch that is less than N_total is counted, and the number is accumulated as the fragmentation cost; λ is a dynamic adjustment factor;

[0029] Determine whether the comprehensive evaluation value is greater than the preset comprehensive evaluation value threshold: if yes, accept the current candidate node cluster; if no, exclude the current candidate node cluster and search again.

[0030] As an improvement, λ is a dynamic adjustment factor, and its adjustment method is as follows:

[0031] When cluster resources are idle, λ takes the smaller value, with affinity being the absolute priority;

[0032] When cluster resources are scarce and there are many queued tasks, λ should be taken as a larger value to prioritize compact layout.

[0033] As an improvement, the following steps are also included:

[0034] In response to the dynamic obstacle avoidance phase of the scheduling system:

[0035] The detection agent continuously monitors the link status. When network degradation is detected, it lowers the bandwidth parameter of the corresponding link in the real-time network status matrix and / or raises its latency parameter.

[0036] Based on the updated real-time network state matrix, the affinity score between each node pair is recalculated, thereby reducing the affinity score between nodes on both sides of the degraded link. For queued tasks that are not ready, when calculating the comprehensive evaluation value for the candidate node cluster, the comprehensive evaluation value of the candidate node combination containing the degraded link is reduced, and it is automatically filtered during scheduling calculation, thus achieving static avoidance.

[0037] As an improvement, the dynamic obstacle avoidance phase also includes a dynamic evaluation and response mechanism for tasks already in operation, specifically including:

[0038] When the degradation causes a latency increase of no more than 10%, execute the first-level response: do not migrate the task, and send instructions to the collective communication library and / or network card of the training framework through the control plane interface to adjust the buffer size of the collective communication or trigger the network card's dynamic routing to avoid congested paths.

[0039] When degradation causes an overall throughput drop of more than 30%, a secondary response is executed: triggering thermal obstacle avoidance, obtaining checkpoint information of the training task, pausing training at the most recent save point, and resuming training after migrating the container on the degradation node to a healthy standby node.

[0040] When a high-speed interconnection failure occurs within a node, a three-level response is executed: the node is marked as unschedulable and an alarm is sent to the operation and maintenance system.

[0041] As an improvement, during the process of performing intra-node topology probing and inter-node network probing, the probing agent employs a non-intrusive probing mechanism, specifically including:

[0042] Asynchronous detection is performed using microsecond-level pulses, with each data transmission being less than a preset number of bytes.

[0043] By isolating probing packets by quality of service priority, the probing packets are assigned to a low-priority queue and the training traffic is assigned to a high-priority queue. When the network is congested, the switch is allowed to drop probing packets first. The probing agent infers the congestion level of the current network link by monitoring the drop rate of the probing packets.

[0044] The physical layer performance counters of the node's network card are read directly, and the network quality is evaluated by analyzing the packet loss rate and retransmission rate of the actual traffic.

[0045] As an improvement, the hierarchical constraint scheduling includes:

[0046] Prioritize constraining tensor parallel groups within the same node and utilize high-speed interconnect communication within the node;

[0047] Secondly, the parallel pipeline groups are constrained between different nodes under the same leaf switch.

[0048] Finally, the data parallel groups are distributed among nodes that can span the backbone switch.

[0049] The beneficial effects of the technical solution of this invention are:

[0050] This invention presents a topology-aware scheduling method for large-scale model training. By constructing a "computing power-network" topology map and affinity matrix, it accurately maps the multi-dimensional parallel communication requirements of distributed training of large models to the physical topology, effectively solving the problem of low computing power utilization caused by the "blindness" of traditional schedulers to network communication bottlenecks. Through the introduction of a multi-objective collaborative optimization scheduling algorithm with a fragmentation penalty term, a dynamic balance is achieved between maximizing network affinity and minimizing computing power fragmentation. A hierarchical dynamic obstacle avoidance mechanism enables real-time perception and hot migration recovery of network degradation without interrupting training. In multi-machine, multi-GPU hybrid training scenarios, the average number of tokens per second for large models can be increased by 15%-30%, significantly shortening the model training cycle. Simultaneously, it effectively alleviates core network congestion and incast packet loss problems in Spine switches of intelligent computing centers, overcoming the shortcomings of existing technologies and possessing high application value. Attached Figure Description

[0051] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0052] Figure 1 This is an overall system architecture diagram of an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the topology automated detection engine according to an embodiment of the present invention;

[0054] Figure 3 This is a topology graph matrix construction module diagram according to an embodiment of the present invention;

[0055] Figure 4 This is a topology affinity scheduler diagram according to an embodiment of the present invention;

[0056] Figure 5 This is a diagram of the cluster infrastructure according to an embodiment of the present invention;

[0057] Figure 6 This is a flowchart illustrating the affinity matrix generation process according to an embodiment of the present invention;

[0058] Figure 7 This is a flowchart of the multidimensional parallel strategy matching and scheduling process according to an embodiment of the present invention;

[0059] Figure 8 This is a flowchart of the dynamic obstacle avoidance mechanism according to an embodiment of the present invention. Detailed Implementation

[0060] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0061] In the description of this invention, it should be noted that the embodiments described in this invention are only some embodiments of this invention, not all embodiments; based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0062] The terms “first,” “second,” “third,” etc., used in this specification, claims, and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0063] In the description of this invention, it should be noted that:

[0064] Large Language Model (LLM) refers to a deep learning model based on the Transformer architecture that contains a massive number of parameters. Its training and inference require a large-scale GPU cluster.

[0065] Distributed training refers to a computational paradigm that splits the training task of a large language model across multiple GPU nodes for parallel execution.

[0066] Tensor Parallelism (TP) refers to the parallel computation of the parameter matrix of a single-layer neural network across multiple GPUs. Communication occurs in each forward and backward propagation step, and it has extremely high requirements for bandwidth and latency.

[0067] Pipeline parallelism (PP) refers to allocating different layers of a model to different GPUs and executing them serially in a pipeline manner, with communication mainly consisting of point-to-point transmission of activation values ​​between layers.

[0068] Data parallelism (DP) refers to distributing training data in batches to different GPUs, with each GPU holding a complete copy of the model and only communicating gradient synchronization at the end of each training step.

[0069] All-Reduce refers to a collective communication operation in distributed training, used to sum the gradients on all GPUs and broadcast them back to each GPU. It is a periodic high-throughput communication operation.

[0070] NVLink is a high-bandwidth GPU interconnect technology developed by NVIDIA for high-speed communication between GPUs within the same node.

[0071] A leaf switch is an access layer switch in a data center network architecture that directly connects to server nodes.

[0072] A backbone switch (Spine Switch) is a core layer switch in a data center network architecture that connects all leaf switches.

[0073] LLDP (Link Layer Discovery Protocol) is a link layer discovery protocol used for network devices to announce and discover neighbor information, which can be used to obtain the switch-level topology.

[0074] RDMA (Remote Direct Memory Access) refers to remote direct memory access technology, which allows one host to directly access the memory of another host, and has the characteristics of low latency and high bandwidth.

[0075] Checkpoint refers to a checkpoint file that periodically saves model parameters and optimizer state during training, and is used to resume training after a breakpoint is interrupted.

[0076] A Pod is the smallest deployable computing unit in Kubernetes, which can contain one or more containers.

[0077] NVLink failure refers to a failure in the physical NVLink connection between GPUs within the same node, resulting in a severe degrade or interruption of inter-GPU communication performance.

[0078] ECN (Explicit Congestion Notification) is an explicit congestion notification mechanism used to mark data packets instead of dropping them directly when network congestion occurs, thereby notifying the sender in advance to reduce the sending rate.

[0079] Priority-based flow control (PFC) is a flow control mechanism that allows for the suspension of traffic in specific priority queues during network congestion, while other priority traffic remains unaffected.

[0080] QoS (Quality of Service) refers to the quality of service used to prioritize different traffic in a network, ensuring bandwidth and latency guarantees for high-priority services.

[0081] Topology affinity is a standardized score used to measure the quality of communication between any two computing nodes in a cluster, taking into account factors such as bandwidth, latency, and hop count across switches.

[0082] Example 1: This example provides a topology-aware scheduling method for large model training, addressing the problem that traditional schedulers cannot perceive the physical network topology, leading to distributed training communication bottlenecks and low computational utilization. Figures 1-4 As shown, it includes:

[0083] S100, the steps for constructing the "computing power-network" topology map and affinity matrix include:

[0084] S101. Deploy a lightweight probe agent on each intelligent computing node. The probe agent is responsible for performing intra-node sensing and inter-node sensing. Intra-node sensing identifies the connection medium and bandwidth between GPUs by parsing the system hardware topology; inter-node sensing combines LLDP to obtain the switch-level topology and periodically sends micro-communication probe packets to measure the actual throughput and latency between nodes.

[0085] To prevent probe packets from interfering with service training, the probe mechanism adopts the following seamless design:

[0086] Timing control: Employs microsecond-level pulse detection, sending only a few hundred bytes of specific control flow each time, with an impact on service traffic lower than a preset ratio, achieving imperceptible detection on the time axis.

[0087] QoS priority isolation: Training communication traffic is divided into high-priority queues, and probe packets are divided into low-priority / best-effort queues. When network congestion occurs, the switch prioritizes forwarding training data, and may even drop probe packets; the agent infers the network congestion status by analyzing the probe packet drop situation.

[0088] Seamless sampling of business data: The agent directly reads the underlying performance counters of the network card and evaluates the network quality by analyzing the packet loss rate and retransmission rate of the actual business traffic. This method does not require sending additional probe packets.

[0089] S102. The collected hardware and network information is transformed into a weighted undirected graph G=(V, E), where V represents computing nodes such as GPUs or network cards, E represents communication links, and the weights of the edges are the original link parameters, including at least bandwidth, latency, and hop count information. This graph serves as the basic topology map.

[0090] S103. Define the affinity score formula, which is used to convert the bandwidth, latency, and hop count of heterogeneous networks into standardized scores that can be compared globally in the subsequent scheduling phase:

[0091] Score = α · Bandwidth - β · Latency - γ · Hop_Count

[0092] Where Bandwidth is the link bandwidth, Latency is the link latency, and Hop_Count is the number of hops across switches; α, β, and γ are weighting coefficients, the specific values ​​of which will be determined in the subsequent scheduling phase based on the task type.

[0093] S104. Based on the collected link bandwidth, latency, and hop count information, a real-time network state matrix is ​​generated for all nodes in the cluster to record the corresponding bandwidth, latency, and hop count parameters. This matrix serves as the basic data layer for the topology map. Figure 2 As shown.

[0094] S200, Execute the perceptual matching scheduling steps based on the large model parallel strategy, including:

[0095] S201. Receive the training task submitted by the user and its declared parallel strategy, including tensor parallelism (TP), pipeline parallelism (PP), and data parallelism (DP).

[0096] The scheduler is pre-configured with a control plane interface that communicates with the training framework and network card.

[0097] S202. Determine the affinity score weighting coefficients based on the task parallelization strategy:

[0098] The weighting coefficients are automatically calculated based on the proportional relationship between the values ​​of each parallelism.

[0099] When TP / (PP+DP) is greater than the first preset threshold, it is identified as a high tensor parallel task. The value of β is increased to a preset multiple of the baseline value, and the value of γ is increased to a preset multiple of the baseline value to avoid cross-switch deployment and prioritize ensuring that all cards are in the same physical rack.

[0100] When PP / (TP+DP) is greater than the second preset threshold, it is identified as a long pipeline parallel task. The value of α is increased to a preset multiple of the base value, and the value of γ is decreased to a preset proportion of the base value, so as to allow the task to be deployed across switches but require that the bandwidth between each node be uniform and stable.

[0101] When DP / (TP+PP) is greater than the third preset threshold, it is identified as a pure data parallel task. α is set to the highest value and β is set to the lowest value, and the scheduling objective is changed to finding the network interface card combination with the largest total bandwidth.

[0102] S203, based on the weight coefficients determined in S202 and the real-time network state matrix generated in S104, calculates the affinity score for each node in the cluster for the current task. On top of the basic topology graph, using the affinity scores as edge weights, a task-aware scheduling graph is constructed for subsequent cluster search.

[0103] S204. Perform multi-dimensional hierarchical constraint scheduling: Based on the sensitivity of each parallel mode to communication bandwidth and latency, constrain the tensor parallel group, pipeline parallel group and data parallel group to the corresponding hierarchical topology constraints, wherein the parallel group with higher sensitivity is constrained to the topology level with higher communication performance.

[0104] In this embodiment, the above-mentioned "hierarchical topology constraints" are specifically manifested as follows: tensor parallelism is most sensitive to communication bandwidth and latency, so it is constrained within the same node and interconnected at high speed using NVLink; pipeline parallelism is less sensitive to communication, so it is constrained between different nodes under the same leaf switch; data parallelism is relatively least sensitive to communication, so it is allowed to be deployed between nodes across backbone switches.

[0105] It is understood that in other embodiments, the specific mapping relationship of hierarchical topology constraints can be adjusted according to the actual cluster topology and the sensitivity ranking of the parallel strategy. For example, when the bandwidth of the leaf switch layer is sufficient, the tensor parallel group can also be constrained between different nodes under the same leaf switch, as long as the hierarchical correspondence principle of "the higher the sensitivity, the tighter the topology constraint" is met.

[0106] S205. When performing specific node selection, a heuristic graph partitioning algorithm based on multi-objective collaborative optimization is adopted:

[0107] S2051. Extract all idle and preemptible GPU nodes in the cluster, and based on the hierarchical constraints of S204, pre-exclude nodes that do not meet the constraints from the candidate pool, and form a scheduling candidate pool from the remaining nodes and the edges between them.

[0108] S2052. Using the number of cards required for parallel needs as a window, find the dense core node cluster with the highest sum of affinity scores between internal node pairs in the candidate pool. This is affinity priority.

[0109] S2053. Calculate the comprehensive evaluation value and introduce a fragmentation penalty term for secondary correction:

[0110] Total_Score = Affinity_Score(S) - λ · Fragmentation_Cost(S)

[0111] Where Affinity_Score(S) is the sum of affinity scores within the selected node set S; Fragmentation_Cost(S) is the fragmentation cost, which is calculated as follows:

[0112] First, determine the total number of GPUs required for the current task based on the parallel strategy submitted by the user: N_total = TP × PP × DP;

[0113] Then, after selecting node set S, the remaining idle GPUs are counted using the physical rack and leaf switch jurisdiction of each node in S as the statistical unit, without duplicate counting within the same rack or leaf switch jurisdiction. The remaining idle GPUs within each rack and leaf switch jurisdiction that are less than N_total are accumulated and used as the fragmentation cost.

[0114] For example, if the current task requires N_total=32 GPUs, and a rack originally has 8 free GPUs, after removing 4, the remaining 4 cannot make up 32 GPUs, then 4 are included in the fragmentation cost; if 8 are left after removing (which is divisible by 32), then they are not included.

[0115] λ is a dynamic adjustment factor.

[0116] S2054. Determine whether the Total_Score of the current cluster is greater than the preset comprehensive evaluation threshold:

[0117] If so, accept the cluster and jump to S2056;

[0118] If not, return to S2052, exclude the current cluster from the candidate pool, and search for the next core node cluster with the highest total affinity score for scoring, until a node combination that meets the score requirement is found.

[0119] S2055, Dynamically adjust λ based on cluster load status:

[0120] When cluster resources are idle, λ is reduced, affinity is given absolute priority, and optimal network performance is guaranteed for the current task even if fragmentation occurs.

[0121] When cluster resources are scarce and there are many queued tasks, λ is increased, compact layout is prioritized, and current tasks are allowed to cross switches to maximize overall resource utilization and ensure that subsequent large tasks can enter smoothly.

[0122] S2056. Output the optimal node combination to complete the scheduling decision, such as... Figure 3 As shown.

[0123] S300, executes network degradation dynamic perception and thermal obstacle avoidance steps, including:

[0124] S301. The detection agent continuously monitors the link status. When a network degradation is detected, the bandwidth parameter of the degraded link is lowered and / or the latency parameter is raised to reflect the network degradation status, specifically by marking it as unavailable. The topology graph matrix construction module updates the corresponding bandwidth and / or latency parameters in the network status matrix accordingly. As a result, in the affinity matrix recalculated based on the updated bandwidth and / or latency parameters in the matrix, the affinity score between the nodes on both sides of the degraded link naturally drops to an extremely low or negative value.

[0125] S302. For tasks in the queue that are not ready: When the scheduler calculates the optimal node combination for a task, the algorithm automatically filters out the area because the total score of the node combination in the degraded area drops significantly, so that the new task can bypass the degraded node and achieve static avoidance.

[0126] S303. For tasks already in operation, implement a dynamic evaluation and three-level response mechanism:

[0127] Level 1 Response (Mild Degradation): If the increase in latency does not exceed a first preset threshold (e.g., 10%), the task is not migrated. The topology affinity scheduler sends communication parameter adjustment instructions to the large model training framework through a preset control plane interface to adjust the collective communication buffer size; and / or sends routing adjustment instructions to the network interface card through the control plane interface to trigger dynamic routing of the network interface card to avoid congested paths.

[0128] Level 2 Response (Moderate Degradation): If degradation causes an overall throughput decrease exceeding a second preset threshold (e.g., 30%), hot obstacle avoidance is triggered. Combined with the large model's checkpoint mechanism, training is paused at the most recent save point, only containers on the degraded nodes are migrated to healthy standby nodes, and then training is quickly resumed to avoid a full rerun.

[0129] Level 3 Response (Severe Failure): If a severe failure such as NVLink failure occurs within a node, the node will be marked as unschedulable and an alarm will be sent to the operation and maintenance system.

[0130] S304, Complete dynamic obstacle avoidance closed-loop processing, such as Figure 4 As shown.

[0131] Through the above steps S100~S300, this method realizes topology-aware scheduling and dynamic obstacle avoidance for large-scale distributed training tasks, effectively overcoming the "blindness" defect of traditional schedulers and improving GPU utilization and training efficiency.

[0132] Example 2: This example provides a topology-aware scheduling system for large model training, such as... Figure 1 As shown, it includes:

[0133] The automated topology detection engine, deployed on each intelligent computing node, performs intra-node topology detection and inter-node network detection. The detection mechanism employs a seamless design, utilizing off-peak asynchronous detection, QoS priority isolation, and unobtrusive sampling of service data to ensure that the impact of the detection process on service traffic is less than a preset percentage.

[0134] The topology graph matrix construction module, connected to the automated topology detection engine, is used to: receive hardware and network information collected by the detection engine, convert it into a weighted undirected graph, where the edge weights are the original link parameters, including at least bandwidth, latency, and hop count information, and this graph serves as the basic topology graph; update node connectivity and original link parameters in real time, generate and maintain a real-time network state matrix; and, during the scheduling phase, calculate the affinity score for each node pair for the current task based on the weight coefficients provided by the scheduler and the affinity score formula Score = α · Bandwidth - β · Latency - γ · Hop_Count, and construct a task-aware scheduling graph on top of the basic topology graph, using the affinity score as the edge weights.

[0135] A topology affinity scheduler, connected to the topology graph matrix construction module, and pre-configured with a control plane interface for communication with the training framework and network interface card, is used to: receive parallel strategy requirements for large model training tasks submitted by users, wherein the parallel strategies include tensor parallelism (TP), pipeline parallelism (PP), and data parallelism (DP); determine the weight coefficients α, β, and γ of the affinity score formula according to the parallel strategies, wherein the weight coefficients are automatically calculated based on the proportional relationship of each parallelism value: when TP / (PP+DP) is greater than a first preset threshold, β and γ are increased to preset multiples of the baseline value; when PP / (TP+DP) is greater than a second preset threshold, α is increased to a preset multiple of the baseline value, and γ is decreased to a preset proportion of the baseline value; when DP / (TP+PP) is greater than a third preset threshold, α is set to the highest and β to the lowest; and provide the weight coefficients to the topology graph matrix construction module to generate an affinity matrix. Multi-dimensional hierarchical constraint scheduling is implemented: Based on the sensitivity of each parallel mode to communication bandwidth and latency, tensor parallel groups, pipeline parallel groups, and data parallel groups are constrained within their respective hierarchical topology constraints. Parallel groups with higher sensitivity are constrained within topology levels with higher communication performance. In this embodiment, the hierarchical topology constraints are specifically manifested as follows: TP groups are forcibly constrained within the same node, PP groups are scheduled under the same leaf switch, and DP groups are scheduled between nodes that can cross backbone switches. When performing specific node selection, node combinations that do not meet the conditions are first pre-excluded based on the hierarchical constraints. Then, a heuristic graph partitioning algorithm based on multi-objective collaborative optimization is used. Topology-aware matching and evaluation are performed using the comprehensive evaluation value Total_Score = Affinity_Score(S) - λ · Fragmentation_Cost(S), where Fragmentation_Cost(S) is calculated as: the total number of GPUs required for the current task is determined according to the parallel strategy: N_total = TP × PP × DP, after selecting node set S, uses the physical rack and leaf switch jurisdiction of each node in S as the statistical unit, without double counting within the same rack or leaf switch jurisdiction, and accumulates the remaining idle GPUs within each range that are less than N_total as the fragmentation cost; λ is dynamically adjusted according to the cluster load status, decreasing when the cluster is idle to prioritize affinity, and increasing when the cluster is busy to prioritize compact layout; if the Total_Score of the current node cluster is greater than the preset threshold, the cluster is accepted, otherwise the current cluster is excluded and a new search is performed; based on the evaluation results, Pod scheduling instructions are issued to allocate tasks to the optimal node combination in the cluster infrastructure.

[0136] The topology affinity scheduler also includes a dynamic obstacle avoidance control function, used to: when the topology automated detection engine detects network degradation, notify the topology graph matrix construction module to immediately update the network state matrix, lower the bandwidth parameter and / or raise the latency parameter of the degraded link, so that the affinity score between nodes in the degraded area in the affinity matrix recalculated based on the updated bandwidth and / or latency parameters in the updated network state matrix will naturally decrease and be automatically filtered in subsequent scheduling; for tasks that are already running, execute a three-level response mechanism, including communication operator fine-tuning when the latency increase does not exceed the first preset threshold, checkpoint hot migration and breakpoint recovery when the overall throughput decreases by more than the second preset threshold, and node isolation and alarm in case of severe failure.

[0137] The cluster infrastructure, which consists of underlying physical resources, including GPUs of different generations, network interface cards, and a physical network composed of leaf switches and backbone switches, is used to receive scheduling instructions issued by the topology affinity scheduler and execute specific training tasks.

[0138] Example 3: This example provides a computer-readable storage medium, including:

[0139] The storage medium stores computer software instructions used to implement the topology-aware scheduling method for large model training as described in Embodiment 1, including programs for executing the aforementioned topology-aware scheduling method. This executable program can be embedded in the topology-aware scheduling system described in Embodiment 2, thus enabling the topology-aware scheduling system to implement the method described in Embodiment 1 by executing the embedded executable program.

[0140] Furthermore, the computer-readable storage medium in this embodiment can be any combination of one or more readable storage media, wherein the readable storage medium includes an electrical, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0141] Unlike existing technologies, this application employs a topology-aware scheduling method for large-scale model training. This method enables topology-aware scheduling and dynamic obstacle avoidance in network degradation for distributed training tasks of large models. It effectively overcomes the communication congestion and low computing power utilization caused by the "blindness" of traditional schedulers regarding physical network topology. It ensures that the multi-dimensional parallel communication requirements of large models are accurately mapped to the optimal positions of the three-layer physical topology of "intra-node, intra-rack, and inter-rack," thereby improving the throughput efficiency of large-scale model training, shortening the model training cycle, and effectively alleviating the core network congestion and Incast packet loss problems of intelligent computing centers. When network degradation occurs, it can achieve hot obstacle avoidance without interrupting training, making up for the shortcomings of existing technologies and possessing high application value.

[0142] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0143] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0144] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0145] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0146] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0147] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0148] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0149] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A topology-aware scheduling method for training large models, characterized in that, Includes the following steps: Responding to the graph construction phase of the scheduling system: A probe agent is deployed on each intelligent computing node to perform intra-node topology probe and inter-node network probe, collect hardware and network information, and generate a real-time network state matrix based on the hardware and network information. The real-time network state matrix includes communication link parameters between node pairs. The perceptual matching scheduling phase of the scheduling system: The system receives training tasks and parallel strategies submitted by users, including tensor parallelism (TP), pipeline parallelism (PP), and data parallelism (DP). Based on the parallel strategy, the weight coefficients of the affinity score formula are determined. The affinity score formula is used to convert the communication parameters of heterogeneous links into standardized scores that can be compared globally. Based on the weighting coefficients and the real-time network state matrix, the affinity score between each node pair is calculated, and a task-aware scheduling graph is constructed. The hierarchical constraint scheduling is performed based on the parallel strategy, wherein the hierarchical constraint scheduling includes constraining parallel groups with different communication sensitivities to a matching physical topology range; A heuristic graph partitioning algorithm is used to extract candidate node clusters from the scheduling graph under the constraints corresponding to the hierarchical constraints. The candidate node clusters are then filtered based on the comprehensive evaluation value to output the optimal node combination to complete the scheduling. The comprehensive evaluation value is determined by the affinity score and the fragmentation cost.

2. The topology-aware scheduling method for large model training according to claim 1, characterized in that: The process of collecting hardware and network information and generating a real-time network state matrix based on the hardware and network information further includes: converting the hardware and network information into a weighted undirected graph as a basic topology graph, wherein the weights of the edges are the original link parameters, including at least bandwidth, latency, and hop count information; and generating a real-time network state matrix containing bandwidth, latency, and hop count information between node pairs based on the original link parameters.

3. The topology-aware scheduling method for large model training according to claim 2, characterized in that: The construction of the task-aware scheduling graph includes replacing the weights of each edge in the basic topology graph with the affinity scores between corresponding node pairs.

4. The topology-aware scheduling method for large model training according to claim 1, characterized in that: The step of determining the weight coefficients of the affinity score formula based on the parallel strategy further includes: the affinity score formula is: Score = α·Bandwidth - β·Latency - γ·Hop_Count, where Bandwidth is the link bandwidth, Latency is the link latency, Hop_Count is the number of hops across switches, and α, β, and γ are weight coefficients; the values ​​of each weight coefficient are determined based on the proportional relationship between the TP value, PP value, and DP value. When TP / (PP+DP) is greater than the first preset threshold, the values ​​of β and γ are increased to prioritize the use of GPUs within the same physical rack and exclude cross-switch deployments. When PP / (TP+DP) is greater than the second preset threshold, the value of α is increased and the value of γ is decreased to allow cross-switch deployment but prioritize ensuring uniform and stable bandwidth between nodes. When DP / (TP+PP) is greater than the third preset threshold, α is set to the highest value and β is set to the lowest value, so as to prioritize finding the network card combination with the largest total bandwidth.

5. The topology-aware scheduling method for large model training according to claim 1, characterized in that: The step of screening candidate node clusters through comprehensive evaluation values ​​further includes: Calculate the comprehensive evaluation value of the candidate node cluster: Total_Score = Affinity_Score(S) - λ·Fragmentation_Cost(S); Wherein, Affinity_Score(S) is the sum of affinity scores between node pairs within the selected node set S; Fragmentation_Cost(S) is the fragmentation cost, which is calculated as follows: first, the total number of GPUs required for the current task is determined according to the parallel strategy, N_total = TP×PP×DP; then, after selecting the node set S, the number of remaining idle GPUs within the jurisdiction of each physical rack and leaf switch that is less than N_total is counted, and the number is accumulated as the fragmentation cost; λ is a dynamic adjustment factor; Determine whether the comprehensive evaluation value is greater than the preset comprehensive evaluation value threshold: if yes, accept the current candidate node cluster; if no, exclude the current candidate node cluster and search again.

6. A topology-aware scheduling method for large model training according to claim 5, characterized in that: The λ is a dynamic adjustment factor, and its adjustment method is as follows: When cluster resources are idle, λ takes the smaller value, with affinity being the absolute priority; When cluster resources are scarce and there are many queued tasks, λ should be taken as a larger value to prioritize compact layout.

7. A topology-aware scheduling method for large model training according to claim 1, characterized in that: It also includes the following steps: In response to the dynamic obstacle avoidance phase of the scheduling system: The detection agent continuously monitors the link status. When network degradation is detected, it lowers the bandwidth parameter of the corresponding link in the real-time network status matrix and / or raises its latency parameter. Based on the updated real-time network state matrix, the affinity score between each node pair is recalculated, thereby reducing the affinity score between nodes on both sides of the degraded link. For queued tasks that are not ready, when calculating the comprehensive evaluation value for the candidate node cluster, the comprehensive evaluation value of the candidate node combination containing the degraded link is reduced, and it is automatically filtered during scheduling calculation, thus achieving static avoidance.

8. A topology-aware scheduling method for large model training according to claim 7, characterized in that: The dynamic obstacle avoidance phase also includes a dynamic evaluation and response mechanism for tasks already in operation, specifically including: When the degradation causes a latency increase of no more than 10%, execute the first-level response: do not migrate the task, and send instructions to the collective communication library and / or network card of the training framework through the control plane interface to adjust the buffer size of the collective communication or trigger the network card's dynamic routing to avoid congested paths. When degradation causes an overall throughput drop of more than 30%, a secondary response is executed: triggering thermal obstacle avoidance, obtaining checkpoint information of the training task, pausing training at the most recent save point, and resuming training after migrating the container on the degradation node to a healthy standby node. When a high-speed interconnection failure occurs within a node, a three-level response is executed: the node is marked as unschedulable and an alarm is sent to the operation and maintenance system.

9. A topology-aware scheduling method for large model training according to claim 1, characterized in that: During the execution of intra-node topology probing and inter-node network probing, the probing agent employs a non-intrusive probing mechanism, specifically including: Asynchronous detection is performed using microsecond-level pulses, with each data transmission being less than a preset number of bytes. By isolating probing packets by quality of service priority, the probing packets are assigned to a low-priority queue and the training traffic is assigned to a high-priority queue. When the network is congested, the switch is allowed to drop probing packets first. The probing agent infers the congestion level of the current network link by monitoring the drop rate of the probing packets. The physical layer performance counters of the node's network card are read directly, and the network quality is evaluated by analyzing the packet loss rate and retransmission rate of the actual traffic.

10. A topology-aware scheduling method for large model training according to claim 1, characterized in that: The hierarchical constraint scheduling includes: Prioritize constraining tensor parallel groups within the same node and utilize high-speed interconnect communication within the node; Secondly, the parallel pipeline groups are constrained between different nodes under the same leaf switch. Finally, the data parallel groups are distributed among nodes that can span the backbone switch.

Images