GPU heterogeneous cluster scheduling method and system oriented to large model training and reasoning

Abstract

The invention relates to the technical field of cluster scheduling, and provides a GPU heterogeneous cluster scheduling method and system oriented to large model training and reasoning, which constructs a set of complete cluster scheduling system by integrating multi-source information such as hardware features, running states and historical task data and applying technologies such as a clustering algorithm, a fuzzy comprehensive evaluation method and reinforcement learning. Comprehensive, intelligent and dynamic management and scheduling of GPU cluster resources are realized, the cluster scheduling system can significantly improve the execution efficiency of GPU heterogeneous clusters in large model training and reasoning tasks, the resource utilization rate is improved, the energy consumption is reduced, and the stability and adaptability of the system are enhanced. And an efficient and reliable solution is provided for large-scale deep learning application.

Description

Technical Field

[0001] The present application relates to the field of cluster scheduling technology, and in particular to a GPU heterogeneous cluster scheduling method and system for large model training and inference. Background Art

[0002] With the rapid development of artificial intelligence, large models are widely used in scenarios such as natural language processing and autonomous driving. The demand for computing resources for their training and inference has increased dramatically, and GPU heterogeneous clusters have become critical. However, large model training tasks in natural language processing and autonomous driving scenarios all contain subtasks with complex dependencies. Traditional scheduling systems often ignore these dependencies when splitting training tasks, resulting in a chaotic execution order of subtasks and affecting the training process. In addition, when resources compete, existing resource conflict detection and resolution mechanisms are not intelligent. For example, in emergency situations in autonomous driving, high-priority real-time road condition inference tasks may not be able to obtain resources in a timely manner because their resources are occupied by low-priority tasks, endangering driving safety. Most systems have not yet addressed the problem of determining the resource status of each GPU node in a GPU heterogeneous cluster and effectively scheduling large model training and inference tasks based on their different types. Summary of the Invention

[0003] To address the shortcomings of the existing technology, this application provides a GPU heterogeneous cluster scheduling method and system for large model training and inference.

[0004] In a first aspect, the present application provides a GPU heterogeneous cluster scheduling system for large-model training and inference, the system comprising: a heterogeneous profiling module, a topology generation module, an elastic sharding module, and a target scheduling module; the heterogeneous profiling module is used to obtain the hardware characteristics and operating status of the GPU cluster, and to determine the memory access pattern and instruction set matching degree through clustering algorithm analysis based on historical task data, and to calculate the scheduling weight factor of each GPU node through weighted summation, and to determine the resource status of each GPU node based on the operating status of the GPU cluster, and to generate a resource profiling library;

[0005] The topology generation module is used to filter GPU nodes according to their resource status and task type to form an available node set, generate a hardware connection topology by task type, and output topology configuration instructions based on the hardware connection topology. It also outputs a resource reservation pool for the filtered high-priority tasks.

[0006] The elastic sharding module is used to split the training task into multiple subtasks based on the task dependencies and resource reservation pool of the historical sharding task graph, generate a sharding task graph, assign priorities to inference tasks, and generate resource preemption logic for inference tasks.

[0007] The target scheduling module is used to generate a scheduling plan through reinforcement learning based on the sharding task graph, resource preemption logic and resource reservation pool. The scheduling plan includes the task deployment location and the sharding task preemption order, and the scheduling plan is selected through the fuzzy comprehensive evaluation method in combination with the resource status of the GPU node.

[0008] As an optional implementation, the logic for determining the resource status of each GPU node includes:

[0009] Obtain the hardware characteristics and operating status of the GPU cluster. The hardware characteristics of the GPU cluster include peak computing power, memory bandwidth, and NVLink connection topology. The operating status of the GPU cluster includes temperature, energy consumption, and failure rate.

[0010] Extract task types, task runtimes, and GPU resource usage from historical task data, and analyze them using a clustering algorithm to determine memory access patterns and instruction set matching.

[0011] The weights of the memory access pattern and instruction set matching are determined by the hierarchical analysis method, and the scheduling weight factor of each GPU node is calculated by weighted summation.

[0012] The resource status of each GPU node is determined by the fuzzy comprehensive evaluation method based on the scheduling weight factor of each GPU node and the operating status of the GPU cluster.

[0013] As an optional implementation manner, the sub-logic for determining the matching degree between the video memory access mode and the instruction set includes:

[0014] Clustering algorithms are used to cluster hardware features, operating status, and historical task data to form clusters.

[0015] By counting the proportion of access times of different video memory address spaces by tasks in the cluster, the video memory access pattern of the cluster is determined;

[0016] The instruction set matching degree is determined by calculating the ratio of the time that the tasks in the cluster use the instruction set to the task running time.

[0017] As an optional implementation, the output logic of the topology configuration instruction includes:

[0018] Monitor the resource status of all GPU nodes, filter GPU nodes based on resource status and task type, and form a set of available nodes;

[0019] Distinguish task types and generate hardware connection topology based on task types;

[0020] Simulate the execution process of tasks under the hardware connection topology, monitor the communication load, computing load and resource utilization between GPU nodes, and iteratively optimize the hardware connection topology through the simulated annealing algorithm;

[0021] Based on the iteratively optimized hardware connection topology, it outputs topology configuration instructions, which include NVLink reconnection strategies and cross-node RDMA path planning.

[0022] As an optional implementation manner, the generation sub-logic of the hardware connection topology includes:

[0023] Task types include training tasks and inference tasks. Hardware connection topology is generated based on the task type. A ring topology is generated for training tasks, and a parallel topology is generated for inference tasks.

[0024] Dynamically adjust the level of the ring topology based on the complexity of the training task and the size of the data, combined with the available node set;

[0025] The parallelism of the parallel topology is dynamically adjusted based on the load of the inference task and the available node set.

[0026] As an optional implementation manner, the output logic of the resource reservation pool includes:

[0027] Sort tasks by priority based on their importance, urgency, and resource requirements, and select high-priority tasks;

[0028] Predict the resource requirements of high-priority tasks based on their resource requirements and historical execution data;

[0029] Perform resource evaluation on the GPU nodes in the available node set to obtain the resource status of the GPU nodes. Based on the resource requirements of high-priority tasks and the resource status of the GPU nodes, select GPU nodes to form a resource reservation pool.

[0030] Reserve a spare node for each resource reservation pool, monitor the execution of high-priority tasks and the resource status of GPU nodes in the resource reservation pool in real time, and adjust the allocation of the resource reservation pool.

[0031] As an optional implementation, the generation logic of the shard task graph includes:

[0032] Obtain the historical sharding task graph of the GPU cluster, analyze the historical sharding task graph of the GPU cluster through graph clustering, and obtain task dependencies;

[0033] Split the training task into multiple subtasks based on task dependencies;

[0034] Evaluate the degree of adaptation of each subtask to each GPU node in the resource reservation pool to obtain the resource adaptation result of the subtask;

[0035] Based on the results of subtask splitting and resource adaptation, a sharding task graph is generated. The nodes of the sharding task graph represent subtasks, and the edges of the sharding task graph represent the dependencies between subtasks.

[0036] As an optional implementation, the resource preemption logic includes:

[0037] Dynamically adjust the priority of inference tasks based on the resource status of the GPU cluster and the execution status of the task;

[0038] Monitor the resource status of each GPU node in the GPU cluster in real time, detect resource conflicts through the threshold comparison method of the sliding window algorithm, and trigger resource conflict warnings;

[0039] Resource preemption is performed based on the priority of the inference task and the degree of resource conflict, while the priority of the subtasks of the training task and the impact of task execution are comprehensively analyzed.

[0040] As an optional implementation manner, the generation logic of the scheduling scheme includes:

[0041] Integrate and standardize the sharding task graph, resource preemption logic, resource reservation pool, and resource status of each GPU node;

[0042] Build a reinforcement learning model that uses the resource status of each GPU node, resource preemption logic, resource reservation pool, and nodes and edges of the shard task graph as the state space, the task deployment location and shard task preemption order as the action space, and task priority, resource utilization, and task completion time as the reward function;

[0043] The reinforcement learning model is trained through the deep Q network to generate a scheduling plan, which includes the task deployment location and the preemption order of sharded tasks.

[0044] In a second aspect, the present application provides a GPU heterogeneous cluster scheduling method for large model training and inference, the method comprising: obtaining the hardware characteristics and operating status of the GPU cluster, and analyzing the historical task data through a clustering algorithm to determine the memory access pattern and instruction set matching degree;

[0045] The scheduling weight factor of each GPU node is calculated by weighted summation, and the resource status of each GPU node is determined in combination with the operating status of the GPU cluster to generate a resource profile library;

[0046] Filter GPU nodes based on their resource status and task type, generate hardware connection topology based on task type, and output topology configuration instructions and resource reservation pool;

[0047] Based on the task dependencies and resource reservation pool of the historical sharding task graph, the training task is split into multiple subtasks to generate a sharding task graph. At the same time, priorities are assigned to inference tasks and resource preemption logic for inference tasks is generated.

[0048] Generate a scheduling plan based on the sharded task graph, resource preemption logic, and resource reservation pool through reinforcement learning. The scheduling plan includes the task deployment location and the sharded task preemption order.

[0049] The scheduling scheme is selected by fuzzy comprehensive evaluation method based on the resource status of GPU nodes.

[0050] Compared with the existing technology, the beneficial effects of this application are: by integrating multi-source information such as hardware characteristics, operating status and historical task data, and using advanced technologies such as clustering algorithms, fuzzy comprehensive evaluation methods and reinforcement learning, a complete cluster scheduling system is constructed, which realizes comprehensive, intelligent and dynamic management and scheduling of GPU cluster resources. This cluster scheduling system can significantly improve the execution efficiency of GPU heterogeneous clusters in large model training and inference tasks, improve resource utilization, reduce energy consumption, enhance system stability and adaptability, and provide an efficient and reliable solution for large-scale deep learning applications.

[0051] By obtaining the hardware characteristics and operating status of the GPU cluster, the system has a comprehensive and accurate understanding of the cluster's hardware foundation and real-time operating status, which provides a solid data foundation for subsequent task allocation and resource scheduling, ensuring that scheduling decisions are based on accurate cluster status; at the same time, combined with historical task data, the memory access pattern and instruction set matching degree are determined through clustering algorithms, which can deeply explore the potential connection between task characteristics and hardware resources. This correlation analysis helps to more accurately evaluate the adaptability of each GPU node to different types of tasks, providing a strong basis for the rational allocation of resources; the scheduling weight factor of each GPU node is calculated through weighted summation, and the resource status is determined in combination with the cluster operating status to generate a resource portrait library. The resource portrait library provides a quantitative description of cluster resources, so that the scheduling process can be carried out according to clear indicators, improving the scientificity and accuracy of resource scheduling and avoiding subjective arbitrariness.

[0052] Nodes are screened and hardware connection topologies are generated based on the resource status and task types of GPU nodes. This can fully consider the computing and communication requirements of different task types, optimize data transmission and collaborative computing between nodes, and improve task execution efficiency. Simulating the execution process of tasks under the topology, optimizing the hardware connection topology, and outputting topology configuration instructions can significantly improve the rationality of the hardware connection topology, reduce communication delays and resource waste, and further improve cluster performance. High-priority tasks are screened based on task priority, their resource requirements are predicted, and a resource reservation pool is built based on the resource status of GPU nodes, providing resource guarantees for high-priority tasks, ensuring that critical tasks can obtain the required resources first, and improving the responsiveness and execution reliability of the cluster scheduling system for important tasks.

[0053] Based on the task dependencies and resource reservation pool of the historical sharding task graph, the training task is split into multiple subtasks and a sharding task graph is generated. This can fully utilize the resources in the resource reservation pool, reasonably arrange the execution order of subtasks according to the task dependencies, improve the parallel execution efficiency of training tasks, and reduce task execution time; assign priorities to inference tasks, dynamically adjust priorities according to the resource status of the GPU cluster and the task execution status, and generate resource preemption logic for inference tasks, realizing dynamic management of inference task resources, ensuring the resource requirements of high-priority inference tasks, improving resource utilization, and avoiding interference of resource conflicts on task execution.

[0054] Based on the sharded task graph, resource preemption logic, and resource reservation pool, a scheduling plan is generated through reinforcement learning, taking into account the task deployment location and the sharded task preemption order. The reinforcement learning model can continuously learn and optimize the scheduling strategy in a complex cluster environment, generating more efficient and more practical scheduling plans, thereby improving the overall efficiency of task execution. Combined with the resource status of the GPU node, a scheduling plan is selected through a fuzzy comprehensive evaluation method to ensure that the selected scheduling plan can achieve a balance on multiple key indicators, meet the system performance requirements in different application scenarios, and improve the overall performance and resource utilization efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be derived from these drawings without inventive work. Among them:

[0056] Figure 1 This is a system flow chart of the GPU heterogeneous cluster scheduling system for large model training and inference provided by an embodiment of the present application;

[0057] Figure 2 A logical diagram for determining the resource status of each GPU node in the GPU heterogeneous cluster scheduling system for large model training and inference provided by an embodiment of the present application;

[0058] Figure 3 A sub-logic diagram for generating the hardware connection topology of a GPU heterogeneous cluster scheduling system for large model training and inference provided in an embodiment of the present application;

[0059] Figure 4 This is a flow chart of the method for scheduling GPU heterogeneous clusters for large model training and inference provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application are clearly and completely described below in conjunction with the drawings in the specification. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0061] Example 1

[0062] like Figure 1 As shown, a system flow chart of a GPU heterogeneous cluster scheduling system for large model training and inference is provided for an embodiment of the present application. The system includes a heterogeneous profiling module, a topology generation module, an elastic sharding module and a target scheduling module.

[0063] The heterogeneous portrait module is used to obtain the hardware characteristics and operating status of the GPU cluster, and to determine the memory access pattern and instruction set matching degree through clustering algorithm analysis based on historical task data. The scheduling weight factor of each GPU node is calculated through weighted summation. The resource status of each GPU node is determined based on the operating status of the GPU cluster to generate a resource portrait library.

[0064] Specifically, if Figure 2 As shown, the logic for determining the resource status of each GPU node includes:

[0065] Obtain the hardware characteristics and operating status of the GPU cluster. The hardware characteristics of the GPU cluster include peak computing power, memory bandwidth, and NVLink connection topology. The operating status of the GPU cluster includes temperature, energy consumption, and failure rate.

[0066] Extract task types, task runtimes, and GPU resource usage from historical task data, and analyze them using a clustering algorithm to determine memory access patterns and instruction set matching.

[0067] The weights of the memory access pattern and instruction set matching are determined by the hierarchical analysis method, and the scheduling weight factor of each GPU node is calculated by weighted summation.

[0068] The resource status of each GPU node is determined by the fuzzy comprehensive evaluation method based on the scheduling weight factor of each GPU node and the operating status of the GPU cluster.

[0069] A comprehensive understanding of the hardware capabilities and real-time operating status of each GPU node in the GPU cluster is the foundation for subsequent resource assessment and scheduling. Different hardware characteristics and operating states directly affect node performance and availability. For example, GPU nodes with high peak computing power are more suitable for processing complex computing tasks, while GPU nodes with excessively high temperatures may face the risk of performance degradation or even failure. Regarding hardware characteristics, system management software is used to obtain information such as peak computing power and memory bandwidth. NVLink connection topology can be monitored and recorded using network management software. Regarding the operating status of the GPU cluster, temperature and energy consumption can be monitored in real time using sensors on GPU nodes, while failure rates can be statistically calculated by collecting historical failure records and current error logs. Raw data is standardized to eliminate dimensionality and make different types of data comparable. For example, temperatures are normalized to a range of 0 to 1, with 0 representing the lowest safe temperature and 1 representing the highest safe temperature. This provides comprehensive and accurate basic data, providing a reliable basis for subsequent analysis and calculations. This data will serve as input for subsequent clustering analysis and weight calculation. The accuracy of hardware characteristics and operating status directly affects the calculation results of memory access patterns, instruction set compatibility, and scheduling weight factors.

[0070] Historical task data reflects the actual performance and resource usage patterns of GPU nodes. Combined with the current hardware characteristics and operating status, it can more accurately evaluate the performance and applicability of the nodes. Different types of tasks have different requirements for GPU resources. Task runtime and resource usage can help us understand the performance of GPU nodes under different loads. Historical task data is extracted from the log database, where task types include training tasks and inference tasks. The historical task data is cleaned and preprocessed to remove invalid and erroneous records. This provides rich historical task data, which provides more feature dimensions for subsequent clustering analysis and helps to more accurately determine the memory access pattern and instruction set matching.

[0071] The clustering algorithm can group tasks and GPU nodes with similar characteristics into one category, thereby finding different memory access patterns and instruction set matching situations, which helps to perform more accurate scheduling based on different task requirements and node characteristics and improve resource utilization. The DBSCAN algorithm is selected to take hardware characteristics, operating status and historical task data as input for clustering analysis. When clustering, the focus is on features such as memory access frequency, access size distribution and the proportion of different instruction sets used in task execution. For each cluster, the proportion of the number of times a task accesses different memory address spaces is counted to determine the memory access pattern, and the proportion of the time a task uses the instruction set to the task running time is calculated to determine the instruction set matching degree. Thus, different memory access patterns and instruction set matching degrees are obtained, which provides an important reference for the subsequent calculation of scheduling weight factors.

[0072] Different factors have different importance for the scheduling of GPU nodes. The hierarchical analysis method can be used to determine the relative weights of memory access mode and instruction set matching, so as to more reasonably calculate the scheduling weight factor and provide a more scientific basis for resource scheduling. A hierarchical model is constructed, with the target layer set as the scheduling weight factor, the criterion layer including factors such as memory access mode and instruction set matching, and the solution layer for each GPU node. The relative importance of each factor in the criterion layer is determined by expert scoring, and a judgment matrix is constructed. The eigenvectors and eigenvalues are calculated to obtain the weight of each factor. The numerical value of each factor is then multiplied by the corresponding weight and summed to obtain the scheduling weight factor of each GPU node. The scheduling weight factor of each GPU node reflects the priority and applicability of the GPU node in resource scheduling.

[0073] The resource status of GPU nodes is affected by many factors, and there is a certain degree of fuzziness and uncertainty between these factors. The fuzzy comprehensive evaluation method can comprehensively consider multiple factors such as the scheduling weight factor and the operating status of the GPU cluster to more accurately describe the resource status of the GPU node; determine the evaluation factor set, including the scheduling weight factor, temperature, energy consumption and failure rate, etc., establish an evaluation set, the evaluation set includes {high resource status, medium resource status, low resource status}, construct a fuzzy relationship matrix, take the scheduling weight factor and the operating status of the GPU cluster as input, and obtain the status of each GPU node through fuzzy transformation. Based on the membership of different resource states, the resource state of the GPU node is determined according to the maximum membership principle. For example, the membership interval of the high resource state is [0.8, 1], the membership interval of the medium resource state is [0.5, 0.8), and the membership interval of the low resource state is [0, 0.5). This accurately determines the resource state of each GPU node, providing an important basis for the subsequent topology generation and task scheduling modules. The resource state of each GPU node will serve as an important basis for the topology generation module to screen GPU nodes, affecting the generation of hardware connection topology and the allocation of resource reservation pools.

[0074] This generates a resource profile library, which includes the memory access mode, instruction set matching degree and resource status of each GPU node.

[0075] Furthermore, the sub-logic for determining the matching degree between the video memory access mode and the instruction set includes:

[0076] Clustering algorithms are used to cluster hardware features, operating status, and historical task data to form clusters.

[0077] By counting the proportion of access times of different video memory address spaces by tasks in the cluster, the video memory access pattern of the cluster is determined;

[0078] The instruction set matching degree is determined by calculating the ratio of the time that the tasks in the cluster use the instruction set to the task running time.

[0079] Hardware features, operating status and historical task data contain rich information, but there is a certain degree of complexity and correlation between the data. The clustering algorithm can classify similar data points into one category to form clusters, thereby discovering the potential patterns and laws in the data, and providing a basis for determining the matching degree of memory access mode and instruction set; the DBSCAN algorithm is selected to first standardize the data to eliminate the dimensional influence between different features, and then initialize the cluster center. Through iterative calculation, the cluster center is continuously updated until the convergence condition is met, and finally different clusters are obtained; thereby effectively classifying the data. Each cluster represents a group of tasks and nodes with similar characteristics, providing a clear structure for subsequent analysis. The formation of clusters is the basis for determining the matching degree of memory access mode and instruction set. Different clusters have different memory access modes and instruction set matching situations.

[0080] The memory access pattern reflects the usage and characteristics of the video memory during the execution of the task. Understanding the memory access pattern helps to optimize video memory management and improve video memory utilization. By counting the proportion of the number of times tasks in the cluster access different video memory address spaces, the memory access pattern of the cluster can be accurately described. In each cluster, the memory access records of the tasks are analyzed, and the video memory address space is divided into several areas. The number of times the tasks access each area is counted, and then the proportion of the number of visits to each area to the total number of visits is calculated to obtain the memory access pattern of the cluster. In this way, the memory access pattern of each cluster is obtained, which provides an important reference for subsequent resource scheduling and optimization. The memory access pattern will be an important factor in calculating the scheduling weight factor, affecting the calculation result of the scheduling weight factor, and then affecting the determination of resource status and task scheduling.

[0081] Different tasks use different instruction sets. The instruction set matching degree reflects the degree of adaptability between the task and the GPU node instruction set. Understanding the instruction set matching degree helps to select more suitable GPU nodes to execute tasks and improve the execution efficiency of tasks. In each cluster, the duration of the task using different instruction sets and the total running time of the task are recorded, and the proportion of the usage time of each instruction set to the total running time is calculated to obtain the instruction set matching degree of the cluster. This provides an important basis for subsequent task scheduling and resource allocation. The instruction set matching degree will be an important factor in calculating the scheduling weight factor, affecting the calculation result of the scheduling weight factor, and then affecting the determination of resource status and task scheduling.

[0082] The topology generation module is used to filter GPU nodes according to their resource status and task type, generate hardware connection topology according to task type, and output topology configuration instructions and resource reservation pool.

[0083] Specifically, the output logic of the topology configuration instruction includes:

[0084] Monitor the resource status of all GPU nodes, filter GPU nodes based on resource status and task type, and form a set of available nodes;

[0085] Distinguish task types and generate hardware connection topology based on task types;

[0086] Simulate the execution process of tasks under the hardware connection topology, monitor the communication load, computing load and resource utilization between GPU nodes, and iteratively optimize the hardware connection topology through the simulated annealing algorithm;

[0087] Based on the iteratively optimized hardware connection topology, it outputs topology configuration instructions, which include NVLink reconnection strategies and cross-node RDMA path planning.

[0088] Different task types have different requirements for GPU node resources, and the resource status of GPU nodes changes in real time. Screening out nodes that meet task requirements and are currently available can ensure that subsequent topology construction is based on effective resources and improve task execution efficiency. For example, training tasks usually have high requirements for computing power and video memory, while reasoning tasks focus more on low latency and a certain degree of parallel processing capabilities; continuously obtain resource status data of GPU nodes, including evaluation sets and corresponding computing power, video memory, temperature, energy consumption and failure rate, etc., divide task types into training tasks and reasoning tasks, and after determining the resource status of each GPU node using the fuzzy comprehensive evaluation method, set the corresponding quantitative value for the membership interval of each resource status. For example, the membership interval [0.8,1] of the high resource status is quantized to the numerical range [80,100], the membership interval [0.5,0.8) of the medium resource status is quantized to [50,80), and the membership interval [0,0.5) of the low resource status is quantized to [0 ,50), when the membership of a GPU node in the high resource state is 0.9, the corresponding quantization value is 90. Based on the above quantization method, the quantization value 90 is used as the resource threshold to judge whether it is greater than 90%. GPU nodes with resource status greater than 90% of the resource threshold are preferentially selected and added to the available node set. Then, nodes are filtered according to different task types, and the node information that meets the conditions is organized into an available node set. For example, for training tasks, GPU nodes with high resource status and stable NVLink connection topology are preferentially filtered, while for inference tasks, GPU nodes with medium resource status and high memory bandwidth are preferentially filtered. In this way, a set of GPU nodes with good resource status for specific task types is obtained, which provides a basis for constructing a reasonable hardware connection topology. The available node set determines the node selection range during the subsequent hardware connection topology construction. Its accuracy and completeness directly affect the performance and applicability of the topology.

[0089] Training tasks and inference tasks differ in terms of computing characteristics and data transmission requirements. Targeted generation of different topologies can optimize data communication and computing collaboration efficiency during task execution. For example, training tasks require large amounts of data and a large amount of data interaction between nodes. A ring topology helps to transmit data in an orderly manner between nodes. Inference tasks have high real-time requirements. A parallel topology enables multiple nodes to process data simultaneously, reducing latency. Therefore, a ring topology is required for training tasks. A graph theory algorithm is used to determine the order of GPU nodes in the ring. Considering the quality and bandwidth of the NVLink connection between nodes, closely connected GPU nodes are preferentially placed adjacent to each other. For the parallel topology of inference tasks, a heuristic algorithm is used to group GPU nodes. The GPU nodes in each group process data in parallel and are reasonably grouped according to the computing power and memory bandwidth of the GPU nodes. This generates a hardware connection topology structure that is suitable for different task types, ensuring the efficient operation of tasks on the GPU cluster. The hardware connection topology is the basis for simulating task execution and performing topology optimization. Its rationality affects the simulation results and optimization direction.

[0090] When actual tasks are executed, the communication and computing loads between GPU nodes will be unbalanced, and resource utilization will also be low. Through simulation and optimization, these problems can be discovered and solved in advance, thereby improving the overall performance of the GPU cluster. Through task simulation software (such as SimGrid), input parameters such as the computing amount, data transmission amount, and task dependency of the task, and simulate execution in combination with the generated hardware connection topology. During the simulation process, the communication load, computing load, and resource utilization between GPU nodes are monitored in real time. The communication load is determined by counting the data transmission amount and transmission time between GPU nodes, the computing load is determined by monitoring the computing power usage of the nodes, and the resource utilization is determined by calculating resources such as video memory and memory. The monitoring data is used as the input of the simulated annealing algorithm. The simulated annealing algorithm randomly adjusts the topology structure, such as changing the order of nodes in the ring topology or the grouping in the parallel topology, and calculates the objective function value under the new topology, such as the comprehensive index of minimizing the communication load and maximizing the resource utilization. The acceptance probability is used to decide whether to accept the new topology. After multiple iterations, the optimized topology is obtained. The optimized hardware connection topology can more effectively balance the communication and computing load between nodes, improve resource utilization, and enhance task execution efficiency. The iteratively optimized hardware connection topology is the basis for generating topology configuration instructions. Its performance directly determines the effect of the final topology configuration instructions on improving cluster performance.

[0091] In order to implement the optimized hardware connection topology in the actual GPU cluster, it is necessary to generate specific configuration instructions to guide the system to adjust and set the hardware connection to achieve the optimized performance effect; for the NVLink reconnection strategy, based on the optimized topology structure, determine which NVLinks need to be reconnected, the connection order and parameter settings, and thus generate the corresponding NVLink reconnection instructions; and for cross-node RDMA path planning, use the network topology analysis tool to determine the optimal RDMA path for data transmission between GPU nodes, generate RDMA path planning instructions containing parameters such as path information and bandwidth allocation, organize these instructions into topology configuration instructions, and output them to the cluster scheduling system; thereby enabling the GPU cluster to adjust and set the hardware connection according to the topology configuration instructions, realize the optimized hardware connection topology, and improve cluster performance. The topology configuration instructions are the operating guide for the actual operation of the GPU cluster, directly affecting the hardware connection status and task execution environment of the GPU cluster, and laying the foundation for subsequent task scheduling and execution.

[0092] Further, if Figure 3 As shown, the generation logic of the hardware connection topology includes:

[0093] Task types include training tasks and inference tasks. Hardware connection topology is generated based on the task type. A ring topology is generated for training tasks, and a parallel topology is generated for inference tasks.

[0094] Dynamically adjust the level of the ring topology based on the complexity of the training task and the size of the data, combined with the available node set;

[0095] The parallelism of the parallel topology is dynamically adjusted based on the load of the inference task and the available node set.

[0096] Training tasks require large-scale data parallel computing and frequent data exchange. Ring topologies can provide good scalability and fault tolerance while ensuring orderly data transmission. Inference tasks emphasize fast response and parallel processing capabilities. Parallel topologies enable multiple nodes to process data simultaneously, meeting their low latency requirements. To generate a ring topology for training tasks, the available node set is first sorted in descending order based on resource indicators such as node computing power and video memory. The nodes are then connected into a ring, and appropriate link parameters such as bandwidth and transmission rate are set using NVLink. For inference tasks, the parallel topology is grouped according to computing power and video memory size, ensuring that the computing power and video memory resources of the nodes in each group are relatively balanced. The nodes in the group are connected via parallel buses or high-speed networks to achieve data parallel processing. This generates a basic hardware connection topology structure that is suitable for different task types and provides a basic framework for efficient task execution on GPU clusters. This basic hardware connection topology is the starting point for subsequent dynamic adjustments, and its rationality affects the effectiveness of dynamic adjustments and the performance of the final topology.

[0097] Training tasks vary in complexity and data size, and the demands for inter-node communication and computing resources also vary. Dynamically adjusting the ring topology hierarchy can better adapt to task requirements, improve resource utilization, and improve task execution efficiency. For example, for complex training tasks with large data sizes, increasing the number of ring topology layers can reduce the number of nodes on a single ring, reducing the communication load. The complexity of the training task is evaluated based on indicators such as the number of neural network layers, number of parameters, and data sample size, and the data size is determined using data scale analysis tools. When the task complexity is high and the data scale is large, some nodes are selected from the available node set to build a new ring, and the new ring is connected to the original ring via high-speed links to form a multi-layer ring topology. By adjusting the distribution of GPU nodes on different rings and link parameters, the allocation of communication and computing resources is optimized. This allows the ring topology to be dynamically adjusted according to the actual needs of the training task, improving adaptability to different training tasks and enhancing cluster resource utilization efficiency. The adjusted ring topology will be used as the object of simulation task execution and further optimization. The rationality of its adjustment affects the effectiveness of subsequent optimization and the generation of the final topology configuration instructions.

[0098] The load of inference tasks will vary with factors such as the amount of input data and model complexity. Dynamically adjusting the degree of parallelism can enable the GPU cluster to maintain efficient operation under different loads, improving resource utilization and inference speed; deploying load monitoring tools to monitor the computational load of inference tasks in real time, such as the number of inferences per second and computing resource occupancy rate. When the computational load increases, appropriate nodes are selected from the available node set to join the existing parallel group, increasing the degree of parallelism, and redistributing tasks to each node through the load balancing algorithm to ensure load balance between nodes. When the load decreases, the number of nodes in the parallel group is appropriately reduced to release resources for other tasks; thereby improving the execution efficiency of inference tasks and the effective utilization of cluster resources. The adjusted parallel topology will be used to simulate task execution and topology optimization. The rationality of its parallelism affects the simulation results and the direction of topology optimization, and thus affects the generation of topology configuration instructions.

[0099] Specifically, the output logic of the resource reservation pool includes:

[0100] Sort tasks by priority based on their importance, urgency, and resource requirements, and select high-priority tasks;

[0101] Predict the resource requirements of high-priority tasks based on their resource requirements and historical execution data;

[0102] Perform resource evaluation on the GPU nodes in the available node set to obtain the resource status of the GPU nodes. Based on the resource requirements of high-priority tasks and the resource status of the GPU nodes, select GPU nodes to form a resource reservation pool.

[0103] Reserve a spare node for each resource reservation pool, monitor the execution of high-priority tasks and the resource status of GPU nodes in the resource reservation pool in real time, and adjust the allocation of the resource reservation pool.

[0104] Different tasks have different importance, urgency, and resource requirements in the business. High-priority tasks are screened out through sorting, and their resource supply is prioritized to ensure the smooth execution of key tasks and improve the reliability and efficiency of the overall business. For example, online reasoning tasks with high real-time requirements or training tasks of important businesses should be given priority in obtaining resources. Weights are set for the importance, urgency, and resource requirements of tasks, and the weights are determined through statistical analysis of historical data. When a task is submitted, the priority score of each task is calculated based on the task description and related metadata, and the tasks are sorted in descending order according to the priority score. Tasks with scores higher than the set threshold are screened out as high-priority tasks. This clarifies which tasks have higher priorities and provides a basis for subsequent resource reservation.

[0105] Accurately predicting the resource requirements of high-priority tasks helps to rationally construct a resource reservation pool and avoid resource waste or shortage. Historical execution data can reflect the resource usage patterns of tasks under similar circumstances, and combined with the resource requirements of current tasks, more accurate predictions can be made. The resource usage of high-priority tasks, such as computing power, video memory, and memory usage, is recorded in the database. The historical execution data and the resource requirement characteristics of the current task (such as task type and data size) are used as input through a time series analysis algorithm to train the model to predict the resource requirements of high-priority tasks. This results in a more accurate prediction of the resource requirements of high-priority tasks, providing a quantitative basis for the construction of the resource reservation pool.

[0106] To build an effective resource reservation pool, it is necessary to understand the resource status of the GPU nodes in the available node set and match it with the resource requirements of high-priority tasks to ensure that the resource reservation pool can both meet task requirements and make reasonable use of resources; obtain the resource status of the GPU nodes in the available node set in real time, including the evaluation set results and the corresponding computing power, video memory, temperature and energy consumption, etc., match them according to the resource requirements of high-priority tasks and the resource status of the GPU nodes, and give priority to selecting GPU nodes with sufficient resources and stable performance to form the resource reservation pool. For example, for tasks that require high computing power and large video memory, select GPU nodes whose corresponding resource indicators meet the requirements; thus, the resource reservation pool is successfully constructed, and suitable GPU node resources are reserved for high-priority tasks. The construction results of the resource reservation pool affect the subsequent reservation of standby nodes and resource allocation adjustments. Its rationality and effectiveness are directly related to the execution guarantee of high-priority tasks.

[0107] During the execution of high-priority tasks, GPU nodes may fail or resource requirements may change. Reserving spare nodes can improve the fault tolerance of the system. Real-time monitoring and adjustment of resource reservation pool allocation can better adapt to changes in task requirements and ensure smooth task execution. Based on the size of the resource reservation pool and the importance of the task, GPU nodes with similar performance are selected as spare nodes in a certain proportion (such as 10%-20%). The task execution progress and resource status of the GPU nodes are obtained in real time. When a GPU node fails or has insufficient resources during task execution, the spare node is automatically added to the resource reservation pool and the task is reallocated. When the task is completed or resource requirements decrease, the resource reservation pool is adjusted in time to release excess resources. This improves the fault tolerance and flexibility of the resource reservation pool, enables it to better cope with various situations during task execution, and ensures the stable execution of high-priority tasks. The adjusted resource reservation pool status provides the target scheduling module with the latest resource information, affecting the generation and execution of task scheduling plans.

[0108] The elastic sharding module is used to split the training task into multiple subtasks based on the task dependencies and resource reservation pool of the historical sharding task graph, generate a sharding task graph, assign priorities to inference tasks, and generate resource preemption logic for inference tasks.

[0109] Specifically, the generation logic of the sharding task graph includes:

[0110] Obtain the historical sharding task graph of the GPU cluster, analyze the historical sharding task graph of the GPU cluster through graph clustering, and obtain task dependencies;

[0111] Split the training task into multiple subtasks based on task dependencies;

[0112] Evaluate the degree of adaptation of each subtask to each GPU node in the resource reservation pool to obtain the resource adaptation result of the subtask;

[0113] Based on the results of subtask splitting and resource adaptation, a sharding task graph is generated. The nodes of the sharding task graph represent subtasks, and the edges of the sharding task graph represent the dependencies between subtasks.

[0114] The historical sharding task graph contains rich task execution information. Different tasks have relationships such as sequence and data dependency. Graph clustering analysis of historical data can uncover these potential task dependencies, providing a reliable basis for subsequent training task splitting and ensuring that the split subtasks can be executed in a reasonable order and logic. The historical sharding task graph data is extracted from the GPU cluster's task management system. This data is typically stored in the form of a graph, with nodes representing tasks and edges representing dependencies between tasks. A spectral clustering algorithm is selected. First, feature extraction is performed on the historical sharding task graph, converting the graph's structural information into vector representations. These vectors are then clustered using the spectral clustering algorithm, grouping tasks with similar dependencies into one category. The task relationships within each cluster are analyzed, and a graph traversal algorithm (such as depth-first search) is used to determine the sequence and data dependency between tasks. This results in a collection of task dependency relationships. This successfully captures the dependencies between tasks, providing a foundation for the reasonable splitting of training tasks and helping to improve the efficiency and accuracy of task execution. Task dependencies are an important basis for splitting training tasks into subtasks, directly influencing the division and order of subtasks.

[0115] Large model training tasks usually have high complexity and computational complexity. Splitting them into several subtasks can better utilize the parallel computing capabilities of the GPU cluster and improve training efficiency. At the same time, splitting according to task dependencies can ensure the correct execution order between subtasks and avoid data inconsistency or calculation errors. According to task dependencies, the training tasks are divided according to functions and computational steps. For example, for the training tasks of deep learning models, they are split into subtasks such as data preprocessing, model forward propagation, loss calculation and backpropagation to ensure that the split subtasks have relative independence, that is, each subtask can be executed independently to a certain extent, while ensuring that the dependencies between subtasks are satisfied, and assigning a unique identifier to each subtask, and recording the dependencies between subtasks to form a subtask dependency graph. In this way, complex training tasks are split into multiple manageable subtasks, which facilitates parallel scheduling and execution on the GPU cluster and improves training efficiency. The task splitting results are the basis for evaluating the degree of adaptability between subtasks and GPU nodes. Different subtasks have different resource requirements, which will affect subsequent resource allocation and scheduling.

[0116] Different subtasks have different resource requirements for GPU nodes (such as computing power, video memory, and bandwidth), and the GPU nodes in the resource reservation pool also have different hardware characteristics and resource status. Evaluating the degree of compatibility between subtasks and GPU nodes can select the most suitable execution node for each subtask, improve resource utilization and task execution efficiency; analyze the resource requirements of each subtask, including computing power, video memory usage, and data transmission volume, etc., which can be estimated through historical task data and task feature information, and obtain the resource status of each GPU node in the resource reservation pool, including the evaluation set results and the corresponding computing power, video memory, Temperature and energy consumption, etc., the resource requirements of the subtask and the resource status of the GPU node are used as input through the TOPSIS method, the fitness score of each subtask and each GPU node is calculated, and then the GPU nodes are sorted according to the fitness score to obtain the resource adaptation result of the subtask; thus, the fitness degree information of each subtask and the GPU node in the resource reservation pool is obtained, which provides a basis for the reasonable allocation of resources and the generation of efficient sharding task graphs. The resource adaptation result of the subtask determines the connection method between nodes (subtasks) and edges (resource allocation relationships) in the sharding task graph, which directly affects the generation of the sharding task graph.

[0117] The sharding task graph is an intuitive representation of the splitting and resource allocation of training tasks. It can clearly show the dependencies and resource allocation between subtasks, providing clear guidance for subsequent task scheduling and execution. According to the results of the subtask splitting, each subtask is added to the graph as a node. According to the task dependencies, edges are added to the graph to indicate the sequence and data dependencies between subtasks. According to the resource adaptation results of the subtasks, the GPU node corresponding to each subtask is marked in the graph, or edges are added to indicate the resource allocation relationship between subtasks and GPU nodes. The sharding task graph is visualized using a graph visualization tool for easy analysis and understanding. This generates a complete sharding task graph, which clearly shows the splitting of the training task and the dependencies and resource allocation between subtasks, providing important input for the target scheduling module. The sharding task graph is the basis for the target scheduling module to generate a scheduling plan. The scheduling plan will determine the deployment location and execution order of the tasks based on the information in the sharding task graph.

[0118] Specifically, the resource preemption logic includes:

[0119] Dynamically adjust the priority of inference tasks based on the resource status of the GPU cluster and the execution status of the task;

[0120] Monitor the resource status of each GPU node in the GPU cluster in real time, detect resource conflicts through the threshold comparison method of the sliding window algorithm, and trigger resource conflict warnings;

[0121] Resource preemption is performed based on the priority of the inference task and the degree of resource conflict, while the priority of the subtasks of the training task and the impact of task execution are comprehensively analyzed.

[0122] The resource status and task execution status of the GPU cluster change dynamically. Different inference tasks have different resource requirements and importance at different times. Dynamically adjusting the priority of inference tasks can better adapt to these changes, ensure that resources are preferentially allocated to more important and urgent tasks, and improve the performance and efficiency of the entire system. The resource status of the GPU cluster is monitored in real time, including the evaluation set results and the corresponding computing power, video memory, temperature, and bandwidth. The execution status of each inference task is tracked, including the task start time, completion progress, and remaining time. Based on the resource status of the GPU cluster and the task execution status, the priority of the inference task is dynamically adjusted taking into account factors such as the importance, urgency, and resource requirements of the task. For example, machine learning algorithms are used to achieve dynamic priority adjustment and update the priority information of each inference task. This allows the priority of inference tasks to be dynamically adjusted according to actual conditions, making resource allocation more reasonable and improving the flexibility and adaptability of the system. The priority of inference tasks is an important basis for triggering resource conflict warnings and executing resource preemption. The dynamic adjustment of priority will affect the detection and handling of subsequent resource conflicts.

[0123] In a GPU cluster, multiple tasks compete for limited resources simultaneously. When resource demand exceeds resource supply, resource conflicts occur. Real-time monitoring of the CPU cluster's resource status and timely detection of resource conflicts can proactively identify potential problems and prevent performance degradation or task failure due to insufficient resources. The system obtains the resource status of each GPU node in real time, including evaluation set results and corresponding computing power, video memory, temperature, and energy consumption. A sliding window algorithm is used to divide resource usage data over a period of time into multiple windows. For each window, resource usage statistics (such as average and maximum values) are calculated, and a threshold is set for each resource metric. When the resource usage statistics within a window exceed the threshold, a resource conflict is determined to have occurred. When a resource conflict is detected, a resource conflict warning is triggered, notifying the system administrator or task management system for processing. This allows for timely detection of resource conflicts in the GPU cluster, providing a time window for subsequent resource preemption and scheduling adjustments, helping to ensure the normal execution of tasks. The resource conflict warning is the trigger condition for executing resource preemption. Only when a resource conflict is detected will resource preemption be performed based on the priority and conflict level of the inference task.

[0124] When resource conflicts occur, measures need to be taken to solve the problem of resource shortage. By executing hierarchical resource preemption, some resources can be released in a targeted manner according to the task priority and the degree of resource conflict to meet the needs of high-priority tasks. At the same time, the priority of the subtasks of the training task and the impact of task execution can be considered to avoid errors or performance degradation in the training task due to resource preemption. According to the priority of the inference task and the degree of resource conflict, resource preemption is divided into different levels. For example, for mild resource conflicts, you can try to suspend low-priority inference tasks first, and for severe resource conflicts, you can consider releasing some resources occupied by subtasks of the training task. When executing resource preemption, low-priority tasks are given priority for resource release. For subtasks of the training task, it is necessary to evaluate Estimate the impact of resource preemption on task execution, and try to select subtasks with less impact on tasks for resource release. During the resource preemption process, record the task information and resource release status of the preempted resources for subsequent resource recovery and task scheduling adjustments. When the resource conflict is alleviated, gradually restore the preempted tasks according to the task priority and resource status; this can effectively solve the resource conflict problem in the GPU cluster, ensure the normal execution of high-priority tasks, and minimize the impact on training tasks, thereby improving the overall performance and stability of the system. Resource preemption operations will change the resource allocation status and task execution status of the GPU cluster. Relevant information needs to be updated in a timely manner to provide the target scheduling module with the latest resources and task status for subsequent scheduling plan adjustments.

[0125] The target scheduling module is used to generate a scheduling plan through reinforcement learning based on the sharding task graph, resource preemption logic, and resource reservation pool. The scheduling plan includes the task deployment location and the sharding task preemption order, and the scheduling plan is selected through a fuzzy comprehensive evaluation method based on the resource status of the GPU node.

[0126] Specifically, the scheduling plan generation logic includes:

[0127] Integrate and standardize the sharding task graph, resource preemption logic, resource reservation pool, and resource status of each GPU node;

[0128] Build a reinforcement learning model that uses the resource status of each GPU node, resource preemption logic, resource reservation pool, and nodes and edges of the shard task graph as the state space, the task deployment location and shard task preemption order as the action space, and task priority, resource utilization, and task completion time as the reward function;

[0129] The reinforcement learning model is trained through the deep Q network to generate a scheduling plan, which includes the task deployment location and the preemption order of sharded tasks.

[0130] The sharding task graph, resource preemption logic, resource reservation pool, and resource status of GPU nodes are key information for generating scheduling plans, but the format and scope of this information are different. Integrating and standardizing this information can unify this information into a processable framework, facilitating the use of subsequent reinforcement learning models and ensuring that the model can accurately learn and understand the relationship between this information. The sharding task graph is obtained from the elastic sharding module, which contains subtasks and their task dependencies. The resource preemption logic is obtained from the elastic sharding module, which specifies the processing strategy in the event of resource conflicts. The resource reservation pool information is obtained from the topology generation module, including reserved GPU nodes and resource allocation status. The real-time resource status of each GPU node, such as the evaluation set results and the corresponding computing power, video memory, temperature, and energy consumption, is obtained in real time.

[0131] Integrate the above information into a data structure, such as a multidimensional array or dictionary. For the sharded task graph, convert it into an adjacency matrix or list form. The resource preemption logic is represented by a rule list, the resource reservation pool information is represented by a node list and a resource allocation matrix, and the GPU node resource status is represented by a vector. Standardize the integrated data to eliminate the dimensional differences between different data features. For numerical data, such as computing power and video memory usage, use the Z-score normalization method to convert the data into a distribution with a mean of 0 and a standard deviation of 1. For categorical data, such as task type, use one-hot encoding for processing. Thus, data with a unified format and range is obtained, which provides high-quality input for the reinforcement learning model and helps the model learn and generate scheduling plans more accurately. The standardized data is the basis for constructing the state space of the reinforcement learning model. Its quality and accuracy directly affect the training effect of the model and the quality of the scheduling plan generation.

[0132] Reinforcement learning is a method suitable for solving complex scheduling problems. By abstracting the problem into a state space, an action space, and a reward function, the reinforcement learning model can learn the optimal scheduling strategy through continuous trial and error. The state space contains the current state information of the system, the action space defines the actions that the model can take, and the reward function is used to evaluate the pros and cons of each action, guiding the reinforcement learning model to learn towards the optimal solution. The standardized resource status of each GPU node, resource preemption logic, resource reservation pool, and node and edge information of the sharded task graph are combined into a high-dimensional vector as the state representation of the reinforcement learning model. For example, the resource state vector of each GPU node, the node allocation list of the resource reservation pool, and the adjacency matrix of the sharded task graph are spliced into a large vector.

[0133] The task deployment location is represented by a vector, where each element represents the GPU node number to which a subtask is assigned. The sharded task preemption order is represented by a permutation vector, where each element represents the preemption priority of the subtask. These two vectors are combined into an action vector, which serves as the action representation of the reinforcement learning model. Task priority is quantified based on its importance and urgency, resource utilization is measured by calculating the average resource utilization of all GPU nodes, and task completion time is estimated by simulating the task execution process. These three indicators are weighted and summed according to certain weights to obtain the value of the reward function. The deep Q network is then selected as the reinforcement learning model. It is a neural network-based reinforcement learning algorithm that can handle high-dimensional state and action spaces. A multilayer perceptron is constructed as the network structure of the deep Q network. The number of neurons in the input layer is the same as the dimension of the state space, and the number of neurons in the output layer is the same as the dimension of the action space. This constructs a complete reinforcement learning model that can transform the scheduling problem into a learnable problem and find the optimal scheduling solution by learning the reward function. The quality and performance of the reinforcement learning model directly affect the generation of the scheduling solution. The trained model will be used to generate specific scheduling solutions.

[0134] Deep Q network is an effective reinforcement learning algorithm. It learns the optimal action strategy by continuously interacting with the environment. By training the deep Q network model, it can select the optimal task deployment location and sharding task preemption order under a given state, thereby generating an efficient scheduling solution. During the training process, the experience replay mechanism is used to store and sample the historical state-action-reward-next state quadruple. Each time, a batch of data is randomly sampled from the experience replay buffer for training to reduce the correlation between data and improve the stability of training. In order to improve the stability of training, the target network is used to calculate the target Q value. The structure of the target network is the same as that of the main network, but the parameter update frequency is lower. The parameters of the main network are copied to the target network every certain number of steps to initialize the main network. The parameters of the network and the target network are set, and hyperparameters such as the learning rate and discount factor are set. In each training step, the reinforcement learning model selects an action based on the current state, executes the action and obtains the reward and the next state, stores the quadruple in the experience replay buffer, and then samples a batch of data from the buffer, calculates the loss between the target Q value and the current Q value, and uses the gradient descent algorithm to update the parameters of the main network. When the model training is completed, for a given current state, the model selects the action with the largest Q value as the optimal action. The task deployment location and shard task preemption order corresponding to this action are the generated scheduling plan; by training the deep Q network model, a reinforcement learning model that can generate the optimal scheduling plan according to the system state is obtained. The generated scheduling plan can improve the execution efficiency of tasks and resource utilization.

[0135] Specifically, the scheduling scheme selection logic includes:

[0136] Determine multiple selection goals, including task completion time, resource utilization, and energy consumption, and determine the weight of each selection goal;

[0137] According to the selection objectives and the weight of each selection objective, the scheduling scheme is evaluated by weighted summation to obtain the score of each scheduling scheme;

[0138] The fuzzy comprehensive evaluation method is used to determine the pros and cons of each scheduling scheme based on its score and the resource status of the GPU node, and a scheduling scheme is selected based on the pros and cons of each scheduling scheme.

[0139] During the execution of the scheduling plan, the operating status of the GPU cluster and the resource status of each GPU node are monitored in real time to dynamically adjust the scheduling plan.

[0140] Different scheduling schemes differ in terms of task completion time, resource utilization and energy consumption. A single goal cannot fully evaluate the pros and cons of a scheduling scheme. By determining multiple selection goals and assigning weights to them, these factors can be comprehensively considered to select a scheduling scheme that best meets system requirements. Task completion time, resource utilization and energy consumption are determined as selection goals. Task completion time reflects the efficiency of the scheduling scheme, resource utilization measures the efficiency of system resource use, and energy consumption is related to the system's operating cost and environmental performance. The weight of each selection goal is determined by a multi-objective optimization algorithm. For example, for time-sensitive tasks, the weight of task completion time is appropriately increased, while for resource-constrained systems, the weight of resource utilization is increased. These weights are represented as a vector, and the sum of the weights is 1. This clarifies the multiple goals and weights for evaluating scheduling schemes, providing a standard for subsequent scheme evaluation. Selecting goals and weights is the basis for evaluating the scoring of scheduling schemes, and directly affects the calculation results of the scoring and the selection of schemes.

[0141] The evaluation results of multiple selection targets can be combined into a single score through weighted summation, which facilitates the comparison and ranking of different scheduling schemes. For each scheduling scheme, its task completion time, resource utilization and energy consumption indicators are calculated separately. The task completion time is estimated by simulating the task execution process, the resource utilization is measured by calculating the average resource utilization of all GPU nodes, and the energy consumption is calculated based on the power consumption and usage time of the GPU nodes. The task completion time, resource utilization and energy consumption indicators of each scheduling scheme are multiplied by the corresponding weights, and then the sum is calculated to obtain the score of the scheme. Thus, a comprehensive score of each scheduling scheme is obtained, which provides a quantitative basis for the subsequent scheme selection. The score of the scheduling scheme is the input of the fuzzy comprehensive evaluation method, which is used to determine the pros and cons of the scheme.

[0142] The scoring of the scheduling scheme is a quantitative indicator, but in actual selection, fuzzy factors such as the resource status of the GPU node need to be considered. The fuzzy comprehensive evaluation method can combine the quantitative scoring and fuzzy resource status information to more comprehensively evaluate the pros and cons of the scheduling scheme, thereby selecting the most appropriate scheme; determine a fuzzy evaluation set , Indicates the pros and cons of the scheduling scheme. For each scheduling scheme score and the resource status of the GPU node, determine its membership to each evaluation level. For example, use a trapezoidal membership function to calculate the membership, and combine the score of each scheduling scheme and the membership of the resource status of the GPU node to the evaluation level into a fuzzy relationship matrix. Then, perform a synthesis operation on the weight vector and the fuzzy relationship matrix to obtain a comprehensive membership vector of each scheduling scheme to the evaluation level, wherein the synthesis operation includes a large-small synthesis method. Finally, according to the comprehensive membership vector, select the scheduling scheme corresponding to the evaluation level with the largest membership as the optimal scheme; thereby comprehensively considering the score of the scheduling scheme and the resource status of the GPU node, more comprehensively evaluating the pros and cons of the scheduling scheme, and selecting the most appropriate scheduling scheme. The selected scheduling scheme will be executed. During the execution process, the system status needs to be monitored in real time for dynamic adjustment.

[0143] The operating status of the GPU cluster and the resource status of the GPU nodes are dynamic. During the execution of the scheduling plan, problems such as resource conflicts and task delays may arise. Real-time monitoring of these states and dynamic adjustment of the scheduling plan can ensure that the system always maintains efficient operation and improves task execution efficiency and resource utilization. The operating status of the GPU cluster and the resource status of each GPU node are monitored in real time, including the evaluation set results and the corresponding computing power, video memory, temperature and energy consumption. Resource usage thresholds and task execution time thresholds are set. When the monitored resource usage or task execution time is greater than the corresponding threshold, it is determined to be an abnormal situation. When an abnormal situation occurs, the scheduling plan generation and selection logic is re-called, a new scheduling plan is generated based on the current system state, and the task deployment location and shard task preemption order are updated. This can promptly detect and handle abnormal situations during the execution of the scheduling plan, ensure the stability and efficiency of the system, and improve the quality of task completion. The dynamically adjusted scheduling plan will continue to guide the execution of tasks, ensuring that the system can maintain the optimal scheduling strategy under different operating conditions.

[0144] Example 2

[0145] like Figure 4 As shown, a flowchart of a method for GPU heterogeneous cluster scheduling for large model training and inference is provided for an embodiment of the present application. The method includes:

[0146] Obtain the hardware characteristics and operating status of the GPU cluster, and use clustering algorithms to analyze historical task data to determine the memory access pattern and instruction set matching degree;

[0147] The scheduling weight factor of each GPU node is calculated by weighted summation, and the resource status of each GPU node is determined in combination with the operating status of the GPU cluster to generate a resource profile library;

[0148] Filter GPU nodes based on their resource status and task type, generate hardware connection topology based on task type, and output topology configuration instructions and resource reservation pool;

[0149] Based on the task dependencies and resource reservation pool of the historical sharding task graph, the training task is split into several subtasks to generate a sharding task graph. At the same time, priorities are assigned to inference tasks and resource preemption logic for inference tasks is generated.

[0150] Generate a scheduling plan based on the sharded task graph, resource preemption logic, and resource reservation pool through reinforcement learning. The scheduling plan includes the task deployment location and the sharded task preemption order.

[0151] The scheduling scheme is selected by fuzzy comprehensive evaluation method based on the resource status of GPU nodes.

[0152] Since the principle of solving the problem by the method in the embodiment of the present application is similar to that of the system described above in the embodiment of the present application, the implementation of the method refers to the implementation of the system, and the repeated parts will not be repeated.

Claims

1. A GPU heterogeneous cluster scheduling system for large model training and inference, characterized by: include: Heterogeneous profiling module, topology generation module, elastic sharding module, and target scheduling module; The heterogeneous profiling module is used to obtain the hardware characteristics and operating status of the GPU cluster. It combines historical task data with clustering algorithm analysis to determine the memory access pattern and instruction set matching degree. It also calculates the scheduling weight factor of each GPU node through weighted summation. It then determines the resource status of each GPU node based on the operating status of the GPU cluster and generates a resource profiling library. The topology generation module is used to filter GPU nodes according to their resource status and task type to form an available node set, generate a hardware connection topology by task type, and output topology configuration instructions based on the hardware connection topology. It also outputs a resource reservation pool for the filtered high-priority tasks. The elastic sharding module is used to split the training task into multiple subtasks based on the task dependencies and resource reservation pool of the historical sharding task graph, generate a sharding task graph, assign priorities to inference tasks, and generate resource preemption logic for inference tasks. The target scheduling module is used to generate a scheduling plan through reinforcement learning based on the sharding task graph, resource preemption logic and resource reservation pool. The scheduling plan includes the task deployment location and the sharding task preemption order, and the scheduling plan is selected through the fuzzy comprehensive evaluation method in combination with the resource status of the GPU node.

2. The GPU heterogeneous cluster scheduling system for large model training and inference according to claim 1, characterized in that: The logic for determining the resource status of each GPU node includes: Obtain the hardware characteristics and operating status of the GPU cluster. The hardware characteristics of the GPU cluster include peak computing power, memory bandwidth, and NVLink connection topology. The operating status of the GPU cluster includes temperature, energy consumption, and failure rate. Extract task types, task runtimes, and GPU resource usage from historical task data, and analyze them using a clustering algorithm to determine memory access patterns and instruction set matching. The weights of the memory access pattern and instruction set matching are determined by the hierarchical analysis method, and the scheduling weight factor of each GPU node is calculated by weighted summation. The resource status of each GPU node is determined by the fuzzy comprehensive evaluation method based on the scheduling weight factor of each GPU node and the operating status of the GPU cluster.

3. The GPU heterogeneous cluster scheduling system for large model training and inference according to claim 2, characterized in that: The sub-logic for determining the matching degree between the video memory access mode and the instruction set includes: Clustering algorithms are used to cluster hardware features, operating status, and historical task data to form clusters. By counting the proportion of access times of different video memory address spaces by tasks in the cluster, the video memory access pattern of the cluster is determined; The instruction set matching degree is determined by calculating the ratio of the time that the tasks in the cluster use the instruction set to the task running time.

4. The GPU heterogeneous cluster scheduling system for large model training and inference according to claim 3, characterized in that: The output logic of the topology configuration instruction includes: Monitor the resource status of all GPU nodes, filter GPU nodes based on resource status and task type, and form a set of available nodes; Distinguish task types and generate hardware connection topology based on task types; Simulate the execution process of tasks under the hardware connection topology, monitor the communication load, computing load and resource utilization between GPU nodes, and iteratively optimize the hardware connection topology through the simulated annealing algorithm; Based on the iteratively optimized hardware connection topology, it outputs topology configuration instructions, which include NVLink reconnection strategies and cross-node RDMA path planning.

5. The GPU heterogeneous cluster scheduling system for large model training and inference according to claim 4, characterized in that: The generation sub-logic of the hardware connection topology includes: Task types include training tasks and inference tasks. Hardware connection topology is generated based on the task type. A ring topology is generated for training tasks, and a parallel topology is generated for inference tasks. Dynamically adjust the level of the ring topology based on the complexity of the training task and the size of the data, combined with the available node set; The parallelism of the parallel topology is dynamically adjusted based on the load of the inference task and the available node set.

6. The GPU heterogeneous cluster scheduling system for large model training and inference according to claim 5, characterized in that: The output logic of the resource reservation pool includes: Sort tasks by priority based on their importance, urgency, and resource requirements, and select high-priority tasks; Predict the resource requirements of high-priority tasks based on their resource requirements and historical execution data; Perform resource evaluation on the GPU nodes in the available node set to obtain the resource status of the GPU nodes. Based on the resource requirements of high-priority tasks and the resource status of the GPU nodes, select GPU nodes to form a resource reservation pool. Reserve a spare node for each resource reservation pool, monitor the execution of high-priority tasks and the resource status of GPU nodes in the resource reservation pool in real time, and adjust the allocation of the resource reservation pool.

7. The GPU heterogeneous cluster scheduling system for large model training and inference according to claim 6, characterized in that: The generation logic of the sharding task graph includes: Obtain the historical sharding task graph of the GPU cluster, analyze the historical sharding task graph of the GPU cluster through graph clustering, and obtain task dependencies; Split the training task into multiple subtasks based on task dependencies; Evaluate the degree of adaptation of each subtask to each GPU node in the resource reservation pool to obtain the resource adaptation result of the subtask; Based on the results of subtask splitting and resource adaptation, a sharding task graph is generated. The nodes of the sharding task graph represent subtasks, and the edges of the sharding task graph represent the dependencies between subtasks.

8. The GPU heterogeneous cluster scheduling system for large model training and inference according to claim 7, characterized in that: The resource preemption logic includes: Dynamically adjust the priority of inference tasks based on the resource status of the GPU cluster and the execution status of the task; Monitor the resource status of each GPU node in the GPU cluster in real time, detect resource conflicts through the threshold comparison method of the sliding window algorithm, and trigger resource conflict warnings; Resource preemption is performed based on the priority of the inference task and the degree of resource conflict, while the priority of the subtasks of the training task and the impact of task execution are comprehensively analyzed.

9. The GPU heterogeneous cluster scheduling system for large model training and inference according to claim 8, characterized in that: The generation logic of the scheduling scheme includes: Integrate and standardize the sharding task graph, resource preemption logic, resource reservation pool, and resource status of each GPU node; Build a reinforcement learning model that uses the resource status of each GPU node, resource preemption logic, resource reservation pool, and nodes and edges of the shard task graph as the state space, the task deployment location and shard task preemption order as the action space, and task priority, resource utilization, and task completion time as the reward function; The reinforcement learning model is trained through the deep Q network to generate a scheduling plan, which includes the task deployment location and the preemption order of sharded tasks.

10. A GPU heterogeneous cluster scheduling method for large model training and inference, implemented based on the GPU heterogeneous cluster scheduling system for large model training and inference according to any one of claims 1 to 9, characterized in that: include: Obtain the hardware characteristics and operating status of the GPU cluster, and use clustering algorithms to analyze historical task data to determine the memory access pattern and instruction set matching degree; The scheduling weight factor of each GPU node is calculated by weighted summation, and the resource status of each GPU node is determined in combination with the operating status of the GPU cluster to generate a resource profile library; Filter GPU nodes based on their resource status and task type, generate hardware connection topology based on task type, and output topology configuration instructions and resource reservation pool; Based on the task dependencies and resource reservation pool of the historical sharding task graph, the training task is split into multiple subtasks to generate a sharding task graph. At the same time, priorities are assigned to inference tasks and resource preemption logic for inference tasks is generated. Generate a scheduling plan based on the sharded task graph, resource preemption logic, and resource reservation pool through reinforcement learning. The scheduling plan includes the task deployment location and the sharded task preemption order. The scheduling scheme is selected by fuzzy comprehensive evaluation method based on the resource status of GPU nodes.

Images