A GPU Cluster Scheduling Strategy Optimization System Based on Deep Reinforcement Learning

Abstract

This invention discloses a GPU cluster scheduling strategy optimization system based on deep reinforcement learning, comprising a simulation environment layer, a reinforcement learning layer, and a policy evaluation layer. The simulation environment layer provides a realistic and reliable GPU cluster scheduling environment, reproducing the core mechanism of actual cluster scheduling, and providing controllable experimental conditions to facilitate policy comparison, evaluation, and iterative optimization. The reinforcement learning layer transforms the GPU cluster scheduling problem into a Markov decision process based on deep neural networks and applies reinforcement learning strategies to obtain the optimal scheduling strategy. The policy evaluation layer statistically analyzes various key indicators, compares and analyzes multiple scheduling strategies, and visualizes the analysis results to optimize the GPU cluster scheduling strategy. This system solves the temporal shortsightedness problem of traditional schedulers, enabling scheduling decisions to consider future impacts, achieving global optimization rather than local optimization, and improving overall resource utilization efficiency.

Description

Technical Field

[0001] This invention relates to the field of cloud computing resource scheduling technology, and in particular to a GPU cluster scheduling strategy optimization system based on deep reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of deep learning in computer vision, natural language processing, and scientific computing, GPUs have become an indispensable computing resource in modern data centers. Compared with traditional CPUs, GPUs have significant advantages in parallel computing, making them the preferred hardware for training large-scale neural network models. However, the high cost of GPU resources and supply chain constraints make how to efficiently utilize these valuable resources a key technical challenge. In large-scale computing clusters, resource scheduling systems play a crucial role. Container orchestration platforms, represented by Kubernetes (k8s), have become the de facto standard for managing distributed computing resources. These platforms use scheduler components to allocate user-submitted computing tasks (called Pods in Kubernetes) to appropriate computing nodes for execution. The quality of scheduling decisions directly affects the resource utilization efficiency, task completion time, and system throughput of the entire cluster.

[0003] Currently, most mainstream cluster scheduling systems employ heuristic rule-based scheduling algorithms. The basic workflow of these algorithms is as follows: when a new task needs to be scheduled, the scheduler first filters out candidate nodes that meet the task's resource requirements, then calculates a score for each candidate node according to a predefined scoring function, and finally selects the node with the highest score as the scheduling target. Although heuristic algorithms are widely used in practice, they have some fundamental limitations, which are particularly prominent when facing modern AI workloads. Temporal dependencies and the lack of global optimization are the core problems. Traditional schedulers use a greedy decision-making mechanism, considering only the local optimum at the current moment during each scheduling operation, failing to assess the impact of the current decision on future scheduling opportunities. This short-sighted decision-making approach often leads to globally suboptimal results when faced with complex task sequences.

[0004] Reinforcement learning, as an important branch of machine learning, is particularly well-suited for solving sequential decision-making problems. In resource scheduling scenarios, reinforcement learning possesses several natural advantages. However, applying reinforcement learning to improve cluster scheduling strategies faces multiple technical challenges. The high dimensionality and dynamic nature of the state space are primary difficulties. The cluster state contains multidimensional resource information from hundreds or thousands of nodes and changes rapidly over time, making the design of an effective state representation a significant challenge. Sparse rewards and delayed feedback problems make learning difficult; the quality of scheduling decisions often cannot be evaluated until the task is completed, which can take hours or even days. Balancing exploration and exploitation is particularly challenging in scheduling scenarios. Overexploration can lead to a decline in service quality, while overexpansion can lead to local optima. The complexity of multi-objective optimization requires finding a balance among conflicting objectives such as resource utilization, task completion time, and fairness. Sample efficiency is also a significant concern; reinforcement learning typically requires numerous interactions to converge to a good policy, and each scheduling decision can impact system performance. These challenges make designing and training effective reinforcement learning scheduling strategies a research direction of considerable value. Summary of the Invention

[0005] The purpose of this invention is to provide a GPU cluster scheduling strategy optimization system based on deep reinforcement learning. This system solves the temporal shortsightedness problem of traditional schedulers, enabling scheduling decisions to consider the impact on the future, achieving global optimization rather than local optimization, while optimizing the coordinated utilization of multi-dimensional resources, reducing resource fragmentation, and improving overall resource utilization efficiency.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A GPU cluster scheduling policy optimization system based on deep reinforcement learning, the system comprising a simulation environment layer, a reinforcement learning layer, and a policy evaluation layer, wherein:

[0008] The simulation environment layer is used to provide a realistic and reliable GPU cluster scheduling environment, reproduce the core mechanisms of actual cluster scheduling, including resource constraint checks, multi-round scheduling processes, and task queue management, while providing controllable experimental conditions to facilitate the comparison, evaluation, and iterative optimization of strategies.

[0009] The reinforcement learning layer transforms the GPU cluster scheduling problem into a Markov decision process based on a deep neural network, and applies a reinforcement learning strategy to obtain the optimal scheduling strategy.

[0010] The strategy evaluation layer statistically analyzes various key indicators, compares and analyzes multiple scheduling strategies, and visualizes the analysis results to optimize the GPU cluster scheduling strategy. The key indicators include GPU utilization, task waiting time, scheduling success rate, resource balance, and number of scheduling rounds.

[0011] As can be seen from the technical solution provided by the present invention, the above system solves the problem of temporal shortsightedness of traditional schedulers, enabling scheduling decisions to consider the impact on the future, achieving global optimization rather than local optimization, while optimizing the collaborative utilization of multi-dimensional resources, reducing resource fragmentation, improving overall resource utilization efficiency, providing a theoretical basis and practical verification for the application of reinforcement learning in actual cluster scheduling, and promoting the development of intelligent scheduling technology. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic diagram of the structure of a GPU cluster scheduling strategy optimization system based on deep reinforcement learning provided in an embodiment of the present invention;

[0014] Figure 2 This is a schematic diagram of the Actor-Critic neural network architecture described in an embodiment of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments, and do not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0016] like Figure 1 The diagram shows the structure of a GPU cluster scheduling policy optimization system based on deep reinforcement learning provided in an embodiment of the present invention. The system includes a simulation environment layer, a reinforcement learning layer, and a policy evaluation layer, wherein:

[0017] The simulation environment layer is used to provide a realistic and reliable GPU cluster scheduling environment, reproduce the core mechanisms of actual cluster scheduling, including resource constraint checks, multi-round scheduling processes, and task queue management, while providing controllable experimental conditions to facilitate the comparison, evaluation, and iterative optimization of strategies.

[0018] The reinforcement learning layer transforms the GPU cluster scheduling problem into a Markov decision process based on a deep neural network, and applies a reinforcement learning strategy to obtain the optimal scheduling strategy.

[0019] The strategy evaluation layer statistically analyzes various key indicators, compares and analyzes multiple scheduling strategies, and visualizes the analysis results to optimize the GPU cluster scheduling strategy. The key indicators include GPU utilization, task waiting time, scheduling success rate, resource balance, and number of scheduling rounds.

[0020] like Figure 1 As shown, the simulation environment layer specifically includes:

[0021] The cluster status manager is used to maintain complete status information of the virtual cluster, including the resource configuration, current usage, and list of running tasks for each node.

[0022] The task generator is used to generate diverse task flows based on the configured load pattern, simulating the workload characteristics of a real production environment.

[0023] The scheduling execution engine is used to implement the core logic of the Kubernetes scheduler, including node filtering, scoring, and binding operations.

[0024] The environment interface adapter is used to encapsulate the GPU cluster scheduling process into a standard reinforcement learning environment interface, providing state observation, action execution, and reward feedback functions.

[0025] In the cluster state manager, each virtual node contains three types of resources: CPU (in terms of cores), memory (in terms of GB), and GPU (in terms of GPUs). The node status tracks the total amount, allocated amount, and remaining amount of each resource in real time. Task Pods are represented by a three-dimensional resource demand vector, supporting pure CPU tasks, single GPU tasks, and multi-GPU tasks. This abstraction retains the core complexity of actual scheduling while avoiding the computational burden caused by too many details.

[0026] The task generator includes three types of tasks: large memory tasks simulating data preprocessing and data extraction-transform-load (ETL) scenarios, characterized by requiring a large amount of memory but not using GPUs; regular GPU tasks simulating standard model training jobs, requiring a moderate amount of CPU, memory, and a single GPU; and large GPU tasks simulating distributed deep learning training, requiring multiple GPUs to work together. The task generator generates task sequences according to the configured proportions and arrival patterns, supporting various load modes such as uniform distribution, Poisson process, and burst mode.

[0027] In the aforementioned scheduling execution engine, a multi-round scheduling mechanism is employed to reflect the behavior of the Kubernetes scheduler. When the Kubernetes scheduler faces a batch of tasks to be scheduled, it does not complete all the scheduling at once, but processes them one by one using a round-robin approach. Specifically, in each round, the Kubernetes scheduler takes a task from the head of the queue, finds a suitable node for the task, and if there is no node that meets the requirements, the Kubernetes scheduler chooses to put the task back to the tail of the queue and retry it in a subsequent round. This mechanism avoids scheduling failures caused by temporary resource shortages and also introduces the possibility of optimizing the scheduling order.

[0028] When the scheduling execution engine performs a binding operation, it first checks whether the target node has sufficient resources. The binding operation is only performed if all resource dimensions are satisfied. After successful binding, the corresponding resources are marked as allocated, and the task is removed from the scheduling queue. In its implementation, the system also implements task completion and resource release logic, supporting dynamic resource reclamation and reallocation.

[0029] like Figure 1 As shown, the reinforcement learning layer specifically includes:

[0030] The core module of the PPO (Proximal Policy Optimization) algorithm includes two deep neural network components: a policy network and a value network. The policy network is responsible for learning the mapping relationship from state to action and outputting the probability distribution of selecting each scheduling action in a given cluster state. The value network is used to evaluate the long-term value of the state and provide a benchmark for policy improvement. The deep neural network used is an Actor-Critic neural network architecture.

[0031] The experience collection module is responsible for running the current strategy and interacting with the environment, collecting state transition and reward information;

[0032] The training optimization module is used to continuously improve the scheduling strategy by updating the parameters of the deep neural network through gradient descent using collected empirical data.

[0033] In its specific implementation, the core module of the PPO algorithm obtains the key information required for scheduling decisions through a designed state representation method. The state vector includes: the characteristics of the task currently to be scheduled, the resource status of all nodes, and global statistical information, wherein:

[0034] The features of the tasks to be scheduled are represented by normalized three-dimensional vectors, which correspond to the requirements of GPU, CPU and memory respectively. Normalization ensures that the resource requirements of different levels are on the same scale, which is beneficial to the learning of neural networks.

[0035] The resource status of all nodes uses an enhanced nine-dimensional feature vector, which includes not only the absolute and relative remaining amount of each resource, but also a binary schedulable flag, a resource imbalance metric, and the standard deviation of utilization. This rich feature design enables reinforcement learning agents to fully understand the node status and make more informed decisions.

[0036] The core module of the PPO algorithm also employs advantage function estimation, where the advantage function A(s,a) represents the degree of improvement of action a in state s relative to the average performance. Specifically, it uses generalized advantage estimation (GAE), which balances bias and variance by introducing a decay factor λ. The calculation process of GAE utilizes the evaluation of the value network and the actual reward obtained. For each time step in the trajectory, the temporal difference error (TD error) is calculated, representing the difference between the current value estimate and the actual value. The advantage estimate is obtained by accumulating future TD errors through exponential decay. The decay factor λ is used to control the decay rate; a larger λ depends more on the actual reward (low bias, high variance), while a smaller λ depends more on the evaluation of the value network (high bias, low variance).

[0037] The core module of the PPO algorithm also incorporates a pruning mechanism to avoid performance crashes caused by excessively large single updates by limiting the difference between the old and new strategies; specifically:

[0038] Define the policy ratio r = π_new(a|s) / π_old(a|s);

[0039] Where π_new(a|s) is the new policy function, representing the probability distribution of taking action a under state s; π_old(a|s) is the old policy function, representing the probability distribution of taking action a under state s.

[0040] Calculate two objective functions: L1 = r * A and L2 = clip(r, 1-ε, 1+ε) * A;

[0041] Where A is the advantage function, which represents the advantage of taking a certain action in a certain state compared to the average level, and is used to evaluate the relative value of the action; clip(r, 1-ε, 1+ε) is the cutoff function, which restricts the value of variable r to the interval [1-ε, 1+ε]; ε is the pruning parameter, which is used to limit the policy update magnitude;

[0042] The ultimate goal is to minimize (L1, L2) while ensuring that the update magnitude does not exceed ε.

[0043] Specifically, when the advantage is positive (taking a good action), the probability of that action is increased, but the increase does not exceed (1+ε) times; when the advantage is negative (taking a bad action), the probability of that action is decreased, but the decrease does not exceed (1-ε) times.

[0044] In the specific implementation, the reward function design in the experience collection module directly affects the optimization objective of reinforcement learning. This application designs a multi-level reward mechanism, comprehensively considering both immediate gains and long-term benefits. The collected reward information includes:

[0045] Instant rewards provide corresponding feedback for each scheduling action: a high positive reward is given for successfully scheduling a GPU task, reflecting the importance of GPU resource optimization; a medium reward is given for successfully scheduling a non-GPU task, encouraging overall scheduling efficiency; a small negative reward is given for choosing delayed scheduling, preventing excessive latency; and a large penalty is given for attempting invalid scheduling, guiding the avoidance of incorrect decisions.

[0046] The final reward, calculated at the end of a scheduling round, is used to comprehensively evaluate the scheduling effect: GPU allocation rate reward, reflecting the efficiency of GPU resource utilization; task success rate reward, ensuring that most tasks can be successfully scheduled and maintaining the basic functions of the system; scheduling efficiency reward, encouraging fast scheduling and reducing task waiting time; resource balancing reward, guiding the reasonable distribution of load among nodes and avoiding hotspots and resource fragmentation; and multi-round penalties, preventing the overuse of latency strategies and ensuring the timeliness of scheduling.

[0047] The transition function defines how the environment responds to scheduling actions. The collected state transition information includes: when the agent chooses to schedule a task to a node, it first verifies the resource constraints. If they are satisfied, it updates the node's resource state and removes the task; otherwise, it keeps the state unchanged and gives a negative reward. When the delay action is chosen, the current task is moved to the end of the queue, and the environment transitions to the state of processing the next task. This deterministic transition rule enables the agent to accurately predict the consequences of actions, which helps to learn effective policies.

[0048] The experience collection module employs a parallel environment design, running multiple simulated environment instances simultaneously. Each instance uses the current policy for scheduling simulation. In each environment, the agent interacts with the environment for a complete round: starting from the initial state, it obtains the current tasks to be scheduled and the cluster state; it samples actions based on the probability distribution output by the policy network; it executes the actions and obtains environmental feedback, including the next state, reward information, and whether the round has ended; it stores the transfer tuple (s, a, r, s', done) in the experience buffer, where s is the current state (containing cluster resource distribution and tasks to be scheduled), a is the executed action (scheduling decision), r is the immediate reward (reflecting scheduling quality), s' is the next state (the new environment state after executing the action), and done is the termination flag (indicating whether the round has ended); this process is repeated until all tasks are processed or the maximum number of steps is reached. The advantages of parallel collection are: increased experience diversity, reduced sample correlation; improved GPU utilization, fully utilizing hardware performance; shortened training time, and accelerated policy improvement.

[0049] In the specific implementation, a delayed scheduling mechanism is adopted in the training optimization module. For a cluster of N nodes, the action space contains N+1 discrete actions. The first N actions correspond to scheduling the current task to a specific node, and the N+1th action represents delayed scheduling, which puts the task back to the end of the queue. This design gives the scheduler the ability to choose the timing, and can strategically postpone some tasks when resources are scarce, waiting for a better scheduling opportunity.

[0050] The training optimization module uses mini-batch gradient descent to update the parameters of the deep neural network, randomly dividing the collected empirical data into mini-batches; calculating the policy loss and value loss for each mini-batch; updating the network parameters using the Adam optimizer; repeating this process for multiple rounds until the policy improvement saturates.

[0051] The key hyperparameters in the training process include: learning rate, which adopts a cosine annealing strategy and gradually decays from the initial value; batch size, which is dynamically adjusted according to GPU memory; PPO pruning parameter ε, set to 0.1-0.2; generalized advantage estimation (GAE) parameter λ, set to 0.95; and discount factor γ, set to 0.99, emphasizing long-term returns.

[0052] like Figure 1 As shown, the strategy evaluation layer specifically includes:

[0053] The performance metric calculation module is used to collect real-time statistics on various key indicators. Among them, GPU utilization is the most critical indicator, used to calculate the proportion of successfully allocated GPUs to the total number of GPUs required; scheduling success rate measures the proportion of successfully scheduled tasks to the total number of tasks; task waiting time is used to calculate the time interval from task submission to scheduling; resource balance is measured using the standard deviation of resource utilization of each node; the number of scheduling rounds reflects the number of rounds required to complete all scheduling, reflecting scheduling efficiency. In the specific implementation, statistical significance testing ensures the reliability of experimental results, and each algorithm is repeatedly run multiple times (usually more than 30 times) under the same configuration; the mean and standard deviation of each indicator are calculated; t-tests or Mann-Whitney U tests are used to compare different algorithms; and 95% confidence intervals and p-values ​​are reported.

[0054] The offline comparison module implements various scheduling strategies and performs fair comparisons with reinforcement learning strategies. These strategies include: the BinPack strategy, which calculates the resource utilization rate after scheduling tasks to nodes and selects the node with the highest resource utilization rate, aiming to minimize the number of active nodes; the Balanced strategy, which strives for balanced resource utilization, calculates the standard deviation of resource utilization for each node after scheduling, and selects the node with the smallest standard deviation; the LeastRequested strategy, which selects the node with the most remaining resources, calculates the remaining proportion for each resource type, and selects the node with the largest average; and the Random strategy, which randomly selects from all feasible nodes as a performance lower bound reference.

[0055] The visualization and analysis module provides intuitive charts to help researchers analyze the behavior and performance of various strategies. Among them, the training curve shows the changes in reward and loss metrics as training progresses; the resource heatmap shows the resource usage of each node at each time point; the scheduling decision distribution statistics show the frequency of each node being selected; and the task waiting time distribution reveals the scheduling latency characteristics.

[0056] The experiment management module supports the management of batch experiment configurations, result storage, and statistical analysis.

[0057] In specific implementation, such as Figure 2 The diagram shown is a schematic of the Actor-Critic neural network architecture described in an embodiment of the present invention. It illustrates the complete network structure from task features and node features as input, through the encoder, linear attention mechanism, feature fusion, to the final output action probability distribution and state value estimation, including the dimensions and activation functions of each layer. Specifically:

[0058] The input encoding layer employs a separate encoding design, with the task encoder and node encoder processing different types of input features respectively. This design has several advantages: it allows for the use of the most suitable network structure for different feature types; it facilitates the independent learning of feature importance; and it improves the interpretability of the model.

[0059] The task encoder uses a three-layer fully connected network to map low-dimensional task features to a high-dimensional representation space. The network structure is as follows: input layer (3D) → hidden layer 1 (16 units, ReLU activation) → hidden layer 2 (32 units, ReLU activation) → output layer (32-dimensional task embedding). Batch normalization is applied after each hidden layer to improve training stability.

[0060] The node encoder processes node features and also employs a three-layer structure: Input layer (9-dimensional) → Hidden layer 1 (32 units, ReLU activation) → Hidden layer 2 (64 units, ReLU activation) → Output layer (64-dimensional node embedding). Its large network capacity enables it to learn complex patterns of node states.

[0061] The decision output layer makes the final prediction based on the fused features. The feature fusion adopts a concatenation method, combining the task embedding (32 dimensions) and attention context (64 dimensions) into a 96-dimensional joint representation. This joint representation contains the current task's requirement information and the global resource status of the cluster.

[0062] The Actor network branch consists of: joint features (96 dimensions) → hidden layers (128 units, ReLU activation) → output layer (N+1 dimensional logits). The Actor output layer does not use an activation function, producing unnormalized action preference scores.

[0063] The Critic network branch consists of: joint features (96 dimensions) → hidden layer (128 units, ReLU activation) → output layer (1-dimensional state value). The Critic's output represents an estimate of the long-term cumulative reward for the current state.

[0064] The Actor and Critic network branches share the underlying encoding and attention layers, but use independent parameters in the final decision layer. This partially shared design leverages common feature learning while maintaining the independence of the Actor and Critic functions.

[0065] Furthermore, the Actor-Critic neural network architecture also includes an attention mechanism. Traditional fully connected layers cannot effectively handle variable-length inputs (the number of nodes may vary), while simple aggregation methods (such as summation and averaging) lose important information. The linear attention mechanism adopted in this application retains the selective aggregation capability of attention while reducing the computational complexity from O(N²) to O(N). The process of the attention mechanism is as follows:

[0066] First, a query vector with a dimension of 64 is generated based on the task embedding. Then, all node embeddings are converted into key vectors and value vectors, both with a dimension of 64. Next, the dot product of the query vector and each key vector is calculated to obtain the original attention score. Attention weights are obtained through softmax normalization to ensure that the sum of the weights is 1. Finally, the attention weights are used to perform a weighted summation of the value vectors to obtain the aggregated context vector.

[0067] This attention mechanism allows the network to selectively focus on the most relevant node information based on the characteristics of the current task. For example, for GPU-intensive tasks, the network may pay more attention to nodes with sufficient GPU resources; for memory-intensive tasks, it may pay more attention to nodes with abundant memory resources.

[0068] Furthermore, in the specific implementation, an action masking mechanism is introduced into the deep neural network used. Specifically, this involves performing feasibility checks based on node resource status and task requirements, generating a mask vector, and applying the generated mask vector to the output of the deep neural network to ensure that only feasible scheduling actions are selected.

[0069] Feasibility checks are performed in real time at each decision point. For the current task to be scheduled and each candidate node, three resource dimensions are checked: whether the number of GPUs is sufficient (remaining GPUs on the node ≥ task-required GPUs); whether the number of CPU cores is sufficient (remaining CPUs on the node ≥ task-required CPUs); and whether the memory capacity is sufficient (remaining memory on the node ≥ task-required memory). Only when all resource dimensions are satisfied is the corresponding scheduling action marked as feasible.

[0070] The mask vector is generated based on the feasibility check results. The mask vector is an N+1 dimensional binary vector, where the first N elements correspond to the scheduling actions of N nodes, and the last element corresponds to the delayed action. The position corresponding to the action is 1, and the position corresponding to the action that cannot be performed is 0. In particular, the delayed action is always feasible (the corresponding position is always 1), ensuring that the system still has a legal action when all nodes fail to meet the requirements.

[0071] Furthermore, soft masking can be applied to the output layer of an Actor-Critic neural network. Instead of simply setting the probability of inactive actions to 0 (hard masking) after the Actor network outputs its original logits, the probability of these actions is subtracted by a large number (e.g., 10^6). After softmax normalization, the probability of these actions is close to 0 but not exactly 0. The advantage of this soft masking method is that it maintains the continuity of the gradient, which is beneficial for training the neural network. At the same time, because the probability is extremely small, these actions are almost never selected during actual sampling.

[0072] The masking information is not only used to constrain action selection, but also serves as additional input information. In the node feature vector, it includes a binary feature indicating whether the current task is schedulable to that node. This allows the Actor-Critic neural network to learn feasibility patterns, such as identifying which node configurations are more likely to meet the needs of specific types of tasks.

[0073] In practice, action masking is a key technology to ensure the feasibility of scheduling decisions. In actual scheduling, not all nodes can meet the resource requirements of the task. Forcing scheduling to a node with insufficient resources will lead to failure. The action masking mechanism ensures that the reinforcement learning agent only considers feasible scheduling options.

[0074] It is worth noting that the contents not described in detail in the embodiments of the present invention belong to the prior art known to those skilled in the art.

[0075] In summary, the system described in the embodiments of the present invention has the following advantages:

[0076] 1. By modeling the scheduling process as a sequential decision problem, we can capture the temporal dependencies between scheduling actions. Each scheduling decision is no longer isolated, but a link in the entire decision sequence, and its value needs to be evaluated from the perspective of long-term cumulative benefits.

[0077] 2. Introducing the concept of delayed scheduling, giving the scheduler the ability to "wait". Traditional schedulers can only reject tasks or forcibly schedule them to suboptimal nodes when faced with insufficient resources. However, this application allows the scheduler to choose to temporarily delay some tasks and wait for a more suitable scheduling opportunity. This mechanism is similar to the decision-making process of human experts - experienced operators will strategically arrange the execution time of tasks based on load patterns and resource conditions.

[0078] 3. By designing a comprehensive state representation and reward mechanism, the reinforcement learning agent can understand and optimize multiple objectives. The state representation includes not only the current resource usage, but also information such as the balance of resource distribution and historical scheduling patterns. The reward function comprehensively considers multiple dimensions such as GPU utilization, task completion efficiency, and resource balance, guiding the agent to learn a balanced scheduling strategy.

[0079] 4. A high-fidelity simulation environment is constructed, enabling safe and efficient reinforcement learning training. The simulation environment accurately reproduces the core mechanisms of actual cluster scheduling, including resource constraint checks, multi-round scheduling processes, and task queue management, while providing controllable experimental conditions to facilitate policy comparison, evaluation, and iterative optimization.

[0080] 5. It has promoted the development of intelligent operation and maintenance. As the scale and complexity of systems increase, it becomes increasingly difficult to manually formulate and adjust scheduling strategies. This application demonstrates the potential of machine learning methods in this field and provides a technical foundation for future automated and intelligent operation and maintenance systems.

[0081] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A GPU cluster scheduling strategy optimization system based on deep reinforcement learning, characterized in that, The system comprises a simulation environment layer, a reinforcement learning layer, and a policy evaluation layer, wherein: The simulation environment layer provides a realistic and reliable GPU cluster scheduling environment, replicating the core mechanisms of actual cluster scheduling, including resource constraint checks, multi-round scheduling processes, and task queue management. It also provides controllable experimental conditions to facilitate comparative evaluation and iterative optimization of strategies. Specifically, the simulation environment layer includes: The cluster state manager is used to maintain complete state information of the virtual cluster, including the resource configuration, current usage, and list of running tasks for each node. In the cluster state manager, each virtual node contains three types of resources: CPU, memory, and GPU. The node status tracks the total amount, allocated amount, and remaining amount of each resource in real time. Task Pods are represented by a three-dimensional resource demand vector, supporting pure CPU tasks, single GPU tasks, and multi-GPU tasks. A task generator is used to generate diverse task flows based on a configured load pattern to simulate the workload characteristics of a real production environment. The task generator includes three types of tasks: large memory tasks to simulate data preprocessing, data extraction-transformation-loading scenarios; regular GPU tasks to simulate standard model training jobs; and large GPU tasks to simulate distributed deep learning training. The task generator generates task sequences according to the configured proportions and arrival patterns. The scheduling execution engine implements the core logic of the Kubernetes scheduler, including node filtering, scoring, and binding operations. The scheduling execution engine employs a multi-round scheduling mechanism to reflect the behavior of the Kubernetes scheduler. When faced with a batch of tasks to be scheduled, the Kubernetes scheduler does not complete all scheduling at once, but processes them one by one using a round-robin approach. Specifically, in each round, the Kubernetes scheduler takes a task from the head of the queue, finds a suitable node for that task, and if there is no node that meets the requirements, the Kubernetes scheduler chooses to put the task back to the tail of the queue and retry it in subsequent rounds. When the scheduling execution engine performs the binding operation, it first checks whether the target node has sufficient resources. The binding operation is only performed when all resource dimensions are satisfied. After successful binding, the corresponding resources are marked as allocated, and the task is removed from the scheduling queue. Environment interface adapter is used to encapsulate the GPU cluster scheduling process into a standard reinforcement learning environment interface, providing state observation, action execution, and reward feedback functions; The reinforcement learning layer transforms the GPU cluster scheduling problem into a Markov decision process based on a deep neural network, and applies a reinforcement learning strategy to obtain the optimal scheduling strategy. The strategy evaluation layer statistically analyzes various key indicators, compares and analyzes multiple scheduling strategies, and visualizes the analysis results to optimize the GPU cluster scheduling strategy. The key indicators include GPU utilization, task waiting time, scheduling success rate, resource balance, and number of scheduling rounds.

2. The GPU cluster scheduling strategy optimization system based on deep reinforcement learning according to claim 1, characterized in that, The reinforcement learning layer includes: The core module of the PPO algorithm includes two deep neural network components: a policy network and a value network. The policy network is responsible for learning the mapping relationship from state to action and outputting the probability distribution of selecting each scheduling action in a given cluster state. The value network is used to evaluate the long-term value of the state and provide a benchmark for policy improvement. The deep neural network used is an Actor-Critic neural network architecture. The experience collection module is responsible for running the current strategy and interacting with the environment, collecting state transition and reward information; The training optimization module is used to continuously improve the scheduling strategy by updating the parameters of the deep neural network through gradient descent using collected empirical data.

3. The GPU cluster scheduling strategy optimization system based on deep reinforcement learning according to claim 2, characterized in that, In the core module of the PPO algorithm, key information required for scheduling decisions is obtained through a designed state representation method. The state representation vector includes: the characteristics of the task currently to be scheduled, the resource status of all nodes, and global statistical information, wherein: The characteristics of the tasks currently to be scheduled are represented by a normalized three-dimensional vector, which corresponds to the requirements of GPU, CPU and memory respectively; The resource status of all nodes uses an enhanced nine-dimensional feature vector, which includes not only the absolute remaining amount and relative remaining ratio of each resource, but also a binary schedulable flag, a resource imbalance metric, and the standard deviation of utilization. The core module of the PPO algorithm also employs advantage function estimation, where the advantage function A(s,a) represents the degree of improvement of action a relative to the average performance in state s. Specifically, it uses generalized advantage estimation (GAE), which balances bias and variance by introducing a decay factor λ. The calculation process of GAE utilizes the evaluation of the value network and the actual rewards obtained. For each time step in the trajectory, the temporal difference error, i.e., TD error, is calculated, representing the difference between the current value estimate and the actual value. The advantage estimate is obtained by accumulating future TD errors through exponential decay. The decay factor λ is used to control the decay rate; a larger λ depends more on the actual reward, while a smaller λ depends more on the evaluation of the value network. The core module of the PPO algorithm also incorporates a pruning mechanism to avoid performance crashes caused by excessively large single updates by limiting the differences between the old and new strategies; specifically: Define the strategy ratio r = π_new(a|s) / π_old(a|s); Where π_new(a|s) is the new policy function, representing the probability distribution of taking action a under state s; π_old(a|s) is the old policy function, representing the probability distribution of taking action a under state s. Calculate two objective functions: L1 = r * A and L2 = clip(r, 1-ε, 1+ε) * A; Where A is the advantage function, which represents the advantage of taking a certain action in a certain state compared to the average level, and is used to evaluate the relative value of the action; clip(r, 1-ε, 1+ε) is the cutoff function, which restricts the value of variable r to the interval [1-ε, 1+ε]; ε is the pruning parameter, which is used to limit the policy update magnitude; The ultimate goal is to minimize (L1, L2) while ensuring that the update magnitude does not exceed ε.

4. The GPU cluster scheduling strategy optimization system based on deep reinforcement learning according to claim 2, characterized in that, In the experience collection module, the collected reward information includes: Instant rewards provide corresponding feedback for each scheduling action: a high positive reward is given for successfully scheduling a GPU task, reflecting the importance of GPU resource optimization; a medium reward is given for successfully scheduling a non-GPU task, encouraging overall scheduling efficiency; a small negative reward is given for choosing delayed scheduling, preventing excessive latency; and penalties are given for attempting invalid scheduling, guiding the avoidance of incorrect decisions. The final reward, calculated at the end of a scheduling round, is used to comprehensively evaluate the scheduling effect: GPU allocation rate reward, reflecting the efficiency of GPU resource utilization; task success rate reward, ensuring that tasks can be successfully scheduled and maintaining the basic functions of the system; scheduling efficiency reward, encouraging fast scheduling and reducing task waiting time; resource balancing reward, guiding the reasonable distribution of load among nodes and avoiding hotspots and resource fragmentation; and multi-round penalties, preventing the overuse of latency strategies and ensuring the timeliness of scheduling. The collected state transition information includes: when the agent chooses to schedule a task to a node, it first verifies the resource constraints. If they are met, it updates the node's resource state and removes the task; otherwise, it keeps the state unchanged and gives a negative reward; when the agent chooses to delay the action, the current task is moved to the end of the queue, and the environment transitions to the state of processing the next task. The experience collection module employs a parallel environment design, running multiple simulated environment instances simultaneously. Each instance uses the current policy for scheduling simulation. In each environment, the agent interacts with the environment for a complete round: starting from the initial state, it obtains the current tasks to be scheduled and the cluster state; it samples actions according to the probability distribution output by the policy network; it executes actions and obtains environmental feedback, including the next state, reward information, and whether the process has ended; it stores the transfer tuple (s, a, r, s', done) in the experience buffer, where s is the current state, a is the executed action, r is the immediate reward, s' is the next state, and done is the termination flag; the operation is repeated until all tasks are completed or the maximum number of steps is reached.

5. The GPU cluster scheduling strategy optimization system based on deep reinforcement learning according to claim 2, characterized in that, The training optimization module employs a delayed scheduling mechanism. For a cluster of N nodes, the action space contains N+1 discrete actions. The first N actions correspond to scheduling the current task to a specific node, and the N+1th action represents delayed scheduling, which puts the task back to the end of the queue. The training optimization module uses mini-batch gradient descent to update the parameters of the deep neural network, randomly dividing the collected empirical data into mini-batches; and calculating the policy loss and value loss for each mini-batch. Update network parameters using the Adam optimizer; repeat for multiple rounds until policy improvement saturation. Key hyperparameters during training include: learning rate, which uses a cosine annealing strategy and gradually decays from the initial value; batch size, which is dynamically adjusted based on GPU memory; PPO pruning parameter ε; generalized advantage estimation (GAE) parameter λ; and discount factor γ.

6. The GPU cluster scheduling strategy optimization system based on deep reinforcement learning according to claim 1, characterized in that, The strategy evaluation layer includes: The performance metrics calculation module is used to collect statistics on various key metrics in real time. Among them, GPU utilization is the most critical metric, used to calculate the proportion of successfully allocated GPUs to the total number of GPUs required; scheduling success rate is used to measure the proportion of successfully scheduled tasks to the total number of tasks; task waiting time is used to count the time interval between task submission and scheduling; resource balance is measured using the standard deviation of resource utilization of each node; and the number of scheduling rounds reflects the number of rounds required to complete all scheduling, reflecting scheduling efficiency. The offline comparison module implements various scheduling strategies and performs fair comparisons with reinforcement learning strategies. These strategies include: the BinPack strategy, which calculates the resource utilization rate after scheduling tasks to nodes and selects the node with the highest resource utilization rate; the Balanced strategy, which calculates the standard deviation of resource utilization for each node after scheduling and selects the node with the smallest standard deviation; the LeastRequested strategy, which selects the node with the most remaining resources, calculates the remaining proportion for each resource type, and selects the node with the largest average; and the Random strategy, which randomly selects from all feasible nodes as a performance lower bound reference. The visualization and analysis module provides intuitive charts to help researchers analyze the behavior and performance of various strategies. Among them, the training curve shows the changes in reward and loss metrics as training progresses; the resource heatmap shows the resource usage of each node at each time point; the scheduling decision distribution statistics show the frequency of each node being selected; and the task waiting time distribution reveals the scheduling latency characteristics. The experiment management module supports the management of batch experiment configurations, result storage, and statistical analysis.

7. The GPU cluster scheduling strategy optimization system based on deep reinforcement learning according to claim 2, characterized in that, The specific architecture of the Actor-Critic neural network is as follows: The input encoding layer adopts a separate encoding design, with the task encoder and node encoder processing different types of input features respectively; the task encoder uses a three-layer fully connected network to map low-dimensional task features to a high-dimensional representation space; the node encoder processes node features and also adopts a three-layer structure. The decision output layer makes the final prediction based on the fused features. The feature fusion adopts a concatenation method, combining the task embedding and attention context into a joint representation. This joint representation contains the current task's requirement information and the global resource status of the cluster. Actor network branch: The Actor output layer does not use an activation function, resulting in unnormalized action preference scores; Critic network branch: The output of Critic represents an estimate of the long-term cumulative reward for the current state; The Actor and Critic network branches share the underlying encoding and attention layers, but use independent parameters in the final decision layer; The Actor-Critic neural network architecture includes an attention mechanism, which works as follows: First, a query vector is generated based on the task embedding; then, all node embeddings are converted into key vectors and value vectors; next, the dot product of the query vector and each key vector is calculated to obtain the original attention score; attention weights are obtained through softmax normalization to ensure that the sum of the weights is 1; finally, the attention weights are used to perform a weighted summation of the value vectors to obtain the aggregated context vector.

8. The GPU cluster scheduling strategy optimization system based on deep reinforcement learning according to claim 2, characterized in that, An action masking mechanism is introduced into the deep neural network used. Specifically, this involves performing feasibility checks based on node resource status and task requirements, generating a mask vector, and applying the generated mask vector to the deep neural network output. This ensures that only feasible scheduling actions are selected. Feasibility checks are performed in real time at each decision point. For the current task to be scheduled and each candidate node, three resource dimensions are checked: whether the number of GPUs is sufficient, whether the number of CPU cores is sufficient, and whether the memory capacity is sufficient. Only when all resource dimensions are satisfied is the corresponding scheduling action marked as feasible. The mask vector is generated based on the feasibility check results. The mask vector is an N+1 dimensional binary vector, where the first N elements correspond to the scheduling actions of N nodes, the last element corresponds to the delay action, the position corresponding to the action is 1, and the position corresponding to the action is 0. The mask information is not only used to constrain action selection, but also serves as additional input information. In the node feature vector, it contains a binary feature indicating whether the current task can be scheduled to the node.

Images