A cloud resource dynamic scheduling method based on security reinforcement learning

Abstract

The application discloses a kind of cloud resource dynamic scheduling methods based on security reinforcement learning, comprising: each application requirement is abstracted as a directed acyclic graph, each job stage is indicated with node, and each edge indicates the data dependency between job stage;The dynamic scheduling problem of computing resource is modeled as Markov decision process, a reinforcement learning strategy model is constructed, and state, action and reward are defined;The reinforcement learning strategy model includes graph neural network and decision network, the output of graph neural network is received by decision network, and the task node and resource allocation upper limit of selection scheduling are generated;With the goal of reward maximization, the reinforcement learning strategy model is trained;After training, for the current time step, the state of the computing cluster is obtained in real time, and the task node to be scheduled and the actual allocated computing resource of each decision step are calculated in order.

Description

Technical Field

[0001] The present invention belongs to the field of cloud resource dynamic scheduling, and in particular relates to a cloud resource dynamic scheduling method based on secure reinforcement learning. Background Art

[0002] The rapid development of cloud computing has made resource scheduling a critical issue. As data centers continue to grow in size, cluster resource management is becoming increasingly important. As a crucial component of resource management, a well-designed scheduling mechanism can significantly improve the overall efficiency of data centers. Reasonable resource scheduling plays a crucial role in cloud computing.

[0003] Cloud computing centers have limited cloud resources. Application requests arrive sequentially, each requesting a certain amount of cloud resources for a specific time period. Each request may consist of multiple phases, with each phase containing numerous task instances corresponding to different resource requirements. Cloud computing service centers must decide whether to initiate application requests and which task instances to allocate to each resource node, while satisfying various constraints. Reasonable cloud resource scheduling can maximize service quality within limited resources.

[0004] Most existing resource scheduling solutions use heuristic algorithms, such as applying the heuristic method of the multi-dimensional box packing problem to the context of cluster schedulers. Graphene abstracts jobs into a directed acyclic graph for scheduling jobs with complex dependencies and heterogeneous resource requirements.

[0005] However, these handcrafted heuristic methods (static strategies) struggle to achieve good results in complex scenarios. As cloud computing resources continue to scale and uncertainty grows, the optimization timeliness of heuristic algorithms is limited. Cloud computing resource scheduling requires more efficient scheduling algorithms to assist and collaborate.

[0006] Cloud resource scheduling is a typical sequential decision-making problem. Using reinforcement learning to convert costs into an optimal policy offline, dealing with various uncertain data during offline training, and then making faster decisions online is a promising approach.

[0007] The DeepRM model uses RL to design online multi-resource cluster scheduling, but treats the cluster as a single resource collection and ignores the machine fragmentation effect. Some work focuses on the time-varying workload (TVW) characteristics, strives to avoid performance degradation caused by resource contention, and builds a state view of each machine.

[0008] In general, reinforcement learning can learn optimal policies without requiring explicit problem models or domain knowledge, is adaptable to diverse problems and environments, and exhibits robustness and versatility. [The following sentences appear unrelated and likely represent unrelated sentences:] ... Summary of the Invention

[0009] In order to solve the problems existing in the background technology, the present invention provides a cloud resource dynamic scheduling method based on secure reinforcement learning. It introduces the idea of ​​secure reinforcement learning to find a scheduling strategy with the maximum expected benefit, while ensuring that some constraints of the scheduling system are met, which can achieve more efficient resource scheduling and improve the utilization efficiency of computing resources.

[0010] A cloud resource dynamic scheduling method based on secure reinforcement learning includes the following steps:

[0011] (1) Each application requirement is abstracted into a directed acyclic graph. Each job stage is represented by a node in the directed acyclic graph. Each job stage contains multiple parallel computing tasks, and each edge represents the data dependency between job stages.

[0012] (2) Model the dynamic scheduling problem of computing resources as a Markov decision process, build a reinforcement learning strategy model, and define the state, action, and reward of the reinforcement learning strategy model;

[0013] The reinforcement learning strategy model consists of a graph neural network and a decision network. The graph neural network receives the observation state and is responsible for aggregating directed acyclic graph information. The decision network receives the output of the graph neural network and generates task nodes for selection and scheduling and resource allocation upper limits. The two pieces of information generated by the decision network are constrained and then used as action outputs.

[0014] (3) Training the reinforcement learning strategy model with the goal of maximizing rewards;

[0015] (4) After the reinforcement learning strategy model is trained, for the current time step, combined with the state of the computing cluster obtained in real time, the task nodes to be scheduled and the actual allocated computing resources for each decision step are calculated in sequence.

[0016] The goal of setting computing resource scheduling decisions in this invention is mainly to optimize the overall computing resource utilization efficiency. Specifically, when each task arrives randomly, the complete task structure and resource requirements are known, and the scheduling platform allocates resources based on the usage of computing resources.

[0017] Unlike simple heuristic scheduling strategies used in the past, this invention treats computing resource scheduling as a sequential decision-making problem. Each decision corresponds to the specific allocation of computing tasks to maximize the utilization efficiency of the computing cluster. By considering the dependencies between task requirements and the potential for parallelization, the long-term overall utilization efficiency of the computing cluster is optimized.

[0018] In step (1), the job phase represented by the parent node can only be run after the job phase represented by the child node is completed.

[0019] In step (2), when the graph neural network is responsible for aggregating the directed acyclic graph information, it includes three levels of information aggregation and outputs three different types of embeddings, namely, information aggregation between computing task nodes, overall information aggregation of each directed acyclic graph task, and information aggregation between different application task requirements as a global embedding vector.

[0020] In step (2), the state is the task requirements and resource status of each job stage of all application requirements under a decision step; the action is to select a job stage of an application requirement and generate and allocate the corresponding upper limit of the number of resource nodes; the reward is the resource utilization efficiency within a period of time, which is expressed by the total time to complete the task.

[0021] In step (3), the reinforcement learning algorithm is used for training with the goal of maximizing the expected cumulative reward, which is specifically expressed as:

[0022]

[0023] Among them, J(π θ ) represents the strategy π θ The performance evaluation function, R(s t ,a t ) is the reward function, which represents the reward obtained when taking action a in state s. It means that in the strategy π θ Next state s t and action a t The expectation of the distribution of

[0024] During training, the discount factor is used to adjust the present value of future rewards, and the transfer function is used to predict the next state.

[0025] The goal is to maximize the expected cumulative reward, which must satisfy a series of hard constraints, expressed as:

[0026]

[0027] in, A generalized operator representing two functions, p(a t |s t ) represents the policy network input state s under feasibility constraintst Produce action a t The probability of π θ (a t |s t ) represents the policy network input state s t Produce action a t The probability of a t 、s t Represent the action and state at time t, respectively, a t |s t Indicates that in s t Take action a in state t ;F(s t ) represents the feasibility constraint, which is used to determine the feasibility of the current state s t Determine which actions are legal; feasibility constraints specifically include resource constraints, allocation constraints, and disaster recovery constraints.

[0028] In step (2), the decision network includes a node scoring network and an allocated computing resource quantity scoring network;

[0029] In the node scoring network, for node v in job i, its score q iv Calculated by the following formula:

[0030] q iv ≈q(e iv ,y i , z)

[0031] Among them, e iv is the embedding vector of the node, y i is the job-level embedding, z is the global embedding, and q() is a nonlinear transformation function used to map the embedding vector to a scalar value representing the priority of the node;

[0032] In the scoring network for allocating computing resources, for job i, the score w of the number of resources is li Calculated by the following formula:

[0033] w li ≈w(y i , z, l)

[0034] where l is the parallelism limit specified for job i and w() is a nonlinear transformation function that generates a score based on the job and global embeddings and the parallelism limit.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] This invention is a scalable real-time computing resource scheduling method that achieves more efficient resource scheduling through global and forward-looking processing, improving computing resource utilization. This method utilizes a scalable graph neural network to represent scheduling policies and combines imitation learning with reinforcement learning to address scheduling problems with dynamic and complex dependent systems. The invention also incorporates constraint correction to meet the requirements of computing resource scheduling systems in practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of matching computing tasks and computing resources according to an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the online allocation process of computing resources in an embodiment of the present invention;

[0039] Figure 3 Schematic diagram of the overall network framework in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0041] A cloud resource dynamic scheduling method based on secure reinforcement learning includes the following steps:

[0042] S1, such as Figure 1 As shown in the figure, each application requirement is considered as a directed acyclic graph (DAG). Each job stage is represented by a node in the graph G. Each stage contains multiple tasks that can be run in parallel, and each edge represents the data dependency between job stages. A graph neural policy network based on the graph transfer mechanism is constructed as the state representation network to aggregate the information in the DAG.

[0043] The specific process of information aggregation representation is:

[0044] This invention treats each application requirement within a computing requirement as a directed acyclic graph (DAG). A message-passing graph neural network (MNN) is employed to parameterize the current computing requirement. In the DAG G, each node represents a job phase, and the edges between each two nodes represent the data dependency between the two job phases. The job phase represented by the child node can only be executed after the parent node's job phase has completed execution.

[0045] Each node in the graph is embedded as a feature vector, which includes information such as the number of tasks, average task time, number of servers, etc. In each information aggregation link, a learnable function is used to aggregate the information of the child nodes of the task node.

[0046] like Figure 3 As shown in Figure 1, the entire graph neural network includes three levels of information aggregation, outputting three different types of embeddings: information aggregation between computing task nodes, overall information aggregation of each directed acyclic graph task, and information aggregation between different application task requirements as a global embedding vector. Therefore, a total of six nonlinear transformations are involved, and each level of information aggregation is represented by two nonlinear transformations.

[0047] A. Information aggregation between computing task nodes captures information of each node and its subnodes:

[0048]

[0049] where e iv represents the embedding vector of node v in job i, and g() represents the nonlinear transformation function used to generate the final embedding of the node. iv Represents the original feature vector of node v in job i. f() represents the function used to calculate the embedding of child nodes, aggregating features from all child nodes u of node v.

[0050] B. Aggregate the overall information of each DAG task to summarize the information of the entire job DAG:

[0051] y i =SUMMARIZE(e iv |v∈G i )

[0052] where y i Represents the embedding vector of job i SUMMARIZE() represents the aggregation function, which is used to summarize the embeddings of all nodes in job i. i Represents the directed acyclic graph of job i.

[0053] C. The global embedding vector combines the per-job embeddings from all jobs to form a cluster-level summary:

[0054] z=SUMMARIZE(y i |i∈[1,n])

[0055] Where z represents the global embedding vector, which indicates the state of the entire cluster; SUMMARIZE() represents the aggregation function, which is used to summarize the embeddings of all jobs; and n represents the number of jobs in the current cluster.

[0056] The information stored in these embeddings is not hard-coded. Through end-to-end training, we automatically learn what is statistically important and how to calculate this information from the input directed acyclic graph. In practice, the feature transformation network is implemented using fully connected layers.

[0057] S2. Model the dynamic scheduling problem of computing resources as a Markov decision process. Given historical data, establish a Markov decision process consisting of a series of states, actions, and rewards.

[0058] like Figure 2 As shown in Figure 1, the policy network consists of two parts: the graph neural network in S1 and the decision network in S3. The graph neural network is responsible for aggregating directed acyclic graph information, and the decision network generates two parts of information, representing the selected task nodes and the specific amount of allocated resources.

[0059] State: The observation space of the policy model consists of information about all available jobs and the server state at each scheduling event. Specifically, the state of each job phase for all application requirements at decision step t is defined as a two-dimensional vector representing task requirements and resource status. The specific physical meaning of each dimension is the number of tasks remaining in that phase, the average task time, the number of available servers, and the number of tasks awaiting decision.

[0060] Action: When there are idle servers in the computing resources, select a job phase that meets the application requirements and generate and allocate the corresponding upper limit of resource nodes.

[0061] Reward: Resource utilization efficiency over a period of time, expressed as the total time to complete the task.

[0062] Discount factor: Using a smaller discount factor of 0.9 makes the model place more emphasis on recent rewards, allowing it to quickly respond to emergencies or instantaneous changes in resources or demand.

[0063] Before actual application, the policy network is initialized and trained through imitation learning. In this algorithm, the Actor network, similar to the subsequent reinforcement learning policy network, is responsible for generating the probability distribution for computing task allocation. This model is then used to hot-start reinforcement learning training, stabilizing and accelerating RL model training.

[0064] The reinforcement learning algorithm is used for training, and the goal of training is to maximize the reward, which is specifically expressed as:

[0065]

[0066] Among them, J(π θ ) represents the performance evaluation function of the strategy, R(s t , a t ) is the reward function, which represents the reward obtained when taking action a in state s. It means that in the strategy π θ Next, state s t and action a t The expectation of the distribution of

[0067] The discount factor is then used to adjust the present value of future rewards, and the transition function is used to predict the next state.

[0068] In the actual scheduling process, new, random task requests continuously appear, meaning that both the pool of pending tasks and the pool of available servers are dynamically changing. When a task completes or a new task arrives, the system begins scheduling and maps the pending job stages to available servers. To handle the dynamic and continuous pattern of task arrival, the reinforcement learning agent also employs this task demand arrival pattern during training. The task input process is exogenous and independent of the policy model's behavior.

[0069] Training uses a curriculum learning approach, initially training the model with shorter task sequences. As policy performance gradually improves, the training difficulty is increased to learn more effective policies. In the early stages of training, the model is first exposed to relatively simple and easy-to-learn samples. As training progresses, more complex and challenging samples are gradually introduced to guide the model to learn stronger scheduling policies.

[0070] The goal is to maximize the expected cumulative reward, and to satisfy a series of hard constraints, such as Figure 2 The constraint correction layer shown, including resource capacity constraints and data dependency constraints, ensures the feasibility of the allocation results. Specifically, the feasibility constraints include the following three constraints:

[0071] A. Resource constraints: Ensure that each job stage has sufficient resources when running

[0072] B. Allocation constraints: Many job stages require special resources when running, such as specific operating system versions, specific hardware, etc.

[0073] C. Disaster recovery constraints: Different operation stages of the same application requirement cannot be placed on the same resource node at the same time.

[0074] When training the policy network, a feasibility constraint is set, which is expressed as:

[0075]

[0076] in, A generalized operator representing two functions, p(a t |s t ) represents the policy network input state s under feasibility constraints t Produce action a t The probability of π θ (a t |s t ) represents the policy network input state s t Produce action at The probability of a t |s t Indicates that in s t Take action a in state t ,F(s t ) represents the feasibility rule, which is used to calculate the feasibility of the current state s t Determine which actions are legal.

[0077] The present invention sets the feasibility constraint F(s t ), based on the current computing resource usage and demand execution, determine the available actions for step t, which can ensure that the resource allocation decision meets the hard constraints of actual task execution. Finally, based on the strategy π θ (a t |s t ) and feasibility constraints F(s t ) are combined to perform actual resource allocation. The policy network is a parameterized function, π represents the policy, and θ represents the parameters of the policy, which define the mapping from state to action.

[0078] The trained policy network generates the computing nodes to be scheduled and the upper limit of resource allocation, which are then applied in the actual online scheduling step.

[0079] S3, such as Figure 3 As shown in Figure 1, the policy network consists of a graph neural network and a decision network. The decision network is responsible for outputting two pieces of information: which task node to select and how much computing resources to allocate for this task. Specifically, the decision network consists of two parts:

[0080] S31. Node Scoring Network

[0081] The input data includes the input features of each node, the global summary of each directed acyclic graph, and the global summary features. These inputs are first transformed into a large input vector after a series of dimension transformations.

[0082] After a series of hidden layers, it passes through a fully connected layer without an activation function to produce an output with the output dimension of (batch size, total number of nodes), with one score for each node.

[0083] A validity mask is applied to adjust each node score, ensuring that only legal (constrained) nodes at the current decision step are considered.

[0084] All scores are normalized using the softmax function to obtain the selection probability distribution of each node.

[0085] For node v in job i, its score q iv Calculated by the following formula:

[0086] q iv ≈q(e iv ,y i , z)

[0087] where e iv is the embedding vector of the node, y i is the job-level embedding, z is the global embedding, and q() is a nonlinear transformation function (a fully connected layer is used here) that maps the embedding vector to a scalar value representing the priority of the node.

[0088] S32. Allocate computing resources to score the network

[0089] The input data includes the input features of each node, the global summary of each directed acyclic graph, and the global summary features. These inputs are first transformed into a large input vector after a series of dimension transformations.

[0090] The state dimension is then added to represent specific actions. The expanded dimension is equal to the number of executors in the compute resources. This embeds action information into the state representation, enabling the model to make decisions based on the combination of state and action. The original input dimension is (batch size, total number of nodes, node feature length), and the expanded state dimension is (batch size, total number of nodes * expanded dimension, node feature length + 1).

[0091] After the expanded state features pass through a series of hidden layers, they pass through a fully connected layer without an activation function to produce an output with an output dimension of (batch size, total number of nodes * upper limit of available computing resources). Each execution of each job is limited to one score.

[0092] Apply a validity mask to adjust assignment scores, ensuring only valid assignments are considered.

[0093] The output is reshaped into (batch size, total number of nodes, upper limit of available computing resources) and normalized using the softmax function to obtain the probability distribution of each execution limit selected by each job.

[0094] For job i, the score w of the number of resources li Calculated by the following formula:

[0095] w li ≈w(y i , z, l)

[0096] Here l is the parallelism limit specified for job i, and w() is also a nonlinear transformation function (here a fully connected layer is used) that generates a score based on the job and global embeddings and the parallelism limit.

[0097] S4. Based on the current decision step scheduling node and the corresponding resource allocation upper limit obtained in S2, combined with the usage status of the computing cluster obtained in real time, calculate the computing resources actually allocated for the current decision step, and calculate all decision step results of the current time step in sequence.

[0098] The specific execution steps corresponding to each decision step are as follows:

[0099] When a scheduling event occurs (such as a stage completion, task exhaustion, or new job arrival), a schedulable stage is selected along with the corresponding parallelism limit.

[0100] If the job currently has fewer executors allocated than the set parallelism limit, additional executors are assigned to the stage until the limit is reached.

[0101] This process is repeated until all executors are assigned or there are no more schedulable stages.

[0102] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A cloud resource dynamic scheduling method based on secure reinforcement learning, characterized in that: The following steps are involved: (1) Each application requirement is abstracted into a directed acyclic graph. Each job stage is represented by a node in the directed acyclic graph. Each job stage contains multiple parallel computing tasks, and each edge represents the data dependency between job stages. (2) Model the dynamic scheduling problem of computing resources as a Markov decision process, build a reinforcement learning strategy model, and define the state, action, and reward of the reinforcement learning strategy model; The reinforcement learning strategy model consists of a graph neural network and a decision network. The graph neural network receives the observation state and is responsible for aggregating directed acyclic graph information. The decision network receives the output of the graph neural network and generates task nodes for scheduling and resource allocation upper limits. The two pieces of information generated by the decision network are constrained and then output as actions. Ultimately, the strategy and feasibility constraints are combined to perform actual resource allocation. The decision network includes a node scoring network and a scoring network for the number of allocated computing resources; In the node scoring network, for node v in job i, its score q iv Calculated by the following formula: q iv ≈q(e iv ,y i ,z) Among them, e iv is the embedding vector of the node, y i is the job-level embedding, z is the global embedding, and q() is a nonlinear transformation function used to map the embedding vector to a scalar value representing the priority of the node; In the scoring network for allocating computing resources, for job i, the score w of the number of resources is li Calculated by the following formula: w li ≈w(y i ,z,l) Where l is the parallelism limit specified for job i, and w() is a nonlinear transformation function that generates a score based on the job and global embeddings and the parallelism limit; (3) Training the reinforcement learning strategy model with the goal of maximizing rewards; (4) After the reinforcement learning strategy model is trained, for the current time step, combined with the state of the computing cluster obtained in real time, the task nodes to be scheduled and the actual allocated computing resources for each decision step are calculated in sequence.

2. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: In step (1), the job phase represented by the parent node can only be run after the job phase represented by the child node is completed.

3. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: In step (2), when the graph neural network is responsible for aggregating the directed acyclic graph information, it includes three levels of information aggregation and outputs three different types of embeddings, namely, information aggregation between computing task nodes, overall information aggregation of each directed acyclic graph task, and information aggregation between different application task requirements as a global embedding vector.

4. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: In step (2), the state is the task requirements and resource status of each operation stage of all application requirements under a decision step; the action is to select a certain operation stage of an application requirement and generate the upper limit of the number of resource nodes allocated accordingly; The reward is the resource utilization efficiency within a period of time, expressed as the total time to complete the task.

5. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: In step (3), the reinforcement learning algorithm is used for training with the goal of maximizing the expected cumulative reward, which is specifically expressed as: Among them, J(π θ ) represents the strategy π θ The performance evaluation function, R(s t ,a t ) is the reward function, which represents the reward obtained when taking action a in state s. It means that in the strategy π θ Next state s t and action a t The expectation of the distribution of .

6. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: During training, the discount factor is used to adjust the present value of future rewards, and the transfer function is used to predict the next state.

7. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 5 is characterized in that: The goal is to maximize the expected cumulative reward, which must satisfy a series of hard constraints, expressed as: in, A generalized operator representing two functions, p(a t |s t ) represents the policy network input state s under feasibility constraints t Produce action a t The probability of π θ (a t |s t ) represents the policy network input state s t Produce action a t The probability of a t 、s t Represent the action and state at time t, respectively, a t |s t Indicates that in s t Take action a in state t ;F(s t ) represents the feasibility constraint, which is used to determine the feasibility of the current state s t Determine which actions are legal; feasibility constraints specifically include resource constraints, allocation constraints, and disaster recovery constraints.

Images