Cloud resource dynamic scheduling method based on security reinforcement learning

Abstract

The invention discloses a cloud resource dynamic scheduling method based on security reinforcement learning, which comprises the following steps: abstracting each application demand into a directed acyclic graph, representing each job stage by a node, and representing data dependence between the job stages by each edge; modeling a dynamic scheduling problem of computing resources into a Markov decision process, constructing a reinforcement learning strategy model, and defining a state, an action and an award; the reinforcement learning strategy model comprises a graph neural network and a decision network, and the decision network receives output of the graph neural network and generates task nodes for selecting scheduling and a resource allocation upper limit; training the reinforcement learning strategy model by taking reward maximization as a target; after training is completed, for the current time step, task nodes to be scheduled and actually distributed computing resources in each decision step are calculated in sequence in combination with the state, obtained in real time, of the computing cluster. According to the invention, more efficient resource scheduling can be realized, and the utilization efficiency of computing resources is improved.

Description

Technical Field

[0001] The present invention belongs to the field of cloud resource dynamic scheduling, and in particular relates to a cloud resource dynamic scheduling method based on secure reinforcement learning. Background Art

[0002] The rapid development of cloud computing has made resource scheduling a key issue that needs to be solved. As the scale of data centers is getting larger and larger, cluster resource management issues are becoming more and more important. As an important part of resource management, a well-designed scheduling mechanism can greatly improve the overall efficiency of data centers. Reasonable resource scheduling plays a vital role in cloud computing.

[0003] The cloud computing center has limited cloud resources. Application requirements arrive one by one. Each application requirement requests the cloud computing center to use a certain amount of cloud resources at a specific time. Each requirement may consist of multiple stages, and each stage has many task instances corresponding to different resource requirements. The cloud computing service center needs to decide whether to start executing application requirements and what task instances to allocate on each resource node under the premise of meeting multiple constraints. Reasonable cloud resource scheduling can maximize service quality under limited resources.

[0004] Most existing resource scheduling schemes use heuristic algorithms, such as applying the heuristic method of the multi-dimensional box packing problem to the context of cluster schedulers. Graphene abstracts the job into a directed acyclic graph for scheduling jobs with complex dependencies and heterogeneous resource requirements.

[0005] However, these manual heuristic methods (static strategies) are difficult to achieve good results in complex scenarios. As the scale of cloud computing resources continues to expand, uncertainty becomes stronger and stronger, and the optimization timeliness of heuristic algorithms is limited. Cloud computing resource scheduling requires more efficient scheduling algorithms to assist and collaborate.

[0006] The cloud resource scheduling problem is a typical sequential decision-making problem. Using reinforcement learning to convert the cost to learn an optimal strategy offline, dealing with various uncertain data during offline training, and then making faster decisions online is a promising direction.

[0007] The DeepRM model uses RL to design online multi-resource cluster scheduling, but regards the cluster as a single resource collection and ignores the machine fragmentation effect; some work focuses on the time-varying workload (TVW) characteristics, strives to avoid performance degradation caused by resource contention, and builds a state view of each machine.

[0008] In general, reinforcement learning can learn optimal strategies without explicit problem models or domain knowledge, can adapt to different problems and environments, and has certain robustness and versatility. The matrix is ​​used to indirectly train some dense layers in the neural network while keeping the pre-trained weights unchanged. There are many constraints in actual cloud computing resource scheduling tasks, such as resource capacity constraints, disaster recovery constraints, etc. Existing deep reinforcement learning algorithms cannot directly generate solutions that meet these constraints. Summary of the invention

[0009] In order to solve the problems existing in the background technology, the present invention provides a cloud resource dynamic scheduling method based on secure reinforcement learning, introduces the idea of ​​secure reinforcement learning, seeks a scheduling strategy with the maximum expected benefit, and ensures that some constraints of the scheduling system are met, so as to achieve more efficient resource scheduling and improve the utilization efficiency of computing resources.

[0010] A cloud resource dynamic scheduling method based on secure reinforcement learning includes the following steps:

[0011] (1) Each application requirement is abstracted into a directed acyclic graph. Each job stage is represented by a node in the directed acyclic graph. Each job stage contains multiple parallel computing tasks, and each edge represents the data dependency between job stages.

[0012] (2) Model the dynamic scheduling problem of computing resources as a Markov decision process, build a reinforcement learning strategy model, and define the state, action, and reward of the reinforcement learning strategy model;

[0013] The reinforcement learning strategy model includes a graph neural network and a decision network. The graph neural network receives the observation state and is responsible for aggregating the directed acyclic graph information. The decision network receives the output of the graph neural network and generates the task nodes and resource allocation upper limit for the selected schedule. The two parts of information generated by the decision network are constrained and corrected as action outputs.

[0014] (3) Training the reinforcement learning strategy model with the goal of maximizing rewards;

[0015] (4) After the reinforcement learning strategy model is trained, for the current time step, combined with the state of the computing cluster obtained in real time, the task nodes to be scheduled and the actually allocated computing resources for each decision step are calculated in sequence.

[0016] The present invention sets the computing resource scheduling decision-making goal mainly to optimize the utilization efficiency of the overall computing resources. Specifically, when each task arrives randomly, the complete task structure and resource requirements can be known, and the scheduling platform allocates resources according to the usage of computing resources.

[0017] Different from the simple heuristic scheduling strategy in the past, the present invention regards the computing resource scheduling problem as a sequential decision problem. Each decision corresponds to the specific allocation of computing tasks to maximize the utilization efficiency of the computing cluster. By considering the dependencies and parallel potential of task requirements, the long-term overall utilization efficiency of the computing cluster is optimized.

[0018] In step (1), the job phase represented by the parent node must be completed before the job phase represented by the child node can be run.

[0019] In step (2), when the graph neural network is responsible for aggregating the directed acyclic graph information, it includes three levels of information aggregation and outputs three different types of embeddings, namely, information aggregation between computing task nodes, overall information aggregation of each directed acyclic graph task, and information aggregation between different application task requirements as a global embedding vector.

[0020] In step (2), the state is the task requirement and resource status of each operation stage of all application requirements under a decision step; the action is to select a certain operation stage of an application requirement and generate and allocate the corresponding upper limit of the number of resource nodes; the reward is the resource utilization efficiency within a period of time, which is expressed by the total time to complete the task.

[0021] In step (3), the reinforcement learning algorithm is used for training, with the goal of maximizing the expected cumulative reward, which is specifically expressed as:

[0022]

[0023] Among them, J(π θ ) represents the strategy π θ The performance evaluation function, R(s t ,a t ) is the reward function, which represents the reward obtained when taking action a in state s. It means that in strategy π θ Next state t and action a t The expectation of the distribution of

[0024] During training, the discount factor is used to adjust the present value of future rewards, and the transfer function is used to predict the next state.

[0025] The goal is to maximize the expected cumulative reward and satisfy a series of hard constraints, expressed as:

[0026]

[0027] in, A generalized operator representing two functions, p(a t |s t ) represents the policy network input state s under the feasibility constraintt Produce action a t The probability of θ (a t |s t ) represents the policy network input state s t Produce action a t The probability of t 、s t Respectively represent the action and state at time t, a t |s t Indicates in s t Take action a in state t ; F(s t ) represents the feasibility constraint, which is used to t Determine which actions are legal; feasibility constraints specifically include resource constraints, allocation constraints, and disaster recovery constraints.

[0028] In step (2), the decision network includes a node scoring network and an allocated computing resource quantity scoring network;

[0029] In the node scoring network, for node v in job i, its score q iv Calculated by the following formula:

[0030] q iv ≈q(e iv ,y i , z)

[0031] Among them, e iv is the embedding vector of the node, y i is the job-level embedding, z is the global embedding, and q() is a nonlinear transformation function used to map the embedding vector to a scalar value representing the priority of the node;

[0032] In the scoring network for allocating computing resources, for job i, the score of resource quantity w li Calculated by the following formula:

[0033] w li ≈w(y i , z, l)

[0034] where l is the parallelism limit specified for job i and w() is also a nonlinear transformation function that generates a score based on the job and global embeddings and the parallelism limit.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention is a scalable real-time scheduling method for computing resources, which realizes more efficient resource scheduling through global and forward-looking processing, and improves the utilization efficiency of computing resources. The present invention uses a scalable graph neural network to represent the scheduling strategy, and combines imitation learning and reinforcement learning to apply to scheduling problems with dynamic changes and complex dependent systems. The present invention also introduces constraint correction to meet the requirements of computing resource scheduling systems in actual application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A schematic diagram of matching computing tasks and computing resources according to an embodiment of the present invention;

[0038] Figure 2 A schematic diagram of the online allocation process of computing resources in an embodiment of the present invention;

[0039] Figure 3 Schematic diagram of the overall network framework in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0041] A cloud resource dynamic scheduling method based on secure reinforcement learning includes the following steps:

[0042] S1, such as Figure 1 As shown in the figure, each application requirement is regarded as a directed acyclic graph, each job stage is represented by a node in the graph G, each stage contains multiple tasks that can be run in parallel, and each edge represents the data dependency between job stages. A graph neural policy network based on the graph transfer mechanism is constructed as a state representation network to aggregate the information of the directed acyclic graph.

[0043] The specific process of information aggregation representation is:

[0044] The present invention regards each application requirement in the computing requirement as a directed acyclic graph. An information transfer graph neural network is used to parameterize the current computing requirement. In the directed acyclic graph G, a node represents a job stage, and the edge between each two nodes represents the data dependency relationship between the two job stages. The job stage represented by the parent node can only be executed after the execution of the job stage represented by the child node.

[0045] Each node in the graph is embedded as a feature vector, which includes information such as the number of tasks, average task time, number of servers, etc. In each information aggregation link, a learnable function is used to aggregate the information of the child nodes of the task node.

[0046] like Figure 3 As shown in Figure 1, the entire graph neural network includes three levels of information aggregation, and outputs three different types of embeddings, namely, information aggregation between computing task nodes, overall information aggregation of each directed acyclic graph task, and information aggregation between different application task requirements as a global embedding vector. Therefore, a total of six nonlinear transformations are included, and each level of information aggregation is represented by two nonlinear transformations.

[0047] A. Information aggregation between computing task nodes captures the information of each node and its child nodes:

[0048]

[0049] where e iv represents the embedding vector of node v in job i, and g() represents the nonlinear transformation function used to generate the final embedding of the node. iv represents the original feature vector of node v in job i. f() represents the function used to calculate the child node embedding, aggregating features from all child nodes u of node v.

[0050] B. The overall information of each directed acyclic graph task is aggregated to summarize the information of the entire job DAG:

[0051] y i =SUMMARIZE(e iv |v∈G i )

[0052] where y i represents the embedding vector of job i. SUMMARIZE() represents the aggregation function, which is used to summarize the embeddings of all nodes in job i. i Represents the directed acyclic graph of job i.

[0053] C. The global embedding vector combines the per-job embeddings from all jobs to form a cluster-level summary:

[0054] z=SUMMARIZE(y i |i∈[1,n])

[0055] Where z represents the global embedding vector, which indicates the state of the entire cluster; SUMMARIZE() represents the aggregation function, which is used to summarize the embeddings of all jobs; and n represents the number of jobs in the current cluster.

[0056] The information stored in these embeddings is not hard-coded, and end-to-end training automatically learns what is statistically important and how to calculate this information from the input directed acyclic graph. In specific implementations, feature change networks are implemented using fully connected layers.

[0057] S2. Model the dynamic scheduling problem of computing resources as a Markov decision process. Given historical data, a Markov decision process is established, which consists of a series of states, actions, and rewards.

[0058] like Figure 2 As shown in the figure, the policy network consists of two parts: the graph neural network in S1 and the decision network in S3. The graph neural network is responsible for aggregating directed acyclic graph information, and the decision network generates two parts of information, representing the selected task nodes and the specific amount of allocated resources respectively.

[0059] State: The observation space of the strategy model is the information of all available jobs and the state of the server at each scheduling event. Specifically, the state of each job stage of all application requirements under decision step t is defined as a two-dimensional vector, which represents the task requirements and resource status. The specific physical meaning of each dimension is the number of tasks remaining in this stage, the average task time, the number of available servers, the number of tasks to be decided, etc.

[0060] Action: When there are idle servers in the computing resources, select a job phase of an application requirement and generate and allocate the corresponding upper limit of the number of resource nodes.

[0061] Reward: Resource utilization efficiency over a period of time, expressed as the total time to complete the task.

[0062] Discount factor: Using a smaller discount factor of 0.9 makes the model pay more attention to recent rewards, so as to respond quickly to emergencies or instantaneous changes in resources or demand.

[0063] Before actual application, the policy network is initialized and trained through imitation learning. In this algorithm, the Actor network is the same as the subsequent reinforcement learning policy network, responsible for generating the probability distribution of the relevant computing task allocation. The subsequent hot start training of the reinforcement learning model based on this model plays a role in stabilizing and accelerating the training of the RL model.

[0064] The reinforcement learning algorithm is used for training, and the goal of training is to maximize the reward, which is specifically expressed as:

[0065]

[0066] Among them, J(π θ ) represents the performance evaluation function of the strategy, R(s t , a t ) is the reward function, which represents the reward obtained when taking action a in state s. It means that in strategy π θ Next, state s t and action a t The expectation of the distribution of

[0067] The discount factor is then used to adjust the present value of future rewards, and the transition function is used to predict the next state.

[0068] In the actual scheduling process, new, random, task requests are constantly appearing, that is, the pool of tasks to be assigned and the pool of available servers are changing dynamically. When a task is completed or a new task arrives, the system starts scheduling and maps the pending job stages to the available servers to run. In order to handle the dynamic and continuous task arrival pattern, the reinforcement learning agent also adopts this task demand arrival pattern during the training process. The task input process is an exogenous process independent of the behavior of the policy model.

[0069] The training adopts the course learning method, first using a shorter task sequence to train the model. As the policy performance gradually improves, the difficulty of training is increased to learn more effective policies. In the early stages of training, the model is first exposed to relatively simple and easy-to-learn samples. As the training progresses, more complex and challenging samples are gradually introduced to guide the model to learn stronger scheduling strategies.

[0070] The goal is to maximize the expected cumulative reward, and to satisfy a series of hard constraints, such as Figure 2 The constraint correction layer shown in the figure includes resource capacity constraints and data dependency constraints, etc., to ensure the feasibility of the allocation results. Specifically, the feasibility constraints include the following three constraints:

[0071] A. Resource constraints: Ensure that each job stage has sufficient resources when running

[0072] B. Allocation constraints: Many job stages require some special resources when running, such as specific operating system versions, specific hardware, etc.

[0073] C. Disaster recovery constraint: Different operation stages of the same application requirement cannot be placed on the same resource node at the same time.

[0074] When training the policy network, a feasibility constraint is set, expressed as:

[0075]

[0076] in, A generalized operator representing two functions, p(a t |s t ) represents the policy network input state s under the feasibility constraint t Produce action a t The probability of θ (a t |s t ) represents the policy network input state s t Produce action at The probability of t |s t Indicates in s t Take action a in state t ,F(s t ) represents the feasibility rule, which is used to calculate the current state s t Determine which actions are legal.

[0077] The present invention sets the feasibility constraint F(s t ), based on the current computing resource usage and demand execution, determine the available actions for step t, which can ensure that the resource allocation decision meets the hard constraints of actual task execution. θ (a t |s t ) and feasibility constraints F(s t ) are combined to perform actual resource allocation. The policy network is a parameterized function, π represents the policy, and θ represents the parameters of the policy, which define the mapping from state to action.

[0078] The trained policy network generates the computing nodes and resource allocation upper limits to be scheduled currently, and is applied in the actual online scheduling step.

[0079] S3, such as Figure 3 As shown in the figure, the policy network consists of two parts: the graph neural network and the decision network. The decision network is responsible for outputting two pieces of information: which task node to select and how much computing resources to allocate for this task. Specifically, the decision network consists of two parts:

[0080] S31. Node scoring network

[0081] The input data includes the input features of each node, the global summary of each directed acyclic graph, and the global summary features. These inputs are first transformed into a large input vector after a series of dimension transformations.

[0082] After a series of hidden layers, a fully connected layer without an activation function is used to generate an output with the output dimension (batch size, total number of nodes), with one score for each node.

[0083] A validity mask is applied to adjust each node score, ensuring that only legal (constrained) nodes at the current decision step are considered.

[0084] All scores are normalized using the softmax function to obtain the selection probability distribution of each node.

[0085] For node v in job i, its score q iv Calculated by the following formula:

[0086] q iv ≈q(e iv ,y i , z)

[0087] where e iv is the embedding vector of the node, y i is the job-level embedding, z is the global embedding, and q() is a nonlinear transformation function (a fully connected layer is used here) that maps the embedding vector to a scalar value, indicating the priority of the node.

[0088] S32. Allocate computing resources to score the network

[0089] The input data includes the input features of each node, the global summary of each directed acyclic graph, and the global summary features. These inputs are first transformed into a large input vector after a series of dimension transformations.

[0090] Then the state dimension is added to represent the specific action. The specific scale of the expanded dimension = the number of actuators in the computing resources. In this way, the action information is inserted into the state representation, so that the model can make decisions based on the combination of state and action. The dimension of the original input is (batch size, total number of nodes, node feature length), and the expanded state dimension is (batch size, total number of nodes * expanded dimension, node feature length + 1).

[0091] After the expanded state features pass through a series of hidden layers, they pass through a fully connected layer without an activation function to produce an output with an output dimension of (batch size, total number of nodes * upper limit of available computing resources), with each execution of each job limited to one score.

[0092] Apply a validity mask to adjust assignment scores, ensuring that only valid assignments are considered.

[0093] The output is reshaped into (batch size, total number of nodes, upper limit of available computing resources), and normalized using the softmax function to obtain the probability distribution of each execution limit selected by each job.

[0094] For job i, the score w of the number of resources li Calculated by the following formula:

[0095] w li ≈w(y i , z, l)

[0096] Here l is the parallelism limit specified for job i, and w() is also a nonlinear transformation function (here a fully connected layer is used) that generates a score based on the job and global embeddings and the parallelism limit.

[0097] S4. Based on the current decision step scheduling node and the corresponding resource allocation upper limit obtained in S2, combined with the usage status of the computing cluster obtained in real time, calculate the computing resources actually allocated for the current decision step, and calculate all decision step results of the current time step in sequence.

[0098] The specific execution steps corresponding to each decision step are as follows:

[0099] When a scheduling event occurs (such as completion of a stage, exhaustion of tasks, or arrival of new jobs), a schedulable stage is selected along with the corresponding parallelism limit.

[0100] If the job currently has fewer executors allocated to it than the set parallelism limit, additional executors are allocated to the stage until the limit is reached.

[0101] This process is repeated until all executors are assigned or there are no more schedulable stages.

[0102] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud resource dynamic scheduling method based on secure reinforcement learning, characterized in that: The following steps are involved: (1) Each application requirement is abstracted into a directed acyclic graph. Each job stage is represented by a node in the directed acyclic graph. Each job stage contains multiple parallel computing tasks, and each edge represents the data dependency between job stages. (2) Model the dynamic scheduling problem of computing resources as a Markov decision process, build a reinforcement learning strategy model, and define the state, action, and reward of the reinforcement learning strategy model; The reinforcement learning strategy model includes a graph neural network and a decision network. The graph neural network receives the observation state and is responsible for aggregating the directed acyclic graph information. The decision network receives the output of the graph neural network and generates the task nodes and resource allocation upper limit for the selected schedule. The two parts of information generated by the decision network are constrained and corrected as action outputs. (3) Training the reinforcement learning strategy model with the goal of maximizing rewards; (4) After the reinforcement learning strategy model is trained, for the current time step, combined with the state of the computing cluster obtained in real time, the task nodes to be scheduled and the actually allocated computing resources for each decision step are calculated in sequence.

2. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: In step (1), the job phase represented by the parent node must be completed before the job phase represented by the child node can be run.

3. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: In step (2), when the graph neural network is responsible for aggregating the directed acyclic graph information, it includes three levels of information aggregation and outputs three different types of embeddings, namely, information aggregation between computing task nodes, overall information aggregation of each directed acyclic graph task, and information aggregation between different application task requirements as a global embedding vector.

4. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: In step (2), the state is the task requirements and resource status of each operation stage of all application requirements under a decision step; the action is to select a certain operation stage of an application requirement and generate and allocate the corresponding upper limit of the number of resource nodes; The reward is the resource utilization efficiency over a period of time, expressed as the total time to complete the task.

5. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: In step (3), the reinforcement learning algorithm is used for training, with the goal of maximizing the expected cumulative reward, which is specifically expressed as: Among them, J(π θ ) represents the strategy π θ The performance evaluation function, R(s t ,a t ) is the reward function, which represents the reward obtained when taking action a in state s. It means that in strategy π θ Next state t and actions at The expectation of the distribution of .

6. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: During training, the discount factor is used to adjust the present value of future rewards, and the transfer function is used to predict the next state.

7. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 5 is characterized in that: The goal is to maximize the expected cumulative reward and satisfy a series of hard constraints, expressed as: in, A generalized operator representing two functions, p(a t |s t ) represents the policy network input state s under the feasibility constraint t Produce action a t The probability of θ (a t |s t ) represents the policy network input state s t Produce action a t The probability of t 、s t Respectively represent the action and state at time t, a t |s t Indicates in s t Take action a in state t ; F(s t ) represents the feasibility constraint, which is used to t Determine which actions are legal; feasibility constraints specifically include resource constraints, allocation constraints, and disaster recovery constraints.

8. The cloud resource dynamic scheduling method based on secure reinforcement learning according to claim 1 is characterized in that: In step (2), the decision network includes a node scoring network and an allocated computing resource quantity scoring network; In the node scoring network, for node v in job i, its score q iv Calculated by the following formula: q iv ≈q(e iv ,y i ,z) Among them, e iv is the embedding vector of the node, y i is the job-level embedding, z is the global embedding, and q() is a nonlinear transformation function used to map the embedding vector to a scalar value representing the priority of the node; In the scoring network for allocating computing resources, for job i, the score of resource quantity w li Calculated by the following formula: w li ≈w(y i ,z,l) where l is the parallelism limit specified for job i and w() is also a nonlinear transformation function that generates a score based on the job and global embeddings and the parallelism limit.

Images