Multidimensional resource task scheduling method based on reinforcement learning in computing power network environment

Abstract

The invention discloses a multi-dimensional resource task scheduling method based on reinforcement learning in a computing power network environment, and belongs to the field of multi-dimensional resource task scheduling. The method comprises the steps that a user in a cloud computing environment submits a request through a cloud computing platform, the cloud computing platform processes the user request, decomposes a task, establishes a reinforcement learning model of multi-dimensional resources, trains the reinforcement learning model of the multi-dimensional resources, and achieves multi-dimensional resource task scheduling based on reinforcement learning. Through the reinforcement learning method, according to the real-time states of the tasks and the resources, task allocation and resource scheduling of the multi-dimensional resources are dynamically performed, the response speed of the cloud computing platform is improved, the energy consumption of the cloud computing platform is reduced, and the resource utilization rate and the task completion time of the cloud computing platform are improved.

Description

Technical Field

[0001] The present invention relates to a method for scheduling multidimensional resource tasks based on reinforcement learning in a computing network environment, and belongs to the field of multidimensional resource task scheduling. Background Art

[0002] Cloud computing can meet the needs of high-performance computing power and massive storage space. It has the advantages of efficient deployment, easy expansion, security and reliability, convenient integration, and cost control. It can provide lower cost, elastic resources, convenient data sharing and access, and on-demand high-quality application services. In the cloud computing environment, there are a large number of tasks with different types of tasks. The resource requirements of various tasks are different. The cloud computing platform can achieve flexible service deployment and function management, but the efficiency is still limited. Resource management efficiency is one of the main constraints of the cloud computing platform. Improving resource management efficiency is an important way to improve the efficiency of the cloud computing platform. For multiple tasks in the cloud computing platform, efficient resource allocation and task scheduling methods can improve the task-oriented service quality and reduce power consumption.

[0003] In the cloud computing platform, efficient resource allocation is very critical for multi-task processing: through efficient computing resource allocation, the execution requirements of tasks can be met, resource utilization can be improved, and the requirements of reducing power consumption can be taken into account. It is necessary to provide flexible resource management and efficient task scheduling methods according to the characteristics, computing volume, duration and other requirements of different tasks. By optimizing the allocation between tasks and computing resources, appropriate computing resources can be provided for tasks, resource utilization can be improved and energy consumption can be reduced, so as to improve the efficiency and performance of the entire cloud computing platform. The task scheduling method in the cloud computing environment is of great significance to the field of cloud computing. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a task scheduling method for multi-dimensional resources based on reinforcement learning in a computing network environment. Through the reinforcement learning method, according to the real-time status of tasks and resources, multi-dimensional resource task allocation and resource scheduling are dynamically performed to improve the response speed of the cloud computing platform, reduce the energy consumption of the cloud computing platform, and improve the resource utilization rate and task completion time of the cloud computing platform. The method specifically includes the following steps:

[0005] Step 1: Users in the cloud computing environment submit requests through the cloud computing platform;

[0006] User requests in a cloud computing environment are multi-dimensional requests, including: creating a virtual machine, uploading a file, and executing a task;

[0007] Step 2: The cloud computing platform processes user requests;

[0008] After receiving the user request, the cloud computing platform parses and identifies the request to determine the type and requirements of the user request;

[0009] Step 3: Decomposition of tasks by the cloud computing platform;

[0010] The cloud computing platform breaks down the tasks to be performed into multiple subtasks and determines the resource needs and time requirements for each subtask;

[0011] Step 4: Multi-dimensional resource task scheduling based on reinforcement learning;

[0012] The subtasks decomposed in step 3 are allocated to computing resources. Based on the reinforcement learning method, task allocation and resource scheduling are dynamically performed according to the real-time status of tasks and resources to realize task processing. Specifically, the following substeps are included:

[0013] Step 4.1: Establish a reinforcement learning model for multi-dimensional resources;

[0014] Determine the state space: The state space includes the attributes and states of the task model to be processed and the available resource environment model;

[0015] The task model to be processed includes: the type, size and processing time of the task;

[0016] The resource environment model includes: the type, specification, quantity, duration and utilization of resources;

[0017] Determine the action space: The action space includes the decisions on how to allocate tasks to different resources;

[0018] Determine the reward function: The reward function is the reward value obtained by the agent when it takes a certain action and observes a certain state. The reward function is used to evaluate the quality of each decision;

[0019] Step 4.2: Train the reinforcement learning model for multi-dimensional resources;

[0020] Through the parameters of the task model to be processed and the available resource environment model, the state space is mapped to the action space, and the reinforcement learning model of multi-dimensional resources is optimized according to the reward function, including:

[0021] Initialize the Q table: The Q table stores the Q value corresponding to each state and action. Each element Q(s,a) in the Q table is the cumulative reward value that can be obtained by taking a certain action in the current state;

[0022] Iterative Q-table: During the training process of the reinforcement learning model of multi-dimensional resources, the agent continuously updates the corresponding Q value in the Q-table by observing the state of the resource environment model and the actions taken, so that the Q value can accurately reflect the cumulative reward value that can be obtained by taking a certain action in the current state. According to the current state and the Q value in the Q-table, the agent selects the next action, and updates the corresponding Q value in the Q-table according to the reward value and the next state fed back by the resource environment model;

[0023] Step 4.3: Task scheduling;

[0024] Perform corresponding task scheduling according to the action space mapped out in step 4.2, and select the corresponding virtual machine to assign tasks;

[0025] Monitor the utilization of various resources in the cloud computing platform, including CPU utilization, memory utilization, storage utilization, and network bandwidth utilization, and determine the usage of resources;

[0026] Monitor the status and performance of virtual machines in the cloud computing platform, including: operating status, CPU usage, memory usage, disk IO, and determine the operating status and performance load of virtual machines;

[0027] Select the optimal task scheduling strategy based on the current state and the Q value in the Q table;

[0028] As a preferred method, the ε-greedy method is adopted to introduce randomness. For the current state s, a random action a is selected with probability ε, and an action a* with the maximum Q value is selected with probability 1-ε, so as to avoid falling into the local optimum, balance the relationship between exploration and utilization, and have better global search capabilities.

[0029] Execute the task scheduling strategy, apply the selected action to the cloud computing platform, and obtain the next state and reward value;

[0030] Step 4.4 Update the Q table;

[0031] According to the deep learning method, the Q value in the Q table is updated so that the Q value more accurately reflects the reward value obtained by taking a certain action in the current state;

[0032] Repeat steps 4.3 and 4.4 until the task scheduling process is completed;

[0033] The method also includes step five, according to the multi-dimensional resource task scheduling based on reinforcement learning in step four, when the task is completed, the computing resources return the result to the cloud computing platform, and the cloud computing platform returns the result to the user, thus realizing the computing process of the cloud platform.

[0034] Beneficial Effects

[0035] 1. The present invention provides a task scheduling method for multi-dimensional resources based on reinforcement learning in a computing network environment, which integrates the multi-dimensional resources of a cloud computing platform, can schedule tasks more accurately, improve task scheduling efficiency, shorten task completion time, and increase resource utilization.

[0036] 2. The present invention provides a multi-dimensional resource task scheduling method based on reinforcement learning in a computing network environment, which can dynamically adjust the task scheduling strategy according to the changes of the cloud computing platform, shorten the task scheduling time, improve the overall efficiency and productivity of the cloud computing platform, and adapt to different application scenarios.

[0037] 3. The present invention provides a task scheduling method for multi-dimensional resources based on reinforcement learning in a computing network environment. By modeling the multi-dimensional resources of the cloud computing environment, a state space, action space and reward function suitable for multi-dimensional resources are constructed, and the optimal task scheduling strategy is obtained through the reinforcement learning method. It can better utilize the multi-dimensional resources of the cloud computing environment and improve resource utilization.

[0038] 4. The present invention provides a task scheduling method for multi-dimensional resources based on reinforcement learning in a computing network environment. By adaptively optimizing the task scheduling strategy, the task scheduling strategy is dynamically adjusted according to the changes in the cloud computing environment to adapt to different application scenarios. Through precise task scheduling, it ensures the full utilization of resources, avoids resource bottlenecks, and improves the stability and reliability of the cloud computing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of a task scheduling method for multi-dimensional resources based on reinforcement learning in a computing power network environment of the present invention;

[0040] Figure 2 This is a flow chart of the reinforcement learning method in step three of the present invention;

[0041] Figure 3 A schematic diagram showing the time required for the method of the present invention and other methods in the embodiments;

[0042] Figure 4 Schematic diagram for comparing resource utilization between the method of the present invention and other methods in the embodiment. DETAILED DESCRIPTION

[0043] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below with reference to the accompanying drawings and examples.

[0044] Embodiment 1:

[0045] In the cloud computing scenario, a large number of computing tasks and resources are usually involved, and there are complex dependencies between tasks. The task scheduling method of multi-dimensional resources based on reinforcement learning in a computing network environment of the present invention is applied to assign tasks to available computing nodes and optimize system performance indicators including task execution time, resource utilization or energy efficiency, including the following steps:

[0046] The multi-resource task scheduling method in cloud computing environment based on reinforcement learning mainly includes the following steps:

[0047] Step 1: Users in the cloud computing environment submit requests through the cloud computing platform;

[0048] User requests in a cloud computing environment are multi-dimensional requests, including: creating a virtual machine, uploading a file, and executing a task;

[0049] Step 2: The cloud computing platform processes user requests;

[0050] After receiving the user request, the cloud computing platform parses and identifies the request to determine the type and requirements of the user request;

[0051] Step 3: Decomposition of tasks by the cloud computing platform;

[0052] The cloud computing platform breaks down the tasks to be performed into multiple subtasks and determines the resource needs and time requirements for each subtask;

[0053] Step 4: Multi-dimensional resource task scheduling based on reinforcement learning;

[0054] In the embodiment, the Q-learning method in the reinforcement learning algorithm is used. At a certain moment, in the state s (s∈S), taking action a (a∈A) can obtain the expected benefit Q(s,a). The environment will feedback the corresponding reward r according to the action of the intelligent agent, and the environment state and action are constructed into a Q table to store the Q value, and the action that can obtain the maximum benefit is selected according to the Q value;

[0055] The multi-dimensional resource task scheduling problem is transformed into a Markov decision process through the agent, environment state, reward, and action. Each grid is used as a state S t ,Π(a|s) is the probability of taking action a strategy in state s and choosing action a in state s to transition to the next state s′ The reward R(s′|s,a) of taking action a in state s to transfer to s′ is the expected value of the strategy with the maximum cumulative reward as shown in formula (1):

[0056]

[0057] Where π is the policy, which is a function that maps states to actions. E is the expected operator, which means averaging over all possible trajectories. t is the time step, which starts at 0 and gradually increases to H. γ is the discount factor, which is used to measure the importance of future rewards and ranges from 0 to 1. R(S t ,A t ,S t+1 ) indicates that in state S t Execute action A t Then, transfer to state S t+1 Instant rewards received.

[0058] Adopt Q-learning method and perform offline learning through temporal difference method;

[0059] The optimal decision sequence of the Markov decision process is determined by the Bellman equation, as shown in formula (2):

[0060]

[0061] Among them, V π (s) is the state value function, which is used to evaluate the quality of the current state. The value of each state is determined not only by the current state but also by the subsequent state. t It represents the utility at time step t, which is used to express the preference or satisfaction for a certain result or state. V(s) is the state value function of the current s.

[0062] The optimal cumulative expectation, that is, the optimal function V * (s)=max π V π (s), as shown in formula (3):

[0063]

[0064] Among them, V*(s) represents the optimal value function, which represents the value function under a given initial state s and optimal strategy.

[0065] Q(s,a) state action value function q π (s,a) is as shown in formula (4):

[0066] q π (s, a) = E π [r t+1 +γr t+2 +γ 2 r t+3 +…|A t =a,S t =s]E π [G t|A t =a,S t =s] (4)

[0067] Among them, G t is the total discounted reward starting at time t.

[0068] Formula (4) shows the influence of the decay value of γ on the Q function. The closer γ is to 1, the more far-sighted it is, and the more it will focus on the value of subsequent states. When γ is close to 0, it will become short-sighted and only consider the impact of current interests. When γ changes from 0 to 1, it will increasingly consider the impact of subsequent returns.

[0069] like Figure 2 As shown in FIG. 1 , the structure diagram of the reinforcement learning method Q-leaming method used in the embodiment, wherein α is the learning rate and γ is the reward decay coefficient, and the time difference method is used for updating. By determining the Q value, reinforcement learning is performed, that is, the Q table is updated, as shown in formula (5):

[0070] Q(s, a)←Q(s, a)+α[r+max a ′Q(s′,a′)-Q(s,a)] (5)

[0071] Among them, α is the learning rate, which controls the weight of new information at each update. A higher learning rate means faster learning, but may lead to instability; a lower learning rate means more stable learning, but slower learning. r represents the immediate reward obtained after executing action a in the current state s. Q(s′, a′) represents the Q value of selecting action a′ in the next state s′. max a′ Q(s′, a′) represents the maximum Q value among all possible actions a′ in the next state s′.

[0072] The maximum value Q(s′, a′) in the next state s′ is selected by multiplying the decay γ plus the real return value to obtain the most realistic Q, and the Q(s, a) in the past Q table is used as the Q estimate;

[0073] The subtasks decomposed in step 3 are allocated to computing resources. Based on the Q-learning method, task allocation and resource scheduling are dynamically performed according to the real-time status of tasks and resources to realize task processing. The specific sub-steps include the following:

[0074] Step 4.1: Establish a reinforcement learning model for multi-dimensional resources;

[0075] Determine the state space: The state space includes the attributes and states of the task model to be processed and the available resource environment model;

[0076] The task model to be processed includes: the type, size and processing time of the task;

[0077] The resource environment model includes: the type, specification, quantity, duration and utilization of resources;

[0078] In the embodiment, the status is the size, bandwidth, and duration of the task to be assigned, and the processing power, size, memory, and bandwidth of the assignable virtual machine;

[0079] Determine the action space: The action space includes the decisions on how to allocate tasks to different resources;

[0080] In the embodiment, the action is to assign the task to a specific virtual machine;

[0081] Determine the reward function: The reward function is the reward value obtained by the agent when it takes a certain action and observes a certain state. The reward function is used to evaluate the quality of each decision;

[0082] Step 4.2: Train the reinforcement learning model for multi-dimensional resources;

[0083] Through the parameters of the task model to be processed and the available resource environment model, the state space is mapped to the action space, and the reinforcement learning model of multi-dimensional resources is optimized according to the reward function, including:

[0084] Initialize the Q table: The Q table stores the Q value corresponding to each state and action. Each element Q(s, a) in the Q table is the cumulative reward value that can be obtained by taking a certain action in the current state.

[0085] Iterative Q-table: During the training process of the reinforcement learning model of multi-dimensional resources, the agent continuously updates the corresponding Q value in the Q-table by observing the state of the resource environment model and the actions taken, so that the Q value can accurately reflect the cumulative reward value that can be obtained by taking a certain action in the current state. According to the current state and the Q value in the Q-table, the agent selects the next action, and updates the corresponding Q value in the Q-table according to the reward value and the next state fed back by the resource environment model;

[0086] In the embodiment, at each time step, the intelligent body will select the next action according to the current state and the Q value in the Q table, and update the corresponding Q value in the Q table according to the reward value fed back by the environment and the next state;

[0087] Step 4.3: Task scheduling;

[0088] Perform corresponding task scheduling according to the action space mapped out in step 4.2, and select the corresponding virtual machine to assign tasks;

[0089] Monitor the utilization of various resources in the cloud computing platform, including CPU utilization, memory utilization, storage utilization, and network bandwidth utilization, and determine the usage of resources;

[0090] Monitor the status and performance of virtual machines in the cloud computing platform, including: operating status, CPU usage, memory usage, disk IO, and determine the operating status and performance load of virtual machines;

[0091] Select the optimal task scheduling strategy based on the current state and the Q value in the Q table;

[0092] In the embodiment, the ε-greedy method is adopted to introduce randomness. For the current state s, a random action a is selected with probability ε, and an action a* with the maximum Q value is selected with probability 1-ε, so as to avoid falling into the local optimum, balance the relationship between exploration and utilization, and have better global search capability.

[0093] Execute the task scheduling strategy, apply the selected action to the cloud computing platform, and obtain the next state and reward value;

[0094] Step 4.4 Update the Q table;

[0095] According to the deep learning method, the Q value in the Q table is updated so that the Q value more accurately reflects the reward value obtained by taking a certain action in the current state;

[0096] Repeat steps 4.3 and 4.4 until the task scheduling process is completed;

[0097] In the embodiment, Matlab software is used to simulate the task scheduling scenario in the cloud computing environment. The total time spent on completing the same number of task scheduling is as follows: Figure 3 As shown in the figure, the resource utilization of completing the same number of task scheduling is as follows Figure 4 As shown, the random allocation algorithm and the particle swarm optimization algorithm are used as comparisons. Figure 3 As shown in FIG. 1 , the total completion time required for the three methods to complete all tasks and allocate resources to virtual machines is compared. The method of the present invention outperforms the random allocation algorithm by 57.97% and the particle swarm optimization algorithm by 46.92%. Figure 4 As shown, the three methods complete all tasks and allocate resources to virtual machines. In terms of CPU utilization, memory utilization, computing utilization, and network bandwidth utilization, the method of the present invention is 69.65% better than the random allocation algorithm and 33.68% better than the particle swarm optimization algorithm. The method of the present invention shortens the time of multi-dimensional resource task scheduling in the cloud computing environment and improves the resource utilization of the system.

[0098] The method also includes step five, according to the multi-dimensional resource task scheduling based on reinforcement learning in step four, when the task is completed, the computing resources return the result to the cloud computing platform, and the cloud computing platform returns the result to the user, thus realizing the computing process of the cloud platform.

[0099] The specific description above further illustrates the purpose, technical solutions and beneficial effects of the invention in detail. It should be understood that the above is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multi-dimensional resource task scheduling method based on reinforcement learning in a computing network environment, characterized by: The following steps are involved: Step 1: Users in the cloud computing environment submit requests through the cloud computing platform; User requests in a cloud computing environment are multi-dimensional requests, including: creating a virtual machine, uploading a file, and executing a task; Step 2: The cloud computing platform processes user requests; After receiving the user request, the cloud computing platform parses and identifies the request to determine the type and requirements of the user request; Step 3: Decomposition of tasks by the cloud computing platform; The cloud computing platform breaks down the tasks to be performed into multiple subtasks and determines the resource needs and time requirements for each subtask; Step 4: Multi-dimensional resource task scheduling based on reinforcement learning; The subtasks decomposed in step 3 are allocated to computing resources. Based on the reinforcement learning method, task allocation and resource scheduling are dynamically performed according to the real-time status of tasks and resources to realize task processing.

2. The method for scheduling multi-dimensional resources based on reinforcement learning in a computing network environment according to claim 1, characterized in that: Step 4: Multi-dimensional resource task scheduling based on reinforcement learning specifically includes the following sub-steps: Step 4.1: Establish a reinforcement learning model for multi-dimensional resources; Determine the state space: The state space includes the attributes and states of the task model to be processed and the available resource environment model; The task model to be processed includes: the type, size and processing time of the task; The resource environment model includes: the type, specification, quantity, duration and utilization of resources; Determine the action space: The action space includes the decisions on how to allocate tasks to different resources; Determine the reward function: The reward function is the reward value obtained by the agent when it takes a certain action and observes a certain state. The reward function is used to evaluate the quality of each decision; Step 4.2: Train the reinforcement learning model for multi-dimensional resources; Through the parameters of the task model to be processed and the available resource environment model, the state space is mapped to the action space, and the reinforcement learning model of multi-dimensional resources is optimized according to the reward function, including: Initialize the Q table: The Q table stores the Q value corresponding to each state and action. Each element Q(s,a) in the Q table is the cumulative reward value that can be obtained by taking a certain action in the current state; Iterative Q-table: During the training process of the reinforcement learning model of multi-dimensional resources, the agent continuously updates the corresponding Q value in the Q-table by observing the state of the resource environment model and the actions taken, so that the Q value can accurately reflect the cumulative reward value that can be obtained by taking a certain action in the current state. According to the current state and the Q value in the Q-table, the agent selects the next action, and updates the corresponding Q value in the Q-table according to the reward value and the next state fed back by the resource environment model; Step 4.3: Task scheduling; Perform corresponding task scheduling according to the action space mapped out in step 4.2, and select the corresponding virtual machine to assign tasks; Monitor the utilization of various resources in the cloud computing platform, including CPU utilization, memory utilization, storage utilization, and network bandwidth utilization, and determine the usage of resources; Monitor the status and performance of virtual machines in the cloud computing platform, including: operating status, CPU usage, memory usage, disk IO, and determine the operating status and performance load of virtual machines; Select the optimal task scheduling strategy based on the current state and the Q value in the Q table; Step 4.4 Update the Q table; According to the deep learning method, the Q value in the Q table is updated so that the Q value more accurately reflects the reward value obtained by taking a certain action in the current state; Repeat steps 4.3 and 4.4 until the task scheduling process is completed.

3. The method for scheduling multi-dimensional resources based on reinforcement learning in a computing network environment according to claim 1, characterized in that: The method also includes step five, according to the multi-dimensional resource task scheduling based on reinforcement learning in step four, when the task is completed, the computing resources return the result to the cloud computing platform, and the cloud computing platform returns the result to the user, thus realizing the computing process of the cloud platform.

4. The method for scheduling multi-dimensional resources based on reinforcement learning in a computing network environment according to claim 1, characterized in that: In the task scheduling of step 4.3, the ε-greedy method is used to introduce randomness. For the current state s, a random action a is selected with probability ε, and an action a* with the maximum Q value is selected with probability 1-ε. This avoids falling into the local optimum, balances the relationship between exploration and utilization, and has better global search capabilities.

Images