Cloud resource elastic scheduling optimization method based on reinforcement learning

Abstract

The invention relates to the technical field of cloud computing, and discloses a cloud resource flexible scheduling optimization method based on reinforcement learning, and the method comprises the steps: obtaining the dynamic demand data of tasks in a cloud computing platform, including but not limited to the type, priority, historical resource consumption, current system load and other information of the tasks; carrying out modeling on the obtained task; carrying out cloud resource scheduling by adopting a reinforcement learning algorithm; and a scheduling strategy is optimized through a feedback mechanism, and multi-objective optimization scheduling is realized by updating a state-action value function. According to the method, the dynamic demand data of the tasks in the cloud computing platform are acquired, the state space and the action space of the tasks are defined, the cloud resource scheduling strategy is optimized through the reinforcement learning algorithm, and the task scheduling is dynamically adjusted through the multi-dimensional reward function, so that the task allocation can be intelligently optimized under the real-time task load and resource constraint, and the task scheduling efficiency is improved. The resource utilization efficiency is improved, the task response time is shortened, and the system energy consumption is reduced.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, specifically to a cloud resource elastic scheduling optimization method based on reinforcement learning. Background Technology

[0002] With the rapid development of cloud computing technology, cloud computing platforms have become the primary execution environment for various large-scale computing tasks. However, current cloud resource scheduling systems still face many challenges when dealing with complex task requirements and changing resource conditions. For example, dynamic task requirements, dynamic changes in platform resources, and sudden fluctuations in load all significantly impact cloud resource scheduling. Traditional scheduling methods often rely on preset rules or static strategies, which cannot adapt to real-time changes in task characteristics and resource load. Therefore, designing an efficient and intelligent scheduling algorithm to address these challenges has become a crucial issue in optimizing current cloud computing platforms.

[0003] Most existing scheduling methods are based on fixed scheduling policies or heuristic algorithms. While they can solve the problem to some extent, they typically cannot provide optimal resource allocation in dynamic task and resource environments. Reinforcement learning, as a technique that learns optimal decisions through interaction with the environment, possesses excellent dynamic adaptability and learning capabilities, making it suitable for solving this type of problem. In recent years, reinforcement learning-based scheduling methods have received widespread attention; however, how to balance multi-objective optimization while ensuring efficient scheduling remains a pressing challenge.

[0004] In view of this, we propose a cloud resource elastic scheduling optimization method based on reinforcement learning. Summary of the Invention

[0005] The purpose of this invention is to provide a cloud resource elastic scheduling optimization method based on reinforcement learning to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A reinforcement learning-based method for optimizing elastic scheduling of cloud resources, comprising:

[0008] Obtain dynamic demand data for tasks in the cloud computing platform, including but not limited to information such as task type, priority, historical resource consumption, and current system load;

[0009] The acquired task is modeled, and the state space and action space of the task are defined. The state space includes various characteristics of the task, such as task size and resource requirements, while the action space includes operations such as resource allocation, task scheduling, and load balancing.

[0010] Reinforcement learning algorithms are used for cloud resource scheduling. The reinforcement learning algorithm dynamically allocates resources based on the real-time state of the task and uses a reward function to measure the value of each state-action pair.

[0011] The scheduling strategy is optimized through a feedback mechanism, and multi-objective optimized scheduling is achieved by updating the state-action value function.

[0012] Preferably, the reward function is:

[0013]

[0014] in, Indicates resource utilization efficiency. Indicates the task response time. Indicates energy consumption. , , These are the weighting coefficients for each indicator.

[0015] Preferably, the reinforcement learning algorithm adopts the Q-learning algorithm or the deep reinforcement learning algorithm, specifically by using the feedback information of historical states and actions to iteratively update the action value function Q(s,a) to optimize the scheduling strategy.

[0016] Preferably, the iterative update action value function Q(s,a) is:

[0017]

[0018] in, For learning rate, As a discount factor, To start from the current state Execute action The reward obtained later For the next state Corresponding actions The maximum expected reward.

[0019] Preferably, the reward function The calculation method incorporates multi-dimensional optimization objectives, specifically including resource utilization efficiency. Task response time and system energy consumption .

[0020] Preferably, the resource utilization efficiency Calculated using the following formula:

[0021]

[0022] in, This indicates the actual amount of resources allocated to the task. This represents the total amount of available resources on the cloud platform in the current state.

[0023] Preferably, the task response time Calculated using the following formula:

[0024]

[0025] in, Indicates the task queuing time. Indicates the task processing time.

[0026] Preferably, the system energy consumption Calculated using the following formula:

[0027]

[0028] in, Let be the power consumption of the i-th resource component. The usage time of the i-th resource component.

[0029] Preferably, the reinforcement learning algorithm continuously learns during the task scheduling process to achieve adaptive scheduling for new task types and sudden loads, and adopts an adaptively adjusted learning rate to cope with different task and load changes.

[0030] By employing the above technical solution, this invention provides a cloud resource elastic scheduling optimization method based on reinforcement learning. It possesses at least the following beneficial effects:

[0031] (1) This invention obtains dynamic demand data of tasks in the cloud computing platform, defines the state space and action space of tasks, optimizes cloud resource scheduling strategy through reinforcement learning algorithm, and dynamically adjusts task scheduling through multi-dimensional reward function. Under real-time task load and resource constraints, it can intelligently optimize task allocation, improve resource utilization efficiency, reduce task response time, and reduce system energy consumption.

[0032] (2) This invention adaptively adjusts the scheduling strategy according to the real-time status of the task and the changes in system resources, performs multi-objective optimization through the reward function, efficiently extracts task features through a deep learning model, and optimizes the scheduling strategy through a real-time feedback mechanism. Attached Figure Description

[0033] The accompanying drawings, which are provided to further illustrate the invention, constitute a part of this application:

[0034] Figure 1 This is a schematic diagram of the overall process of the cloud resource elastic scheduling optimization method based on reinforcement learning of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] Please see Figure 1 As shown, this invention proposes a cloud resource elastic scheduling optimization method based on reinforcement learning, the method comprising:

[0037] S1. Obtain dynamic demand data for tasks in the cloud computing platform, including but not limited to information such as task type, priority, historical resource consumption, and current system load;

[0038] S2. Model the acquired task and define the task's state space and action space. The state space includes various characteristics of the task, such as task size and resource requirements, while the action space includes operations such as resource allocation, task scheduling, and load balancing.

[0039] S3. Reinforcement learning algorithm is used for cloud resource scheduling. The reinforcement learning algorithm dynamically allocates resources based on the real-time state of the task and measures the value of each state-action pair through a reward function.

[0040] S4. Optimize the scheduling strategy through the feedback mechanism and achieve multi-objective optimization scheduling by updating the state-action value function.

[0041] It should be noted that the cloud computing platform includes multiple resource pools, which respectively include a computing resource pool, a storage resource pool, and a network bandwidth resource pool. The state of each resource pool is described by the following equation:

[0042]

[0043] in, This represents the overall resource status of the resource pool. Let be the available quantity of the i-th resource.

[0044] It should be noted that this invention can be applied to various cloud computing scenarios, including but not limited to public cloud, private cloud, and hybrid cloud.

[0045] It should be noted that the task's state space includes factors such as task type, task priority, and historical task execution data. The task scheduling strategy is dynamically updated using the following formula:

[0046]

[0047] in, In the state The optimal action to choose from.

[0048] It should be noted that the feature extraction for the task using a deep learning network model is calculated using the following formula:

[0049]

[0050] in, For the target value, For predicted values, For the parameters of the model, This represents the number of samples.

[0051] Specifically, a neural network is used as an approximator for the Q-value function. The network model is trained to learn the complex relationship between the task's state and actions. The loss function used is the mean squared error loss function. The network parameters are optimized using the backpropagation algorithm.

[0052] It should be noted that the scheduling strategy is adaptively adjusted based on the real-time needs of the task, the platform resource status, and historical scheduling data, and the scheduling order and resource allocation strategy of the task are adjusted through an optimized reward function.

[0053] Specifically, the API interface is used to collect information on the type, priority, historical resource consumption, and current system load of each task in real time, and this data is used as the status input for the task.

[0054] In some embodiments, the reward function is:

[0055]

[0056] in, Indicates resource utilization efficiency. Indicates the task response time. Indicates energy consumption. , , These are the weighting coefficients for each indicator.

[0057] It should be noted that the reinforcement learning algorithm mentioned above adopts the Q-learning algorithm or the deep reinforcement learning algorithm. Specifically, it utilizes the feedback information of historical states and actions to iteratively update the action value function Q(s,a), thereby optimizing the scheduling strategy.

[0058] In some embodiments, the iterative update action value function Q(s,a) is:

[0059]

[0060] in, For learning rate, As a discount factor, To start from the current state Execute action The reward obtained later For the next state Corresponding actions The maximum expected reward.

[0061] Specifically, during each task scheduling, based on the current state... The optimal action is selected based on the maximum Q value to ensure maximum resource utilization, minimum task response time, and optimal energy consumption.

[0062] In some embodiments, the reward function The calculation method incorporates multi-dimensional optimization objectives, specifically including resource utilization efficiency. Task response time and system energy consumption .

[0063] In some embodiments, the resource utilization efficiency Calculated using the following formula:

[0064]

[0065] in, This indicates the actual amount of resources allocated to the task. This represents the total amount of available resources on the cloud platform in the current state.

[0066] In some embodiments, the task response time Calculated using the following formula:

[0067]

[0068] in, Indicates the task queuing time. Indicates the task processing time.

[0069] In some embodiments, the system energy consumption Calculated using the following formula:

[0070]

[0071] in, Let be the power consumption of the i-th resource component. The usage time of the i-th resource component.

[0072] In some embodiments, the reinforcement learning algorithm achieves adaptive scheduling for new task types and sudden loads by continuously learning during task scheduling, and uses an adaptively adjusted learning rate to cope with different task and load changes:

[0073] in, The initial learning rate, As the attenuation factor, For time steps.

[0074] The above embodiments of the present invention have the following beneficial effects: The cloud resource elastic scheduling optimization method based on reinforcement learning of the present invention can realize the dynamic adaptation and continuous optimization of the scheduling strategy by adopting the cloud resource scheduling optimization method based on reinforcement learning. It can intelligently adjust the resource allocation scheme according to the real-time changes of task load and system resource status, thereby effectively improving resource utilization and system processing efficiency.

[0075] By constructing task state space and action space models and designing multi-dimensional reward functions, the system can not only optimize task response time and computing resource allocation, but also maintain scheduling stability and service quality in task-intensive or sudden load scenarios, achieving multi-objective collaborative optimization of resource scheduling.

[0076] By introducing Q-learning or deep reinforcement learning algorithms for policy training and real-time updates, an automatic learning mechanism for resource scheduling policies without human intervention is realized, which solves the problem of fixed scheduling policies that cannot adapt to dynamic changes in resources in the background technology, and significantly improves the level of intelligent management in the cloud computing environment.

[0077] By comprehensively evaluating indicators such as task processing efficiency, system energy consumption, and resource load through a multi-dimensional reward function, a dynamic balance between energy consumption reduction and response speed optimization is achieved, solving the problems of lack of multi-objective trade-off mechanism and serious resource waste in traditional scheduling methods in the background technology.

[0078] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0079] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A cloud resource elastic scheduling optimization method based on reinforcement learning, characterized in that, The method includes: Obtain dynamic demand data for tasks in the cloud computing platform, including but not limited to information such as task type, priority, historical resource consumption, and current system load; The acquired task is modeled, and the state space and action space of the task are defined. The state space includes various characteristics of the task, such as task size and resource requirements, while the action space includes operations such as resource allocation, task scheduling, and load balancing. Reinforcement learning algorithms are used for cloud resource scheduling. The reinforcement learning algorithm dynamically allocates resources based on the real-time state of the task and uses a reward function to measure the value of each state-action pair. The scheduling strategy is optimized through a feedback mechanism, and multi-objective optimized scheduling is achieved by updating the state-action value function.

2. The cloud resource elastic scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, The reward function is: ； in, Indicates resource utilization efficiency. Indicates the task response time. Indicates energy consumption. , , These are the weighting coefficients for each indicator.

3. The cloud resource elastic scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning algorithm mentioned above employs either the Q-learning algorithm or a deep reinforcement learning algorithm. Specifically, it utilizes feedback information from historical states and actions to iteratively update the action value function Q(s,a), thereby optimizing the scheduling strategy.

4. The cloud resource elastic scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, The iterative update action value function Q(s,a) is: ； in, For learning rate, As a discount factor, To start from the current state Execute action The reward obtained later For the next state Corresponding actions The maximum expected reward.

5. The cloud resource elastic scheduling optimization method based on reinforcement learning according to claim 2, characterized in that, The reward function The calculation method incorporates multi-dimensional optimization objectives, specifically including resource utilization efficiency. Task response time and system energy consumption .

6. The cloud resource elastic scheduling optimization method based on reinforcement learning according to claim 5, characterized in that, resource utilization efficiency Calculated using the following formula: ； in, This indicates the actual amount of resources allocated to the task. This represents the total amount of available resources on the cloud platform in the current state.

7. The cloud resource elastic scheduling optimization method based on reinforcement learning according to claim 5, characterized in that, The task response time Calculated using the following formula: ； in, Indicates the task queuing time. Indicates the task processing time.

8. The cloud resource elastic scheduling optimization method based on reinforcement learning according to claim 5, characterized in that, The system energy consumption Calculated using the following formula: ； in, Let be the power consumption of the i-th resource component. The usage time of the i-th resource component.

9. The cloud resource elastic scheduling optimization method based on reinforcement learning according to claim 1, characterized in that, The reinforcement learning algorithm achieves adaptive scheduling for new task types and sudden loads by continuously learning during task scheduling, and uses an adaptively adjusted learning rate to cope with different task and load changes.

Images