Cloud computing resource scheduling method and system based on elastic scaling two-layer scheduling framework

Abstract

The present invention discloses a cloud computing resource scheduling method and system based on an elastically scalable two-tier scheduling framework. This method relates to the field of cloud computing resource scheduling and establishes a two-tier scheduling framework based on a three-tier cloud computing market. The two-tier scheduling framework includes a task allocation scheduler and a virtual machine automatic scaling scheduler. The task allocation scheduler determines the state, action, and reward function of a first agent by collecting task characteristics and virtual machine usage data from the current cloud computing platform. The virtual machine automatic scaling scheduler determines the state, action, and reward function of a second agent by collecting workload and resource utilization data from the current cloud computing platform. A two-tier scheduling algorithm is used to resolve task allocation decisions for the first agent, and a two-tier scheduling algorithm is used to resolve virtual machine adjustment decisions for the second agent. This innovative approach proposes an elastically scalable two-tier scheduling framework based on deep reinforcement learning to achieve efficient task scheduling and elastic resource allocation.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing resource scheduling, and in particular to a cloud computing resource scheduling method and system based on an elastically scalable two-layer scheduling framework. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Due to the reliability and flexibility of cloud computing, more and more companies are migrating their applications to the cloud. These companies lease appropriate cloud resources from Infrastructure as a Service (IaaS) providers, enabling them to build their own software application services at a lower cost. These Software as a Service (SaaS) providers play a dual role in the cloud market. As service providers for end users, SaaS applications must process service requests and execute user tasks to meet diverse service needs. On the other hand, as users of IaaS providers, they pay to lease virtual machine instances from the cloud platform to deploy their SaaS applications. In this business model, SaaS providers aim to ensure the quality of their SaaS application services while minimizing the cost of leasing virtual machine instances. Therefore, effectively selecting virtual machine leasing plans and allocating user-submitted tasks to appropriate resources presents a significant challenge for SaaS providers.

[0004] Because transactional application workloads vary dynamically over time, the demand for cloud resources also fluctuates accordingly. When a SaaS provider leases fewer resources than are needed to handle the application workload, this results in underprovisioning, which can degrade application performance and impact service quality. Conversely, when a SaaS provider uses more resources than the application workload requires, this leads to overprovisioning, resulting in unnecessary costs. Due to the elastic nature of cloud computing, the resources used by SaaS applications can be flexibly scaled up or down.

[0005] Existing research can be broadly categorized into two main areas: task scheduling and resource elastic scaling. Machine learning algorithms address task scheduling and resource allocation in cloud computing by learning and predicting task patterns, optimizing resource allocation strategies, and dynamically adapting to environmental changes. Resource elastic scaling is a key feature of cloud computing, enabling cloud platforms or SaaS applications to flexibly and dynamically allocate or release resources in response to fluctuations in user demand. Elastic resource provisioning can be categorized into vertical scaling and horizontal scaling, depending on the scaling approach. Since horizontal scaling addresses common resource issues in SaaS applications, both rule-based or threshold-based resource scaling strategies and prediction-based resource scaling strategies have been proposed. Furthermore, platforms such as Amazon AWS and Google Cloud offer Auto Scaling, which automatically increases or decreases the number of VM instances based on the load on the compute instances. Amazon AWS's Auto Scaling feature automatically adjusts the number of Amazon EC2 instances based on load while ensuring high availability and performance. Google Cloud's task scheduler rationally allocates tasks to the most appropriate VM instances based on resource demand and load. However, these technologies often operate independently for resource elastic scaling and do not work well with task scheduling.

[0006] Cloud computing environments are highly dynamic and complex, with factors such as task load, resource requirements, system failures, and resource availability constantly changing. Task scheduling typically involves multiple optimization objectives, such as resource utilization, response time, task completion time, energy efficiency, and cost control. Traditional scheduling methods often struggle to strike a balance between these multiple objectives. On the one hand, cloud computing resource pools are often dynamically changing; on the other hand, task resource requirements, execution times, and priorities can all be uncertain. Furthermore, elastic scaling is a core feature of cloud platforms, allowing resources to be automatically adjusted based on workload fluctuations. Coordinating this with autoscaling mechanisms is a key issue in task scheduling in cloud computing. Summary of the Invention

[0007] To overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a cloud computing resource scheduling method and system based on an elastically scalable two-layer scheduling framework, and innovatively proposes an elastically scalable two-layer scheduling framework based on deep reinforcement learning, which realizes efficient task scheduling and elastic resource allocation.

[0008] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0009] In a first aspect, the present invention provides a cloud computing resource scheduling method based on an elastically scalable two-tier scheduling framework, comprising:

[0010] A two-tier scheduling framework is established based on a three-tier cloud computing market; the three-tier cloud computing market includes infrastructure service providers, software operation service providers, and end users; the two-tier scheduling framework includes a task allocation scheduler and a virtual machine automatic scaling scheduler;

[0011] The task allocation scheduler defines a first agent and determines the state, action, and reward function of the first agent by collecting task characteristics and virtual machine usage data of the current cloud computing platform;

[0012] The virtual machine auto-scaling scheduler defines a second agent and determines the state, action, and reward function of the second agent by collecting workload and resource utilization data of the current cloud computing platform;

[0013] A two-layer scheduling algorithm is used to solve the task allocation decision of the first intelligent agent, and a two-layer scheduling algorithm is used to solve the virtual machine adjustment decision of the second intelligent agent.

[0014] Further technical solution, the task response time in the two-layer scheduling framework is expressed as:

[0015]

[0016] in, represents the task response time, Indicates the task execution time, Indicates the task waiting time.

[0017] According to a further technical solution, the state of the first agent is represented as:

[0018]

[0019] The action is represented as:

[0020]

[0021] in, Represents the task allocation scheduler The state space of a task, Indicates the number of virtual machine instances in the current resource pool. Indicates that the The waiting time for each virtual machine instance is Indicates the first The action of assigning a task to a virtual machine, Indicates the virtual machines.

[0022] In a further technical solution, the reward function of the first agent is expressed as:

[0023]

[0024] in, Represents the task allocation scheduler The reward function value of each task, Indicates the total time the task stays on the virtual machine instance. Indicates the actual execution time of the task.

[0025] In a further technical solution, the state of the second agent is represented as:

[0026]

[0027] The action is represented as:

[0028]

[0029] in, Indicates that the virtual machine auto-scaling scheduler executes The status of the secondary virtual machine lease adjustment plan, Indicates the number of tasks submitted by users in the recent period. Indicates the number of virtual machine instances rented in the recent period. is a specific time within the workload cycle, and Represent the average response time and success rate of all tasks in the time period, for The resource utilization of all virtual machine instances in the resource pool during the period, Indicates the maximum range within which virtual machine instances can be increased or decreased.

[0030] In a further technical solution, the reward function of the second agent is defined as:

[0031]

[0032] in, Represents a period Resource utilization during the period, Indicates the penalty that needs to be given when the success rate is below a certain threshold. Represents the weight of punishment on reward.

[0033] According to a further technical solution, the two-layer scheduling algorithm is composed of two layers of DQN algorithms, wherein the first layer of DQN algorithm is responsible for task allocation, and the second layer of DQN algorithm is responsible for virtual machine adjustment.

[0034] In a second aspect, the present invention provides a cloud computing resource scheduling system based on an elastically scalable two-tier scheduling framework, comprising:

[0035] A framework building module is configured to: establish a two-tier scheduling framework based on a three-tier cloud computing market; the three-tier cloud computing market includes infrastructure service providers, software operation service providers, and end users; the two-tier scheduling framework includes a task allocation scheduler and a virtual machine automatic scaling scheduler;

[0036] The first scheduling module is configured to: define a first agent by a task allocation scheduler, and determine the state, action, and reward function of the first agent by collecting task characteristics and virtual machine usage data of the current cloud computing platform;

[0037] The second scheduling module is configured to: define a second agent by the virtual machine automatic scaling scheduler, and determine the state, action and reward function of the second agent by collecting workload and resource utilization data of the current cloud computing platform;

[0038] The scheduling decision module is configured to: use a two-layer scheduling algorithm to solve the task allocation decision of the first intelligent agent, and use a two-layer scheduling algorithm to solve the virtual machine adjustment decision of the second intelligent agent.

[0039] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the cloud computing resource scheduling method based on an elastically scalable two-layer scheduling framework as described in the first aspect.

[0040] In a fourth aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the steps in the cloud computing resource scheduling method based on the elastically scalable two-layer scheduling framework as described in the first aspect are implemented.

[0041] One or more of the above technical solutions have the following beneficial effects:

[0042] To reduce the resource waste and additional costs associated with renting virtual machines while ensuring user satisfaction and QoS requirements, this paper proposes a two-tiered elastic scaling scheduling framework based on deep reinforcement learning. This framework addresses the issues of reasonable task scheduling and resource elastic scaling, using deep reinforcement learning as a foundation. On the one hand, it selects the most appropriate virtual machine in the current environment, aiming to reduce task completion time and improve user experience while meeting user needs. On the other hand, it elastically adjusts resources based on the environment's task load, using deep reinforcement learning to dynamically increase or decrease a certain number of virtual machine rentals. This reduces the cost of renting virtual machines for cloud service providers while ensuring task QoS requirements.

[0043] The present invention proposes an elastically scalable two-layer scheduling framework based on deep reinforcement learning, and provides an intelligent two-layer scheduling algorithm for SaaS providers. Through the collaborative work of QoS-aware task scheduling and workload-aware virtual machine automatic scaling, efficient task scheduling and elastic resource allocation are achieved. The first layer provides a QoS-aware task scheduling function, which uses the DQN algorithm to select the most suitable virtual machine instance for the current task from the resource pool, while ensuring the task QoS requirements and shortening the average task response time as much as possible; the second layer uses the DQN algorithm to provide a workload-aware virtual machine automatic scaling function, which is responsible for dynamically adjusting the number of rented virtual machine instances based on the load changes of the task and the properties of the currently rented virtual machine. When the virtual machine instances in the resource pool are saturated, the number of virtual machine instances can be appropriately reduced to avoid resource waste and reduce rental costs. Conversely, when the number of tasks in a certain time period is small, more virtual machine instances can be appropriately rented to ensure the success rate of the task and improve user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0045] Figure 1 This is a flowchart of the scheduling process in a three-tier cloud computing market based on a two-tier scheduling framework according to an embodiment of the present invention;

[0046] Figure 2 This is a flowchart of a two-tier scheduling algorithm according to an embodiment of the present invention;

[0047] Figure 3 This is a graph showing the changes in the number of tasks and virtual machines in a two-tier scheduling framework under a simulation scenario according to an embodiment of the present invention;

[0048] Figure 4 This is a graph showing changes in the number of tasks and virtual machines in a two-layer scheduling framework under a real scenario according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0050] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0051] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0052] Professional term explanation:

[0053] SaaS: Software as a Service; IaaS: Infrastructure as a Service; VM: Virtual Machine; QoS: Quality of Service; DQN: Deep Q-Network; DRL: Deep Reinforcement Learning; DNN: Deep Neural Network.

[0054] Example 1

[0055] like Figure 1 As shown, this embodiment discloses a cloud computing resource scheduling method based on an elastically scalable two-tier scheduling framework, the method comprising the following steps:

[0056] S1: Establish a two-tier scheduling framework based on a three-tier cloud computing market; the three-tier cloud computing market includes infrastructure service providers, software operation service providers, and end users; the two-tier scheduling framework includes a task allocation scheduler and a virtual machine automatic scaling scheduler;

[0057] In this embodiment, a typical three-tier cloud computing market is considered, involving three roles: IaaS providers, SaaS providers, and end users. IaaS providers use virtualization technology to abstract underlying physical resources and provide users with on-demand rental services of virtual machine instances, using the on-demand rental service of virtual machine instances as the basic unit. SaaS providers lease the appropriate type and number of virtual machine instances from IaaS providers based on actual demand, build resource pools, and deploy applications, providing convenient services to end users. End users can submit task requests with QoS requirements anytime and anywhere. From the perspective of SaaS providers, effectively managing leased resources and allocating tasks to appropriate virtual machine instances has become their primary concern.

[0058] like Figure 1 As shown, the present invention designs a two-tier scheduling framework for SaaS applications, including a virtual machine auto-scaling scheduler and a task allocation scheduler, which rely on and collaborate with each other to achieve their respective goals. Specifically, the auto-scaling scheduler is workload-aware and is responsible for adjusting virtual machine leasing plans based on resource supply and demand. The task allocation scheduler is QoS-aware and can allocate tasks submitted by end users to appropriate virtual machine instances in the resource pool. By adopting appropriate scheduling strategies and algorithms, as well as real-time monitoring and dynamic adjustment, efficient virtual machine auto-scaling and task scheduling can be achieved, thereby improving application performance and resource utilization.

[0059] The workload of a SaaS application consists of independent tasks submitted by many users, which arrive at the application in real time. For most transactional SaaS applications, the task workload varies at different times, but usually has a certain regularity over a long period of time (such as a day or a week). Therefore, assuming that in a given short time period The total number of tasks submitted by users is , a long period with a certain regularity is called , the changes in task load and the attributes of each task cannot be known in advance. The types of tasks submitted by end users are divided into two categories: memory-optimized and compute-optimized. Based on the above, each user task can be expressed as:

[0060]

[0061] in, Indicates the User tasks, Indicates the The sequence number of the task, Indicates the The arrival time of a task, Indicates the The type of task, Indicates the The length of the task, Indicates the QoS requirements of each task.

[0062] In the cloud computing market, IaaS providers usually offer various types of virtual machine instances and classify them according to their configuration and computing power. Common types include general-purpose, graphics processing, compute-optimized, and so on. Different types of virtual machine instances are selected for optimization for different types of tasks to achieve optimal performance. Different types of virtual machines may have very different speeds in performing the same task, so a suitable match can reduce the execution time of the task. In the study, it was simplified to only two types of virtual machine instances: memory-optimized and compute-optimized. Assuming that the resource pool contains a limited number of virtual machine instances rented from the IaaS provider, with the help of an auto-scaling scheduler, the SaaS provider can flexibly adjust the virtual machine leasing plan according to changes in task workloads. The number of virtual machine instances rented during the time period is , each virtual machine instance is defined as follows:

[0063]

[0064] in, Indicates the virtual machines, Indicates the The serial number of the virtual machine, Indicates the The type of virtual machine, Indicates the Virtual machines in The rental price, Indicates the The processing speed of a virtual machine for memory-optimized tasks, Indicates the The processing speed of a virtual machine for compute-optimized tasks.

[0065] For the cycle The rental cost of a virtual machine for a certain period of time can be calculated as:

[0066]

[0067] in, Indicates the rental cost of the virtual machine.

[0068] The task dispatch scheduler assigns tasks to designated VM instances based on a scheduling algorithm. When a new task arrives, it is quickly processed by the task dispatcher and assigned to a specific VM instance. If the assigned VM instance is idle, the task is executed immediately. Otherwise, if the assigned VM instance is occupied, the task enters the waiting queue for that VM instance. Tasks in the queue are executed on a first-come, first-served basis, ensuring that VM instances execute tasks in the order they arrive. This means that a task remains in a waiting state until all earlier tasks in the queue have completed. Furthermore, a VM instance can only execute a single task at a time, and tasks are not interrupted during processing.

[0069] Task response time It is defined as the total time a task stays in the system, which can be expressed as:

[0070]

[0071] in, Indicates the task execution time, Indicates the task waiting time.

[0072] Hypothetical Task Assign to virtual machine Instance, the execution time is calculated based on the task attributes and the assigned virtual machine instance , expressed as:

[0073]

[0074] virtual machine instances The waiting time of a task is determined by the number of tasks in the waiting queue, so we can define for:

[0075]

[0076] in, Represents a virtual machine instance The number of tasks in the waiting queue, Represents a virtual machine instance Execute the task free time, Indicates a task Arrival time.

[0077]

[0078] in, Represents a virtual machine instance Execute the task free time, Indicates a task On a virtual machine instance The execution time on Indicates a task Arrival time.

[0079] In the field of cloud computing task scheduling, task response time is a key QoS metric for transactional SaaS applications. For a specific task, if its total time in the system is within the user's acceptable response time, the task is considered successful; otherwise, the task is considered failed. The formula for successful execution is as follows:

[0080]

[0081] in, Indicates a task QoS requirements.

[0082] To maximize the advantages of SaaS providers, a two-tier scheduling framework needs to improve task success rates while reducing the total cost of renting virtual machine instances. To achieve this goal, a workload-aware virtual machine auto-scaling scheduler and a QoS-aware task allocation scheduler work together to flexibly adjust the number of rented virtual machine instances and optimize the allocation of tasks to virtual machines.

[0083] S2: The task allocation scheduler defines the first agent and determines the state, action, and reward function of the first agent by collecting task characteristics and virtual machine usage data of the current cloud computing platform;

[0084] In this embodiment, the QoS-aware task allocation scheduler in the two-tier scheduling framework focuses on task allocation, that is, how to select the most appropriate virtual machine instance from the resource pool for the current task. The following is a reinforcement learning model (RL model) for solving the online task scheduling problem.

[0085] State and Action: The task allocation scheduler records the characteristics of the task and the usage of virtual machine instances in the resource pool, which can be expressed as:

[0086]

[0087] in, Represents the task allocation scheduler The state space of a task, Indicates the number of virtual machine instances in the current resource pool. Indicates that the The waiting time for each virtual machine instance.

[0088] Task Allocation Scheduler The action of assigning a task to a virtual machine It can be recorded as a collection of all virtual machine instances in the resource pool, that is:

[0089]

[0090] in, Indicates the virtual machines.

[0091] Reward function: For transactional SaaS applications, task response time is the main indicator that determines application performance and user satisfaction. Therefore, we set The reward function is as follows:

[0092]

[0093] in, Represents the task allocation scheduler The reward function value of each task, Indicates the total time the task stays on the virtual machine instance. Indicates the actual execution time of the task. Therefore, the smaller the percentage of waiting time to the total time, the better the allocation result.

[0094] S3: The virtual machine auto-scaling scheduler defines a second agent and determines the state, action, and reward function of the second agent by collecting workload and resource utilization data of the current cloud computing platform;

[0095] In this embodiment, within a two-tier scheduling framework, a workload-aware VM auto-scaling scheduler continuously adjusts the number of leased VM instances based on changes in task workload and the current utilization of leased resources, as well as periodically adjusts VM leasing plans to achieve flexible resource allocation. When the number of tasks is low over a period of time, the number of VM instances can be appropriately reduced to avoid resource waste and reduce leasing costs. Conversely, when the resource pool is saturated with VM instances, leasing more VM instances can ensure the success rate of user tasks and improve user satisfaction. The reinforcement learning model (RL model) used to solve the VM auto-scaling scheduler is described below.

[0096] Status and Action: The VM auto-scaling scheduler executes Status of the secondary VM lease adjustment plan It can be expressed as:

[0097]

[0098] in, Indicates the number of tasks submitted by users in the recent period. Indicates the number of virtual machine instances rented in the recent period. is a specific time within the workload cycle, and Represent the average response time and success rate of all tasks in the time period, for Resource utilization of all virtual machine instances in the resource pool during the period.

[0099] In the action selection phase, the agent chooses to increase or decrease the number of rented virtual machine instances based on the current state. The action set is the range of changes in the number of virtual machine instances, which can be expressed as:

[0100]

[0101] in, Indicates the maximum range of virtual machine instances that can be increased or decreased. For example, the action Indicates increase virtual machine instance, action Indicates deletion virtual machine instances.

[0102] Reward Function: Adjusting the number of leased VM instances through the VM autoscaling scheduler affects not only resource utilization but also task response time. The optimization goal of the two-tier scheduling framework is to reduce the cost of leasing VM instances while ensuring service quality. Therefore, the design of the reward function needs to consider both the task success rate and VM instance utilization. The reward function is defined as follows:

[0103]

[0104] in, Represents a period Resource utilization during the period, Indicates the penalty that needs to be given when the success rate is below a certain threshold. Indicates the weight of punishment on reward, punishment Defined as:

[0105]

[0106] in, Represents a period The success rate of tasks during the period.

[0107] S4: A two-layer scheduling algorithm is used to solve the task allocation decision of the first intelligent agent, and a two-layer scheduling algorithm is used to solve the virtual machine adjustment decision of the second intelligent agent.

[0108] In the three-tier cloud market, the task workloads of SaaS applications are unpredictable and dynamically changing. Allocating continuously arriving tasks submitted by users to a limited number of virtual machine instances and adjusting the number of leased virtual machine instances based on resource usage and workload changes are two major challenges facing SaaS providers. SaaS providers can only monitor the status of their leased virtual machine instances in the resource pool and the current task workload, but cannot predict the future arrival times and resource requirements of tasks. Therefore, intelligent online scheduling algorithms that can adaptively solve such complex problems are needed. Compared to traditional task scheduling that relies on predefined rules and static models, DRL algorithms can more effectively adapt to changes in dynamic environments without requiring any prior knowledge of the environment.

[0109] Aiming to design a QoS-aware task allocation scheduler and a workload-aware VM auto-scaling scheduler, we propose a DRL-based two-tier scheduling algorithm for SaaS providers. Through continuous learning and improvement, this algorithm significantly improves cloud resource utilization and system performance. The DRL-based two-tier scheduling algorithm for SaaS providers consists of two layers of DQN algorithms: the first layer is responsible for task allocation, and the second layer is responsible for dynamically adjusting the number of leased VM instances. These two DQN layers interact with each other, and each DQN layer consists of two collaborative phases: an online decision-making phase and an offline training phase.

[0110] Furthermore, in this two-tier scheduling framework, the task allocation scheduler and the VM auto-scaling scheduler jointly optimize system resource utilization and task execution efficiency through real-time collaboration and feedback. The task scheduler drives VM demand and dynamically allocates tasks to available VMs based on the current state of the task queue. When the VM scheduler detects resource shortages, it triggers a request to expand the capacity of new VMs to alleviate resource bottlenecks. The VM scheduler provides feedback on resource supply, monitoring the real-time status of the resource pool and dynamically scaling the number of VMs. During expansion, the newly leased VM information is synchronized with the task scheduler to update its available resource list. During scaling, the VM scheduler ensures that the executing tasks on the VMs are released in advance to ensure seamless handover. Through closed-loop feedback optimization, the task scheduler's allocation strategy directly affects VM utilization, triggering the VM scheduler's response. If the task scheduler evenly distributes tasks to all nodes, the VM scheduler can maintain a stable scale. If the task scheduler causes some nodes to be overloaded, the VM scheduler quickly replenishes resources and guides the task scheduler to reassign tasks. The two synchronize periodic indicators to achieve a dynamic balance, avoiding resource waste or performance degradation. Through this layered collaboration mechanism, the system can adapt to dynamic loads while meeting the core requirements of high performance, high availability and low cost.

[0111] like Figure 2As shown in Figure 1, the online decision-making phase is responsible for making task-to-VM allocation decisions and VM adjustment decisions for the task allocation scheduler and VM autoscaling scheduler, respectively. Specifically, for the task allocation scheduler, when a new user-submitted task arrives in the system, the DRL agent observes the current state of the environment and uses a deep neural network (DNN) to calculate the Q-values ​​of all leased VM instances in the current state. A VM instance is allocated from the resource pool according to the "ϵ-greedy" policy to perform the task. In addition, the DRL agent receives an immediate reward for the task scheduling decision. For the VM autoscaling scheduler, the attributes of tasks and VM instances over time are recorded and used as part of the state. The DRL agent adjusts the number of VM instances based on the current state and the output of the DNN.

[0112] Offline training phase: Deep neural networks are the core components of DRL, responsible for approximating the value function, policy function, and model function. Deep neural networks utilize multi-layered neuron structures and nonlinear activation functions to process complex, high-dimensional state spaces. Due to their strong generalization and flexible adaptability, DNNs offer significant advantages in solving the proposed problem. This example constructs two different DNNs for the two-layer DQN algorithm, differing in the number of layers and neurons per layer. During the offline training phase, past choices and corresponding results are used to train the underlying DNN, continuously revising and updating the state-action value function. To improve the stability and performance of the DQN algorithm, experience replay and fixed target network techniques are employed. Experience replay stores past experiences in a replay buffer and randomly samples them to train the agent. This approach disrupts the connection between successive experiences, improving the efficiency and stability of the learning process. The fixed target network technique involves maintaining a separate target network, whose parameters are updated less frequently than the main network, and using the target network to generate target Q values. This technique can reduce oscillations and divergence during training.

[0113] Furthermore, due to the fully connected nature of deep neural networks (DNNs), the solution process may slightly overfit or become trapped in a local optimum. To prevent overfitting or underfitting, an L2 regularization term is added to the network's loss function. The loss function ensures that the model's output is consistent with the desired target. Regularization improves the model's generalization ability by adding a penalty to the loss function, significantly enhancing the robustness of the DQN algorithm.

[0114] like Figure 1As shown in the figure, the two-layer scheduling framework is divided into three parts, namely, end users, IaaS providers, and SaaS providers. Among them, end users are responsible for submitting tasks, IaaS providers are responsible for providing physical equipment, and SaaS providers divide the physical equipment provided by IaaS providers into resource pools consisting of several virtual machine instances and allocate tasks submitted by users. The controller is responsible for managing all parts of SaaS, including the user portal, task allocation scheduler, virtual machine automatic scaling scheduler, and resource pool.

[0115] The specific steps are: the user submits the task, the task assignment scheduler generates status according to the task ( ), the first agent according to the state Get the action assigned to a virtual machine instance ( ), then according to the arrival and Get reward value ( ), the first agent according to the reward value Learning is carried out to continuously evolve and obtain the optimal task allocation strategy. During the task allocation process, the second-layer virtual machine automatic scaling scheduler is executed once every set time period (set to 5 minutes in this embodiment). The second agent collects changes in multiple aspects of the task within the set time period and then generates a state. ( ), the second agent obtains the state Select the number of virtual machines to increase or decrease within the next set time period ( ), and finally according to the obtained and Get reward value ( ), similarly, the second agent learns and evolves according to the reward value to obtain the optimal resource adjustment strategy.

[0116] It should be noted that 、 、 Refers to all states, actions, and reward values ​​during the first agent's reinforcement learning process, not to a specific task; similarly, 、 and Refers to all states, actions, and reward values ​​during the reinforcement learning process of the second agent, not a specific time period.

[0117] The test experiment is described below.

[0118] In testing experiments, this invention used both simulated and real workloads. At each auto-scaling decision moment, i.e., every five minutes, the second layer of the two-tier scheduling framework generates a virtual machine adjustment decision to change the number of virtual machine instances in the resource pool. The two-tier scheduling algorithm was first applied to the simulated workload.

[0119] Figure 3 The experimental results of the two-tier scheduling framework in a simulation environment for 8 days are presented. The axis represents days, The axis represents the change in the number of tasks and virtual machines every 5 minutes. The figure describes the change in the task workload and the number of rented virtual machine instances using the two-layer scheduling framework. Figure 3 As shown in the figure, the first three days of learning usually involve 8 to 10 virtual machine instances. On the 4th to 5th day of learning, the number of virtual machine instances begins to gradually adapt to the load trend, and the changes in virtual machine instances begin to adapt to the changes in tasks. In the last three days, the changes in the number of virtual machine instances have fully adapted to the changes in workload. Under low load, the number of virtual machine instances is generally 3 to 5; under high load, the number of virtual machine instances is generally 8 to 10.

[0120] Table 1 Comparison results of the two-tier scheduling framework in the simulation environment

[0121]

[0122] Table 1 presents the specific experimental results of each algorithm in a simulated environment. It can be seen that although the two-layer DQN algorithm has a slightly longer average response time, it achieves a higher success rate and lower rental costs compared to the single-layer DQN-10VM algorithm with 10 fixed virtual machine instances, reducing rental costs by approximately 43%. Compared to the single-layer DQN algorithm with 3 fixed virtual machine instances, the two-layer DQN algorithm significantly outperforms the DQN-3VM algorithm in terms of average response time and success rate, and also outperforms the DQN-3VM algorithm in terms of rental costs. In the comparison of the two-layer algorithms, using DQN in both the first and second layers, the two-layer DQN algorithm not only improves the average response time and success rate but also reduces rental costs by 24.8% and 21.1% compared to the FCFS-DQN and DQN-ARIMA algorithms, respectively.

[0123] To further validate the performance of the two-layer scheduling framework, we used a real-world NASA dataset for testing. The NASA workload trace used contains all HTTP requests issued by NASA servers at the Kennedy Space Center in Florida during August 1995. Table 2 reports the experimental results of the two-layer DQN algorithm and other comparison algorithms throughout the month.

[0124] Table 2 Comparison results of the two-tier scheduling framework in real environment

[0125]

[0126] Table 2 shows that both the single-layer DQN-10VM algorithm and the two-layer DQN algorithm, which use a fixed 10 virtual machine instances, significantly outperform the other compared algorithms in terms of average task response time and success rate. However, the single-layer DQN algorithm cannot dynamically adjust the number of leased virtual machine instances based on workload changes, resulting in wasted resources and increased leasing costs. The two-layer DQN algorithm balances QoS requirements and cost optimization. Although its average response time and success rate are slightly lower than those of the single-layer DQN algorithm, which uses a fixed 10 virtual machine instances, it achieves a 43.3% cost savings.

[0127] Since there are many tasks in a month, Figure 4 Only the changes in workload and number of rented virtual machine instances for the first 8 days using the two-layer DQN algorithm in the actual scenario are shown. Figure 3 The learning process shown is similar, and the change in the number of rented virtual machine instances also gradually adapts to the workload trend.

[0128] Efficient task scheduling and flexible resource provisioning are two major challenges facing application providers. This paper addresses this issue by proposing an intelligent two-tier scheduling framework. This algorithm utilizes DRL-based scheduling. The first tier uses a DQN algorithm to assign tasks to appropriate virtual machine instances, while the second tier dynamically adjusts the number of virtual machine instances to reduce the cost of renting them. Both the single-tier and two-tier DQN algorithms were compared with other algorithms. Experimental results demonstrate the effectiveness of this approach in ensuring high success rates and reducing rental costs.

[0129] In other implementations, for the second layer of elastic resource expansion, vertical resource expansion rather than horizontal resource expansion is used to dynamically adjust the resource configuration of a single instance, or predictive scaling technology is used to predict future load based on historical data, adjust resources in advance, reduce response latency, and avoid over-configuration. Therefore, the elastic scaling two-layer scheduling framework based on deep reinforcement learning proposed in this invention may be modified into an elastic resource scheduler based on vertical resource expansion or predictive resource scaling, thereby realizing the scheduling framework based on deep reinforcement learning proposed in this invention from another perspective.

[0130] Example 2

[0131] This embodiment discloses a cloud computing resource scheduling system based on an elastically scalable two-tier scheduling framework, including:

[0132] A framework building module is configured to: establish a two-tier scheduling framework based on a three-tier cloud computing market; the three-tier cloud computing market includes infrastructure service providers, software operation service providers, and end users; the two-tier scheduling framework includes a task allocation scheduler and a virtual machine automatic scaling scheduler;

[0133] The first scheduling module is configured to: define a first agent by a task allocation scheduler, and determine the state, action, and reward function of the first agent by collecting task characteristics and virtual machine usage data of the current cloud computing platform;

[0134] The second scheduling module is configured to: define a second agent by the virtual machine automatic scaling scheduler, and determine the state, action and reward function of the second agent by collecting workload and resource utilization data of the current cloud computing platform;

[0135] The scheduling decision module is configured to: use a two-layer scheduling algorithm to solve the task allocation decision of the first intelligent agent, and use a two-layer scheduling algorithm to solve the virtual machine adjustment decision of the second intelligent agent.

[0136] Example 3

[0137] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of embodiment 1 when executing the program.

[0138] Example 4

[0139] The purpose of this embodiment is to provide a computer-readable storage medium, a computer-readable storage medium having a computer program stored thereon, which performs the steps of the method of embodiment 1 when executed by a processor.

[0140] The steps involved in the apparatuses of Examples 3 and 4 above correspond to those of Method Example 1. For detailed implementation, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any of the methods of the present invention.

[0141] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0142] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

[0143] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A cloud computing resource scheduling method based on an elastically scalable two-tier scheduling framework, characterized in that: include: Based on the three-tier cloud computing market, a two-tier scheduling framework is established; The three-tier cloud computing market includes infrastructure service providers, software operation service providers, and end users; the two-tier scheduling framework includes a task allocation scheduler and a virtual machine automatic scaling scheduler; The task allocation scheduler defines a first agent and determines the state, action, and reward function of the first agent by collecting task characteristics and virtual machine usage data of the current cloud computing platform; The virtual machine auto-scaling scheduler defines a second agent. By collecting the workload and resource utilization data of the current cloud computing platform, the state, action, and reward function of the second agent are determined. The reward function of the second agent is defined as: in, Represents a period Resource utilization during the period, Indicates the penalty that needs to be given when the success rate is below a certain threshold. Indicates the weight of punishment on reward; A two-layer scheduling algorithm is used to solve the task allocation decision of the first intelligent agent, and a two-layer scheduling algorithm is used to solve the virtual machine adjustment decision of the second intelligent agent.

2. The cloud computing resource scheduling method based on the elastic scaling two-tier scheduling framework according to claim 1, characterized in that: The task response time in the two-layer scheduling framework is expressed as: in, represents the task response time, Indicates the task execution time, Indicates the task waiting time.

3. The cloud computing resource scheduling method based on the elastic scaling two-tier scheduling framework according to claim 1, characterized in that: The state of the first agent is expressed as: The action is represented as: in, Represents the task allocation scheduler The state space of a task, Indicates the number of virtual machine instances in the current resource pool. Indicates that the The waiting time for each virtual machine instance is Indicates the first The action of assigning a task to a virtual machine, Indicates the virtual machines.

4. The cloud computing resource scheduling method based on the elastic scaling two-tier scheduling framework according to claim 3 is characterized in that: The reward function of the first agent is expressed as: in, Represents the task allocation scheduler The reward function value of each task, Indicates the total time the task stays on the virtual machine instance. Indicates the actual execution time of the task.

5. The cloud computing resource scheduling method based on the elastic scaling two-tier scheduling framework according to claim 1, characterized in that: The state of the second agent is expressed as: The action is represented as: in, Indicates that the virtual machine auto-scaling scheduler executes The status of the secondary virtual machine lease adjustment plan, Indicates the number of tasks submitted by users in the recent period. Indicates the number of virtual machine instances rented in the recent period. is a specific time within the workload cycle, and Respectively represent the average response time and success rate of all tasks in the recent period, for The resource utilization of all virtual machine instances in the resource pool during the period, Indicates the maximum range within which virtual machine instances can be increased or decreased.

6. The cloud computing resource scheduling method based on the elastic scaling two-tier scheduling framework according to claim 1, characterized in that: The two-layer scheduling algorithm consists of two layers of DQN algorithms. The first layer of DQN algorithm is responsible for task allocation, and the second layer of DQN algorithm is responsible for virtual machine adjustment.

7. A cloud computing resource scheduling system based on an elastically scalable two-tier scheduling framework, characterized in that: include: A framework building module is configured to: establish a two-tier scheduling framework based on a three-tier cloud computing market; The three-tier cloud computing market includes infrastructure service providers, software operation service providers, and end users; the two-tier scheduling framework includes a task allocation scheduler and a virtual machine automatic scaling scheduler; The first scheduling module is configured to: define a first agent by a task allocation scheduler, and determine the state, action, and reward function of the first agent by collecting task characteristics and virtual machine usage data of the current cloud computing platform; The second scheduling module is configured as follows: the virtual machine automatic scaling scheduler defines a second agent, and determines the state, action, and reward function of the second agent by collecting workload and resource utilization data of the current cloud computing platform; the reward function of the second agent is defined as: in, Represents a period Resource utilization during the period, Indicates the penalty that needs to be given when the success rate is below a certain threshold. Indicates the weight of punishment on reward; The scheduling decision module is configured to: use a two-layer scheduling algorithm to solve the task allocation decision of the first intelligent agent, and use a two-layer scheduling algorithm to solve the virtual machine adjustment decision of the second intelligent agent.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the cloud computing resource scheduling method based on an elastically scalable two-layer scheduling framework as described in any one of claims 1 to 6 are implemented.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the cloud computing resource scheduling method based on an elastic scaling two-layer scheduling framework as described in any one of claims 1 to 6 are implemented.

Images