Dynamic scheduling method for cloud computing resource pool

Abstract

The invention discloses a dynamic scheduling method for a cloud computing resource pool in the technical field of cloud computing. Comprising the following steps: S1, analyzing resource demands; s2, performing performance prediction on the resource pool through load prediction; s3, predicting task demand resources; s4, resource allocation decision making; s5, task scheduling; and S6, resource monitoring and feedback. According to the invention, load prediction is carried out through the long-short-term memory network, an adaptive mechanism is introduced, and a load prediction model is updated along with continuous change of data, so that the accuracy is improved; the resource demand of the task is predicted through a machine learning algorithm, a reasonable resource allocation decision is made according to the state of the current resource pool and the resource demand predicted by the task, the utilization rate of the resource is improved, stable operation calculation of the task is guaranteed, efficient task allocation and scheduling are achieved, resource utilization maximization is guaranteed, and the resource utilization rate is increased. And the change of the system load is effectively responded.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing, and in particular to a method for dynamic scheduling of a cloud computing resource pool. Background Art

[0002] Cloud computing resource scheduling refers to the process of optimizing, coordinating and managing computing, storage, network and other resources in a cloud computing environment to achieve efficient resource utilization and balanced allocation. Dynamic resource allocation is an important part of cloud computing resource scheduling, which can help improve system resource utilization, reduce costs, and adapt to different business needs and changes.

[0003] Resource scheduling algorithms are the key to dynamic resource allocation. In a cloud computing environment, resource scheduling algorithms need to consider multiple factors, such as resource demand, resource type, and resource availability. By comprehensively evaluating these factors, the optimal resource scheduling solution can be selected. Currently, commonly used resource scheduling algorithms include load balancing, static resource allocation, and dynamic resource allocation.

[0004] Resource monitoring is the premise for dynamic resource allocation. In a cloud computing environment, different types of resources need to be monitored in real time to detect resource changes and anomalies in a timely manner. Through resource monitoring, we can understand resource usage and predict future resource requirements, thereby providing data support for resource scheduling.

[0005] Resource prediction is the basis for dynamic resource allocation. In a cloud computing environment, it is necessary to analyze and predict historical data to predict future resource demand and usage trends. Through resource prediction, resource scheduling can be planned in advance to avoid waste and shortage of resources.

[0006] However, there are some problems with the existing technology: the existing dynamic scheduling of resource pools has the problem that load balancing cannot be adjusted dynamically, resulting in elastic expansion of the resource pool, and it cannot automatically increase or decrease resources according to load changes, resulting in the resource pool being unable to improve overall performance and energy efficiency optimization. Therefore, we propose a dynamic scheduling method for cloud computing resource pools. Summary of the invention

[0007] In view of the deficiencies in the prior art, the present invention provides a method for dynamic scheduling of cloud computing resource pools, which allocates tasks through task requirements and load prediction, thereby improving the elastic expansion of resources and improving the overall performance and energy efficiency optimization of the resource pool.

[0008] The object of the present invention is achieved by: a cloud computing resource pool dynamic scheduling method, comprising the following steps: S1. Analyze resource requirements: perform load monitoring and analysis on resource pools of multiple servers, analyze task requirements, and obtain operation history data analysis of server resource pools; S2. Predict the performance of the resource pool through load prediction: Based on the collected historical data and real-time monitoring information, load prediction is performed through the long short-term memory network to foresee the resource demand in the future. An adaptive mechanism is introduced to update the load prediction model as the data changes to improve accuracy. S3. Predicting the required resources for tasks: Predicting the resource requirements of tasks through machine learning algorithms, and predicting the computing, storage, and network resources required for different tasks in the future; S4, Resource allocation decision: After obtaining the load forecast, the scheduling system needs to make reasonable resource allocation decisions based on the current resource pool status and the predicted resource requirements of the task, and allocate resources through resource constraint judgment, priority management and resource optimization; S5, Task Scheduling: After the resource pool allocation decision is made, the task is scheduled to the resource pool of a specific server; S6. Resource monitoring and feedback: After task scheduling and resource allocation, the system operation is continuously monitored and dynamic adjustments are made based on the monitoring data, mainly including performance monitoring, dynamic adjustment and feedback mechanisms.

[0009] Specifically, the load monitoring in S1 is used to monitor the resource pool of the server and collect the usage of the current resource pool, including real-time monitoring data of CPU, memory, storage, and network bandwidth; The task demand is to analyze the resource demand of each task or application, including computing-intensive, storage-intensive or network-intensive workloads, and obtain the resource consumption of each task; The historical data analysis is to identify load fluctuation patterns through historical performance data.

[0010] Specifically, the calculation formula of the long short-term memory network in S2 is as follows: The calculation formula is: Forget Gate: , Input Gate: , Candidate memory cells: , Memory unit update: , Output Gate: , Hide status updates: , in: , , , is the weight matrix of the input gate, forget gate, candidate memory unit and output gate; , , , is the weight matrix from the hidden layer to each gate; , , , is the bias term of each gate; is the state of the memory unit; The LSTM network can effectively capture long-term dependencies in a sequence through its gating mechanism and predict resource requirements within a certain period of time in the future.

[0011] Specifically, the adaptive mechanism in S2 combines the control of the forget gate and the input gate, so that the state of the memory unit is adaptively updated with the input data and task requirements, and can automatically learn and memorize sequence information at different time steps, and adaptively adjust the state and information flow of its memory unit according to different input and context information; And the calculation formula of the adaptive mechanism is as follows: Memory unit update: .

[0012] Specifically, the calculation formula of the machine learning algorithm in S3 is as follows: Assume that the input feature vector is and an output , the expression of the linear regression model is: , in, is the target variable, i.e. the predicted output; is the intercept term; It is a feature The regression coefficient of is the error term, which represents the difference between the model’s prediction and the true value; Use the least squares method to find the regression coefficients, that is, minimize the sum of squared errors: , in, is the number of samples, It is The true value of the sample, is the value predicted by the model, which is The target variable is obtained by minimizing the loss function to obtain the optimal estimate of the regression coefficient.

[0013] Specifically, the resource constraint judgment in S4 is used to determine whether the current resource pool has sufficient resources to meet the task requirements. If the resource pool resources are insufficient, it is necessary to trigger an expansion operation to increase virtual machine instances or container instances; Priority management: When resources are insufficient, the scheduling system needs to allocate resources according to the priority of tasks; Resource optimization is based on load balancing algorithms to reasonably allocate resources to avoid overloading or idleness of certain resources.

[0014] Specifically, the calculation formula of the load balancing algorithm is as follows: Each resource pool The current load performance is , the weight is , and the required resources for the task are , then the assignment of tasks to resource pools is represented as: , And all resource pools are calculated through weighted polling. Each request is allocated in the polling order, and the weight of each resource pool is considered at the same time to allocate tasks to The resource pool whose calculated value is closest to the weight.

[0015] Specifically, the steps of triggering the extension operation are as follows: Create an Auto Scaling Group and define its minimum, maximum, and desired number of instances; Set expansion policies and set expansion thresholds based on CPU usage; Configure CloudWatch Alarm to trigger expansion when CPU usage exceeds 80%, and to trigger reduction when CPU usage falls below 30%; Configure an Auto Scaling Policy to expand the size of the resource pool when the load increases and remove the expanded resource pool size when the load decreases.

[0016] Specifically, the performance monitoring in S6 is used to continuously collect the operation data of the resource pool, including CPU usage, memory usage, network bandwidth, and I / O performance, to ensure efficient use of resources; Dynamic adjustment is used to adjust resource allocation and scheduling strategies based on real-time monitoring data and task execution feedback. When it is found that the load of a resource pool is too high, task scheduling is adjusted to avoid resource bottlenecks. The feedback mechanism is used to provide feedback based on the execution results of the task, including completion time, resource consumption and task failure, to optimize the scheduling decision algorithm and continuously improve the accuracy and reliability of the scheduling system through feedback data.

[0017] Specifically, the cloud computing resource pool has high availability and fault recovery capabilities, including fault detection, failover and redundant deployment; The fault detection is used to monitor each node in the resource pool in real time and detect node or resource faults; The failover is used to migrate the affected tasks to other normal resource nodes when a failure occurs; The redundant deployment is used to ensure the redundancy of the system by deploying tasks or applications to multiple resource pool nodes.

[0018] Compared with the prior art, the present invention has the following beneficial effects: When the present invention is used, it analyzes and processes the real-time monitoring data and operation history data of the server resource pool, mainly performs load prediction through long short-term memory network, and introduces adaptive intelligence to update the load prediction model as the data changes continuously to improve accuracy; And through machine learning algorithms, it can predict the resource requirements of tasks, and make reasonable resource allocation decisions based on the current state of the resource pool and the predicted resource requirements of the task. Through resource constraint judgment, priority management and resource optimization, resource allocation is carried out to make the predicted resource requirements of the task closest to the state of the resource pool, improve resource utilization, and ensure the stability of task operation and calculation, realize efficient task allocation and scheduling, so as to ensure maximum resource utilization and effectively respond to changes in system load; In addition, resource constraint judgment triggers expansion by determining CPU usage, which facilitates expansion or reduction of the resource pool, improves resource utilization, and monitors and provides feedback on resource pool resources to ensure stable operation or task processing of the resource pool. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0020] Figure 1 A schematic diagram of the steps of the present invention is provided. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] like Figure 1 A cloud computing resource pool dynamic scheduling method is shown, comprising the following steps: S1. Analyze resource requirements: perform load monitoring and analysis on resource pools of multiple servers, analyze task requirements, and obtain operation history data analysis of server resource pools; S2. Predict the performance of the resource pool through load prediction: Based on the collected historical data and real-time monitoring information, load prediction is performed through the long short-term memory network to foresee the resource demand in the future. An adaptive mechanism is introduced to update the load prediction model as the data changes to improve accuracy. S3. Predicting the required resources for tasks: Predicting the resource requirements of tasks through machine learning algorithms, and predicting the computing, storage, and network resources required for different tasks in the future; S4, Resource allocation decision: After obtaining the load forecast, the scheduling system needs to make reasonable resource allocation decisions based on the current resource pool status and the predicted resource requirements of the task, and allocate resources through resource constraint judgment, priority management and resource optimization; S5, Task Scheduling: After the resource pool allocation decision is made, the task is scheduled to the resource pool of a specific server; S6. Resource monitoring and feedback: After task scheduling and resource allocation, the system operation is continuously monitored and dynamic adjustments are made based on the monitoring data, mainly including performance monitoring, dynamic adjustment and feedback mechanisms.

[0023] In this embodiment, preferably, the load monitoring in S1 is used to monitor the resource pool of the server and collect the usage of the current resource pool, including real-time monitoring data of CPU, memory, storage, and network bandwidth; The task demand is to analyze the resource demand of each task or application, including computing-intensive, storage-intensive or network-intensive workloads, and obtain the resource consumption of each task; The historical data analysis is to identify load fluctuation patterns through historical performance data; It should be noted that monitoring the resource pool of the server is achieved through load monitoring, which facilitates the prediction of the operating capacity of the resource pool of the server based on real-time monitoring data and historical data analysis.

[0024] In this embodiment, preferably, the calculation formula of the long short-term memory network in S2 is as follows: The calculation formula is: Forget Gate: , Input Gate: , Candidate memory cells: , Memory unit update: , Output Gate: , Hide status updates: , in: , , , is the weight matrix of the input gate, forget gate, candidate memory unit and output gate; , , , is the weight matrix from the hidden layer to each gate; , , , is the bias term of each gate; is the state of the memory unit; The LSTM network can effectively capture the long-term dependencies in the sequence through its gating mechanism and predict the resource demand within a certain period of time in the future; It should be noted that the state of the server's resource pool can be predicted through the long short-term memory network, which makes it easier to obtain resource requirements and allocate tasks.

[0025] In this embodiment, preferably, the adaptive mechanism in S2 is combined with the control of the forget gate and the input gate, so that the state of the memory unit is adaptively updated with the input data and task requirements, and can automatically learn and memorize sequence information of different time steps, and adaptively adjust the state and information flow of its memory unit according to different input and context information; And the calculation formula of the adaptive mechanism is as follows: Memory unit update: ; It should be noted that the state of the memory unit is adaptively updated according to the input data and task requirements through the adaptive mechanism, so as to adaptively adjust the state and information flow of the memory unit.

[0026] In this embodiment, preferably, the calculation formula of the machine learning algorithm in S3 is as follows: Assume that the input feature vector is and an output , the expression of the linear regression model is: , in, is the target variable, i.e. the predicted output; is the intercept term; It is a feature The regression coefficient of is the error term, which represents the difference between the model’s prediction and the true value; Use the least squares method to find the regression coefficients, that is, minimize the sum of squared errors: , in, is the number of samples, It is The true value of the sample, is the value predicted by the model, which is Target variable; get the optimal estimate of the regression coefficient by minimizing the loss function; It should be noted that the linear regression model is used to predict the task requirements, and the regression coefficient is found through the least squares method. By minimizing the loss function, the optimal estimate of the regression coefficient is obtained, which is convenient for improving the calculation accuracy of the linear regression model.

[0027] In this embodiment, preferably, the resource constraint judgment in S4 is used to judge whether the current resource pool has sufficient resources to meet the task requirements. If the resource pool resources are insufficient, it is necessary to trigger an expansion operation to increase virtual machine instances or container instances; Priority management: When resources are insufficient, the scheduling system needs to allocate resources according to the priority of tasks; Resource optimization is based on load balancing algorithms to reasonably allocate resources to avoid overloading or idleness of some resources; It should be noted that the resource pool is judged through resource constraint judgment, and the expansion operation is triggered, and resources are allocated according to the priority of the task to avoid overloading or idleness of certain resources and improve the full utilization rate of the resource pool.

[0028] In this embodiment, preferably, the calculation formula of the load balancing algorithm is as follows: Each resource pool The current load performance is , the weight is , and the required resources for the task are , then the assignment of tasks to resource pools is represented as: , And all resource pools are calculated through weighted polling. Each request is allocated in the polling order, and the weight of each resource pool is considered at the same time to allocate tasks to The resource pool whose calculated value is closest to the weight; It should be noted that when calculating, if the task demand is less than the load performance of the resource pool, the task demand can be fully and stably operated or processed in the resource pool, even if The calculated value is less than 1 and as close to 1 as possible.

[0029] In this embodiment, preferably, the steps of triggering the extension operation are as follows: Create an Auto Scaling Group and define its minimum, maximum, and desired number of instances; Set expansion policies and set expansion thresholds based on CPU usage; Configure CloudWatch Alarm to trigger expansion when CPU usage exceeds 80%, and to trigger reduction when CPU usage falls below 30%; Configure an Auto Scaling Policy to expand the size of the resource pool when the load increases and remove the expanded resource pool size when the load decreases; It should be noted that by calculating the CPU usage, expansion is triggered when the CPU usage exceeds 80%, and reduction is triggered when the CPU usage is less than 30%, thereby achieving adaptive adjustment of the resource pool.

[0030] In this embodiment, preferably, the performance monitoring in S6 is used to continuously collect the operation data of the resource pool, including CPU usage, memory usage, network bandwidth, and I / O performance, to ensure efficient use of resources; Dynamic adjustment is used to adjust resource allocation and scheduling strategies based on real-time monitoring data and task execution feedback. When it is found that the load of a resource pool is too high, task scheduling is adjusted to avoid resource bottlenecks. The feedback mechanism is used to provide feedback based on the execution results of the task, including completion time, resource consumption, and task failure, to optimize the scheduling decision algorithm and continuously improve the accuracy and reliability of the scheduling system through feedback data; It should be noted that by continuously monitoring the performance of the resource pool, it is convenient to adaptively adjust the resource pool when the task is allocated, run or processed, so as to stably improve the stability of the resource pool.

[0031] In this embodiment, preferably, the cloud computing resource pool has high availability and fault recovery capabilities, including fault detection, failover and redundant deployment; The fault detection is used to monitor each node in the resource pool in real time and detect node or resource faults; The failover is used to migrate the affected tasks to other normal resource nodes when a failure occurs; The redundant deployment is used to ensure the redundancy of the system by deploying tasks or applications to multiple resource pool nodes; It should be noted that cloud computing resource pool scheduling needs to have high availability and fault recovery capabilities to ensure that the system can continue to operate stably under hardware failures or resource bottlenecks.

[0032] The specific operation process of the present invention is as follows: Step 1: Analyze resource requirements: Perform load monitoring and analysis on the resource pools of multiple servers, analyze task requirements, and obtain operation history data analysis of the server resource pools; Step 2: Predict the performance of the resource pool through load prediction: Based on the collected historical data and real-time monitoring information, load prediction is performed through the long short-term memory network to foresee the resource demand in the future. An adaptive mechanism is introduced to update the load prediction model as the data changes to improve accuracy. Step 3: Predict the required resources for the task: Use machine learning algorithms to predict the resource requirements of the task and predict the computing, storage, and network resources required for different tasks in the future. Step 4: Resource allocation decision: After obtaining the load forecast, the scheduling system needs to make reasonable resource allocation decisions based on the current resource pool status and the resource requirements predicted by the task, and allocate resources through resource constraint judgment, priority management and resource optimization; Step 5: Task Scheduling: After the resource pool allocation decision is made, the task is scheduled to the resource pool of a specific server; Step 6: Resource monitoring and feedback: After task scheduling and resource allocation, continuously monitor the system operation and make dynamic adjustments based on the monitoring data, mainly including performance monitoring, dynamic adjustment and feedback mechanism.

[0033] The above embodiments are only used to help understand the method and core idea of ​​the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A cloud computing resource pool dynamic scheduling method, characterized in that: The steps include: S1. Analyze resource requirements: perform load monitoring and analysis on resource pools of multiple servers, analyze task requirements, and obtain operation history data analysis of server resource pools; S2. Predict the performance of the resource pool through load prediction: Based on the collected historical data and real-time monitoring information, load prediction is performed through the long short-term memory network to foresee the resource demand in the future. An adaptive mechanism is introduced to update the load prediction model as the data changes to improve accuracy. S3. Predicting the required resources for tasks: Predicting the resource requirements of tasks through machine learning algorithms, and predicting the computing, storage, and network resources required for different tasks in the future; S4, Resource allocation decision: After obtaining the load forecast, the scheduling system needs to make reasonable resource allocation decisions based on the current resource pool status and the predicted resource requirements of the task, and allocate resources through resource constraint judgment, priority management and resource optimization; S5, Task Scheduling: After the resource pool allocation decision is made, the task is scheduled to the resource pool of a specific server; S6. Resource monitoring and feedback: After task scheduling and resource allocation, the system operation is continuously monitored and dynamic adjustments are made based on the monitoring data, mainly including performance monitoring, dynamic adjustment and feedback mechanisms.

2. A cloud computing resource pool dynamic scheduling method according to claim 1, characterized in that: The load monitoring in S1 is used to monitor the resource pool of the server and collect the usage of the current resource pool, including real-time monitoring data of CPU, memory, storage, and network bandwidth; The task demand is to analyze the resource demand of each task or application, including computing-intensive, storage-intensive or network-intensive workloads, and obtain the resource consumption of each task; The historical data analysis is to identify load fluctuation patterns through historical performance data.

3. A cloud computing resource pool dynamic scheduling method according to claim 1, characterized in that: The calculation formula of the long short-term memory network in S2 is as follows: The calculation formula is: Forget Gate: , Input Gate: , Candidate memory cells: , Memory unit update: , Output Gate: , Hide status updates: , in: , , , is the weight matrix of the input gate, forget gate, candidate memory unit and output gate; , , , is the weight matrix from the hidden layer to each gate; , , , is the bias term of each gate; is the state of the memory unit; The LSTM network can effectively capture long-term dependencies in a sequence through its gating mechanism and predict resource requirements within a certain period of time in the future.

4. A cloud computing resource pool dynamic scheduling method according to claim 3, characterized in that: The adaptive mechanism in S2 combines the control of the forget gate and the input gate, so that the state of the memory unit is adaptively updated with the input data and task requirements, and can automatically learn and memorize sequence information at different time steps, and adaptively adjust the state and information flow of its memory unit according to different input and context information; And the calculation formula of the adaptive mechanism is as follows: Memory unit update: .

5. The method for dynamic scheduling of a cloud computing resource pool according to claim 1, characterized in that: The calculation formula of the machine learning algorithm in S3 is as follows: Assume that the input feature vector is and an output , the expression of the linear regression model is: , in, is the target variable, i.e. the predicted output; is the intercept term; It is a feature The regression coefficient of is the error term, which represents the difference between the model’s prediction and the true value; Use the least squares method to find the regression coefficients, that is, minimize the sum of squared errors: ， in, is the number of samples, It is The true value of the sample, is the value predicted by the model, which is The target variable is obtained by minimizing the loss function to obtain the optimal estimate of the regression coefficient.

6. A cloud computing resource pool dynamic scheduling method according to claim 1, characterized in that: The resource constraint judgment in S4 is used to determine whether the current resource pool has sufficient resources to meet the task requirements. If the resource pool resources are insufficient, it is necessary to trigger an expansion operation to increase virtual machine instances or container instances; Priority management: When resources are insufficient, the scheduling system needs to allocate resources according to the priority of tasks; Resource optimization is based on load balancing algorithms to reasonably allocate resources to avoid overloading or idleness of certain resources.

7. A cloud computing resource pool dynamic scheduling method according to claim 6, characterized in that: The calculation formula of the load balancing algorithm is as follows: Each resource pool The current load performance is , the weight is , and the required resources for the task are , then the assignment of tasks to resource pools is represented as: ， And all resource pools are calculated through weighted polling. Each request is allocated in the polling order, and the weight of each resource pool is considered at the same time to allocate tasks to The resource pool whose calculated value is closest to the weight.

8. A method for dynamic scheduling of a cloud computing resource pool according to claim 6, characterized in that: The steps for triggering the extension operation are as follows: Create an Auto Scaling Group and define its minimum, maximum, and desired number of instances; Set expansion policies and set expansion thresholds based on CPU usage; Configure CloudWatch Alarm to trigger expansion when CPU usage exceeds 80% and to trigger reduction when CPU usage falls below 30%; Configure an Auto Scaling Policy to expand the size of the resource pool when the load increases and remove the expanded resource pool size when the load decreases.

9. A cloud computing resource pool dynamic scheduling method according to claim 1, characterized in that: The performance monitoring in S6 is used to continuously collect the operation data of the resource pool, including CPU usage, memory usage, network bandwidth, and I / O performance, to ensure efficient use of resources; Dynamic adjustment is used to adjust resource allocation and scheduling strategies based on real-time monitoring data and task execution feedback. When it is found that the load of a resource pool is too high, task scheduling is adjusted to avoid resource bottlenecks. The feedback mechanism is used to provide feedback based on the execution results of the task, including completion time, resource consumption and task failure, to optimize the scheduling decision algorithm and continuously improve the accuracy and reliability of the scheduling system through feedback data.

10. A cloud computing resource pool dynamic scheduling method according to claim 1, characterized in that: The cloud computing resource pool has high availability and fault recovery capabilities, including fault detection, failover and redundant deployment; The fault detection is used to monitor each node in the resource pool in real time and detect node or resource faults; The failover is used to migrate the affected tasks to other normal resource nodes when a failure occurs; The redundant deployment is used to ensure the redundancy of the system by deploying tasks or applications to multiple resource pool nodes.

Images