Cloud platform computing power scheduling method based on distributed computing

Abstract

The invention provides a cloud platform computing power scheduling method based on distributed computing. The cloud platform computing power scheduling method comprises the following steps of S1, performing short-term and ultra-short-term prediction on loads of areas administered by virtual resource pools through a decomposition and integration prediction analysis method; s2, establishing a short-term load computing power demand prediction model based on LSTM; s3, establishing an ultra-short-term load computing power demand prediction model based on XGBoost; and S4, establishing a distributed computing cloud platform and each virtual resource pool, and performing management and scheduling optimization on computing power. According to the method, the computing power required by each virtual resource pool load task is subjected to short-term and ultra-short-term prediction, and on the basis of a decomposition and integration prediction technology, an LSTM (long short-term memory) model and an XGBoost model are respectively used for prediction. And the cloud platform central server groups the virtual resource pools according to the prediction curve, and only the virtual resource pools in the same group can perform resource scheduling and information transmission.

Description

Technical Field

[0001] This invention relates to the field of cloud computing technology, and in particular to a cloud platform computing power scheduling method based on distributed computing. Background Technology

[0002] In recent years, cloud computing technology has been widely adopted across various industries. Cloud computing differs from traditional IT in its technical architecture, and its resource pooling and elastic provisioning features have dramatically changed IT usage patterns. Furthermore, cloud computing can innovatively optimize how modern enterprises schedule and deploy IT resources, thereby generating new and powerful momentum for enterprises' IT capabilities.

[0003] In traditional architectures, the call relationships between business applications and services are relatively fixed, typically implemented in layers such as presentation layer, application layer, data layer, and database. Application services, log files, and data architecture are centrally stored and managed. Experienced operations personnel can quickly calculate computing power requirements by obtaining business operation indicator data from the resource and application sides. However, in a cloud environment, business applications are implemented through a microservice architecture. Large businesses are broken down into independent service modules according to their functional responsibilities. Businesses possess better scalability, independent upgradeability, maintainability, and robustness. As the scale of business grows rapidly, the number of business services also increases, and the call relationships between services become increasingly complex. Computing power strategies need to coordinate basic service modules with computing power needs at different times and locations. Simply relying on manual investigation is no longer sufficient to cope with the computing power scheduling needs of complex business services.

[0004] Industry experts and scholars have conducted relatively mature and comprehensive research on resource scheduling, primarily focusing on scheduling optimization objectives and resource scheduling algorithms, and have achieved a series of results. Current research mainly concentrates on the integration and improvement of multiple algorithms, as well as the efficiency, computational complexity, and sensitivity of scheduling algorithms. Many scholars and researchers have studied and optimized cloud platform resource scheduling algorithms from various perspectives. However, these studies mainly focus on the existing resource scheduling level, rarely considering the changing patterns of computing resources required by different services, making it difficult to set optimal resource scheduling schemes for various application scenarios.

[0005] Existing scheduling methods rely solely on existing user load information. However, user load information in cloud platforms changes constantly and updates rapidly. Using existing information cannot accurately schedule and quickly respond to computing resources. When user load fluctuates significantly, computing resources need to be frequently scheduled, resulting in resource waste. Summary of the Invention

[0006] To overcome the shortcomings of the prior art, this invention discloses a cloud platform computing power scheduling method based on distributed computing.

[0007] The technical solution provided by this invention is: a cloud platform computing power scheduling method based on distributed computing, comprising the following steps:

[0008] S1: Decomposition and integration predictive analysis method to make short-term and ultra-short-term predictions of the load in the areas under the jurisdiction of each virtual resource pool.

[0009] S2: Establish a short-term load computing power demand prediction model based on LSTM;

[0010] S3: Establish an ultra-short-term load computing power demand prediction model based on XGBoost;

[0011] S4: Build a distributed computing cloud platform and various virtual resource pools to manage and optimize computing power.

[0012] Preferably, in the decomposition and integration analysis method of step S1, the time series data is decomposed into the following three components:

[0013] Trend: The trend reflects the long-term change of the mean of time series data over time;

[0014] Periodic factors: Periodic effects are trends that repeat at fixed intervals in a time series; periodic effects are simply repeating effects.

[0015] The impact of non-periodic factors: The impact of non-periodic factors is the remaining changes in the time series after any trend and periodic effects have been removed;

[0016] A simple model of time series data can be represented by an additive or multiplicative model, with the following formula:

[0017] Xt=mt+st+et (1)

[0018] Where mt represents trend, st represents periodic change, and et represents non-periodic change.

[0019] In a further preferred embodiment, in step S2, the LSTM is a chain structure, where Xt, ht, and Ct represent the input vector, hidden state, and module state at time t, respectively. The repeating unit has four network layers. By introducing a gate structure inside the module, information is deleted or added to the module state. The gate structure selects which information is deleted and which information can pass through, and adds a coefficient to the information that passes through the gate structure.

[0020] When LSTM analyzes and predicts three time series data separately, and integrates the prediction results as the final prediction result, the short-term load computing power demand prediction model can be constructed.

[0021] Further preferably, step S3 specifically includes:

[0022] The prediction model of the XGBoost algorithm is:

[0023]

[0024] In the formula, K represents the number of decision trees, and fk represents the prediction result of the k-th decision tree. The objective function of the XGBoost algorithm, representing the prediction result for sample i, is:

[0025]

[0026] The first term in the formula is the training error of sample I, and the second term is the sum of all decision tree regularization terms, where:

[0027]

[0028] In the formula, T is the number of leaf nodes in each decision tree, ω is the weight on the leaf node, and γ and λ are the corresponding coefficients.

[0029] The XGBoost algorithm determines whether to split a node and the threshold for splitting based on the change in the objective function before and after the split. It uses gain to evaluate the merits of each feature and feature split point, and then uses an approximation algorithm or a greedy algorithm to select features and split nodes. The formula for calculating the gain is:

[0030]

[0031] In the formula, GL is the sum of the first-order gradients of all samples on the left leaf node after splitting, HL is the sum of the second-order gradients of all samples on the left leaf node after splitting, and GR and HR are the sums of the first-order and second-order gradients of all samples on the right leaf node after splitting, respectively.

[0032] In a further preferred embodiment, in step S4, based on time-series feature data, a short-term prediction of the computing power required for the load in the area under the jurisdiction of the virtual resource pool is made. The time granularity is on the hourly level, and a short-term load prediction is made every 4 hours. Each prediction shows the change curve of the computing power required for the load in the next 4 hours. Each virtual resource pool management server will periodically report the predicted change curve of the computing power required for the load to the cloud platform's central management server. After receiving and aggregating the load change curves reported by each virtual resource pool, the cloud platform management server will analyze and process them, and divide all virtual resource pools in the cloud platform into different virtual resource pool groups according to computing power resources, network latency, and the change curve of the computing power required for the load.

[0033] In a further preferred embodiment, in step S4, the cloud platform management server collects all load prediction curve data at regular intervals (load prediction time intervals) and re-divides the virtual resource pool groups.

[0034] In a further preferred embodiment, in step S4, all virtual resource pools within the virtual resource pool group can perform operations such as computing power scheduling, computing task allocation, and computing data transmission. Based on the ultra-short-term computing power demand prediction curve of each virtual resource pool load, computing power resources, and network latency, computing power and computing tasks are allocated. The load prioritizes using the computing power of the local virtual resource pool. When computing power is insufficient, computing tasks are allocated to other virtual resource pools within the group for computing.

[0035] The beneficial effects of this invention are:

[0036] To achieve the effectiveness and real-time performance of cloud platform computing power scheduling, a cloud platform load prediction and computing power scheduling method based on distributed computing is proposed. The computing power required for the load tasks of each virtual resource pool is predicted in the short term and ultra-short term. Based on the decomposition and integration prediction technology, the LSTM (long short-term memory) model and the XGBoost model are used for prediction respectively.

[0037] The computing resources of a cloud platform can be divided into multiple virtual resource pools. The computing power and performance of different virtual resource pools may vary significantly. Based on the principle of allocating computing power as close as possible to reduce resource and energy consumption, the scope of each virtual resource pool is allocated according to its computing power. The decomposition-integration predictive analysis method is used to make short-term and ultra-short-term load predictions for the areas under the jurisdiction of each virtual resource pool. While ensuring prediction accuracy, the calculation speed of the prediction model is maximized and additional computing power consumption is reduced. Based on the prediction results, the virtual resource pools are grouped. Only virtual resource pools within the same group can perform resource scheduling and information transmission to improve the cluster resource utilization and balance. Attached Figure Description

[0038] Figure 1 A flowchart illustrating a cloud platform computing power scheduling method based on distributed computing provided by this invention; Figure 2 A schematic diagram of the LSTM module structure in a cloud platform computing power scheduling method based on distributed computing provided by the present invention; Figure 3 This invention provides a calculation roadmap for a short-term load prediction model in a cloud platform computing power scheduling method based on distributed computing. Figure 4 This invention provides a calculation roadmap for an ultra-short-term load prediction model in a cloud platform computing power scheduling method based on distributed computing. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] This invention provides a low-carbon demand response method for power systems based on carbon emission flows, such as... Figure 1 As shown, it includes the following steps:

[0041] S1: Decomposition and integration predictive analysis method to make short-term and ultra-short-term predictions of the load in the areas under the jurisdiction of each virtual resource pool.

[0042] The decomposition-integration method is essentially "decomposition followed by integration." By performing "multi-scale decomposition" on time series data, it effectively helps the model grasp the internal patterns of the data at different scales, thereby reducing modeling difficulty and improving prediction performance. Decomposition-integration analysis can more comprehensively and accurately consider various attributes of complex systems. Furthermore, this method effectively reduces the difficulty of modeling and improves the effectiveness and accuracy of predictive analysis. The time series data is decomposed into the following three components:

[0043] Trend: The trend reflects the long-term change of the mean of time series data over time;

[0044] Periodic factors: Periodic effects are trends that repeat at fixed intervals in a time series; periodic effects are simply repeating effects.

[0045] The impact of non-periodic factors: The impact of non-periodic factors is the remaining changes in the time series after any trend and periodic effects have been removed;

[0046] A simple model of time series data can be represented by an additive or multiplicative model, with the following formula:

[0047] Xt=mt+st+et (1)

[0048] Where mt represents trend, st represents periodic change, and et represents non-periodic change;

[0049] S2: Establish a short-term load computing power demand prediction model based on LSTM;

[0050] All recurrent neural networks (RNNs) have the form of a chain of repeating modules. In a typical RNN, this repeating module has a very simple structure, integrating the current input information with the memory information. LSTMs also have a chain structure, with the internal structure of the modules as follows: Figure 2 As shown;

[0051] LSTM is a chain structure. Xt, ht, and Ct represent the input vector, hidden state, and module state at time t, respectively. There are four network layers inside the repeating unit. By introducing gate structures inside the module, information is deleted or added to the module state. The gate structure selects which information is deleted and which information can pass through, and adds a coefficient to the information that passes through the gate structure.

[0052] When LSTM analyzes and predicts three time series data separately, and integrates the prediction results as the final prediction result, the construction of a short-term load computing power demand prediction model can be completed.

[0053] S3: Establish an ultra-short-term load computing power demand prediction model based on XGBoost;

[0054] Among them, the XGBoost algorithm is an ensemble learning method that combines decision trees as base learners. In the application, a series of decision tree models are trained and combined together as a strong learner for classification or prediction.

[0055] The prediction model of the XGBoost algorithm is as follows:

[0056]

[0057] In the formula, K represents the number of decision trees, and fk represents the prediction result of the k-th decision tree. The objective function of the XGBoost algorithm, representing the prediction result for sample i, is:

[0058]

[0059] The first term in the formula is the training error of sample I, and the second term is the sum of all decision tree regularization terms, where:

[0060]

[0061] In the formula, T is the number of leaf nodes in each decision tree, ω is the weight on the leaf node, and γ and λ are the corresponding coefficients.

[0062] The XGBoost algorithm determines whether to split a node and the threshold for splitting based on the change in the objective function before and after the split. It uses gain to evaluate the merits of each feature and feature split point, and then uses an approximation algorithm or a greedy algorithm to select features and split nodes. The formula for calculating the gain is:

[0063]

[0064] In the formula, GL is the sum of the first-order gradients of all samples on the left leaf node after splitting, HL is the sum of the second-order gradients of all samples on the left leaf node after splitting, and GR and HR are the sum of the first-order and second-order gradients of all samples on the right leaf node after splitting, respectively.

[0065] The ultra-short-term load forecasting model is similar to the short-term load forecasting model. It also decomposes the load characteristic data into three time series data points. XGBoost is used to analyze and forecast these three time series data points separately, and the forecast results are integrated as the final forecast result. This completes the ultra-short-term load data forecasting. The overall calculation process is as follows: Figure 3 As shown;

[0066] S4: Build a distributed computing cloud platform and various virtual resource pools to manage and optimize computing power scheduling;

[0067] To address issues such as the independent and dispersed nature of cloud platform computing resources, poor resource management and coordination, and uneven distribution of computing power, a computing cloud platform and various virtual resource pools are built to manage and optimize computing power scheduling.

[0068] Based on time-series characteristic data, short-term predictions of the computing power required by the load in the areas under the jurisdiction of virtual resource pools are made at the hourly level, with short-term load predictions every 4 hours. Each prediction forecasts the change curve of the computing power required by the load in the next 4 hours. Each virtual resource pool management server reports the predicted change curve of the computing power required by the load to the cloud platform's central management server at regular intervals. The cloud platform management server receives and aggregates the load change curves reported by each virtual resource pool, analyzes and processes them, and divides all virtual resource pools in the cloud platform into different virtual resource pool groups based on computing power resources, network latency, and the change curve of the computing power required by the load. Only within the same virtual resource pool group can virtual resource pools perform operations such as computing power scheduling, computing task allocation, and computing data transmission. Virtual resource pools belonging to different groups cannot transmit or allocate computing tasks or computing data. The cloud platform management server aggregates all load prediction curve data at regular intervals (load prediction intervals) and re-divides the virtual resource pool groups. This scheduling method can greatly reduce the amount of data transmission in the network, improve network utilization, and reduce the complexity of joint scheduling of network computing resources.

[0069] Within a virtual resource pool group, all virtual resource pools can perform operations such as computing power scheduling, computing task allocation, and computing data transmission. Based on the ultra-short-term computing power demand prediction curves, computing resources, and network latency of each virtual resource pool, computing power and computing tasks are allocated. The load prioritizes using the computing power of the local virtual resource pool; if computing power is insufficient, computing tasks are allocated to other virtual resource pools within the group. This is based on a distributed computing cloud platform computing power scheduling model, such as... Figure 4As shown in the diagram, only the parts connected by arrows can exchange information and data.

[0070] It should be understood that, inspired by the inventive concept, those skilled in the art can make various improvements and modifications based on the above description without departing from the scope of the invention, and these modifications still fall within the protection scope of the invention.

[0071] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0072] It should be understood that the present invention is not limited to the precise structure shown in the above description, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A cloud platform computing power scheduling method based on distributed computing, characterized in that, Includes the following steps: S1: Decomposition and integration predictive analysis method to make short-term and ultra-short-term predictions of the load in the areas under the jurisdiction of each virtual resource pool. S2: Establish a short-term load computing power demand prediction model based on LSTM; S3: Establish an ultra-short-term load computing power demand prediction model based on XGBoost; S4: Build a distributed computing cloud platform and various virtual resource pools to manage and optimize computing power.

2. The cloud platform computing power scheduling method based on distributed computing according to claim 1, characterized in that, In the decomposition and integration analysis method of step S1, the time series data is decomposed into the following three components: Trend: The trend reflects the long-term change of the mean of time series data over time; Periodic factors: Periodic effects are trends that repeat at fixed intervals in a time series; periodic effects are simply repeating effects. The impact of non-periodic factors: The impact of non-periodic factors is the remaining changes in the time series after any trend and periodic effects have been removed; A simple model of time series data can be represented by an additive or multiplicative model, with the following formula: Xt=mt+st+et (1) Where mt represents trend, st represents periodic change, and et represents non-periodic change.

3. The cloud platform computing power scheduling method based on distributed computing according to claim 1, characterized in that, In step S2, the LSTM is a chain structure, where Xt, ht, and Ct represent the input vector, hidden state, and module state at time t, respectively. The repeating unit has four network layers. By introducing gate structures inside the module, information is deleted or added to the module state. The gate structure selects which information is deleted and which information can pass through, and adds a coefficient to the information that passes through the gate structure. When LSTM analyzes and predicts three time series data separately, and integrates the prediction results as the final prediction result, the short-term load computing power demand prediction model can be constructed.

4. The cloud platform computing power scheduling method based on distributed computing according to claim 1, characterized in that, Step S3 specifically includes: The prediction model of the XGBoost algorithm is: In the formula, K represents the number of decision trees, and fk represents the prediction result of the k-th decision tree. The objective function of the XGBoost algorithm, representing the prediction result for sample i, is: The first term in the formula is the training error of sample I, and the second term is the sum of all decision tree regularization terms, where: In the formula, T is the number of leaf nodes in each decision tree, ω is the weight on the leaf node, and γ and λ are the corresponding coefficients. The XGBoost algorithm determines whether to split a node and the threshold for splitting based on the change in the objective function before and after the split. It uses gain to evaluate the merits of each feature and feature split point, and then uses an approximation algorithm or a greedy algorithm to select features and split nodes. The formula for calculating the gain is: In the formula, GL is the sum of the first-order gradients of all samples on the left leaf node after splitting, HL is the sum of the second-order gradients of all samples on the left leaf node after splitting, and GR and HR are the sums of the first-order and second-order gradients of all samples on the right leaf node after splitting, respectively.

5. The cloud platform computing power scheduling method based on distributed computing according to claim 1, characterized in that, In step S4, based on time-series feature data, short-term predictions of the computing power required for the load in the area under the jurisdiction of the virtual resource pool are made. The time granularity is on the hourly level, and short-term load predictions are made every 4 hours. Each prediction shows the change curve of the computing power required for the load in the next 4 hours. Each virtual resource pool management server will periodically report the predicted change curve of the computing power required for the load to the cloud platform's central management server. After receiving and aggregating the load change curves reported by each virtual resource pool, the cloud platform management server will analyze and process them, and divide all virtual resource pools in the cloud platform into different virtual resource pool groups according to computing power resources, network latency, and the change curve of the computing power required for the load.

6. The cloud platform computing power scheduling method based on distributed computing according to claim 1, characterized in that, In step S4, the cloud platform management server collects all load prediction curve data at regular intervals (load prediction time intervals) and re-divides the virtual resource pool groups.

7. The cloud platform computing power scheduling method based on distributed computing according to claim 1, characterized in that, In step S4, all virtual resource pools within the virtual resource pool group can perform operations such as computing power scheduling, computing task allocation, and computing data transmission. Based on the ultra-short-term computing power demand prediction curve of each virtual resource pool load, computing power resources, and network latency, computing power and computing tasks are allocated. The load prioritizes using the computing power of the local virtual resource pool. When computing power is insufficient, computing tasks are allocated to other virtual resource pools within the group for computing.

Images