Cloud platform resource usage prediction method based on big data
By collecting, cleaning, and processing historical data on a cloud platform, extracting key features, and using a linear regression algorithm to build a prediction model, the problem of inaccurate predictions in existing methods is solved, achieving more accurate resource usage predictions and service stability.
Patent Information
- Application Number
- CN202411356884.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2025-11-11
AI Technical Summary
Existing big data-based cloud platform resource usage prediction methods may oversimplify real-world scenarios, neglecting complexity and dynamic factors, leading to inaccurate predictions, especially when dealing with highly nonlinear or complexly interacting systems. Furthermore, these methods rely on large amounts of historical data, and prediction accuracy is affected by poor data quality or missing data.
Historical usage data is collected through the cloud platform's built-in monitoring and logging modules. The data is cleaned and transformed, outliers are identified and processed, key features are extracted, a predictive model is built using a linear regression algorithm, and the model's performance is evaluated through training and testing sets. Finally, it is applied to predict resource usage at future points in time.
It improves the comprehensiveness, accuracy, and real-time nature of data, reduces interference from irrelevant variables, enhances the interpretability and reliability of the model, and ensures the stability of the service and the accuracy of resource scheduling.
Smart Images
Figure CN120929237A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data processing technology, and more specifically, to a method for predicting cloud platform resource usage based on big data. Background Technology
[0002] The big data-based cloud platform resource usage prediction method is a way to predict future cloud resource demand using big data technology and machine learning algorithms. This method collects and analyzes historical resource usage data, and uses techniques such as time series analysis, machine learning, and deep learning to build predictive models to improve resource utilization efficiency and reduce costs while ensuring service quality.
[0003] A major drawback of existing big data-based cloud platform resource usage prediction methods is that the models may oversimplify real-world scenarios, ignoring certain complexities and dynamic factors. This simplification can lead to inaccurate predictions, especially when dealing with highly nonlinear or complexly interacting systems. Furthermore, these methods often rely on large amounts of historical data for training, but prediction accuracy can be affected by poor data quality or missing data. Therefore, this paper proposes a big data-based cloud platform resource usage prediction method. Summary of the Invention
[0004] The purpose of this invention is to provide a cloud platform resource usage prediction method based on big data, addressing a major drawback of existing big data-based cloud platform resource usage prediction methods mentioned in the background: the models may oversimplify real-world situations, ignoring certain complex and dynamic factors. This simplification can lead to inaccurate predictions, especially when dealing with highly nonlinear or complexly interacting systems. Furthermore, these methods often rely on large amounts of historical data for training, but prediction accuracy may be affected by poor data quality or missing data.
[0005] To achieve the above objectives, the present invention aims to provide a cloud platform resource usage prediction method based on big data, comprising the following steps:
[0006] S1. Collect historical usage data of cloud platform resources through the built-in monitoring and log recording modules of the cloud platform;
[0007] S2. Preprocess the collected historical usage data through cleaning and transformation;
[0008] S3. Extract key features from the collected historical usage data that are helpful for prediction;
[0009] S4. Construct a prediction model based on the linear regression algorithm;
[0010] S5. Train the prediction model using historical data and evaluate the performance of the prediction model;
[0011] S6. Apply the trained prediction model to predict resource usage at future points in time.
[0012] As a further improvement to this technical solution, in S1, the historical usage data of cloud platform resources includes CPU utilization and memory usage.
[0013] As a further improvement to this technical solution, the specific steps of data preprocessing in step S2 are as follows:
[0014] S2.1 Check if there are null values and non-numeric values in the dataset, and remove invalid and missing data;
[0015] S2.2 By standardizing and normalizing the data range, different historical data are transformed into a unified scale to avoid bias caused by scale differences;
[0016] S2.3 Identify and process outliers that deviate from the normal range based on the lQR algorithm.
[0017] As a further improvement to this technical solution, in S2.3, the specific steps for identifying and processing outliers deviating from the normal range based on the lQR algorithm are as follows:
[0018] Let Q1 be the first quartile (lower quartile) and Q2 be the third quartile (upper quartile), then IQR is Q3 - Q1;
[0019] The lower bound L and upper bound U of outliers can be expressed as:
[0020] L = Q1 - 1.5 × IQR;
[0021] U = Q³ + 1.5 × IQR;
[0022] Values less than the lower limit L or greater than the upper limit U are considered outliers and removed from the dataset;
[0023] Replace values less than the lower limit L with L, and replace values greater than the upper limit U with U.
[0024] As a further improvement to this technical solution, the specific steps for extracting key features that are helpful for prediction in step S3 are as follows:
[0025] S3.1. Determine relevant characteristic variables based on domain knowledge;
[0026] Among them, the relevant characteristic variables include CPU load, utilization, and memory usage;
[0027] S3.2 Calculate the basic statistical characteristics of key variables;
[0028] S3.3. Use the least squares method to fit the trend line model to extract the time series features of key features.
[0029] Among them, the time series features of key features represent features that can reflect data trends.
[0030] As a further improvement to this technical solution, in S3.3, the specific expression of the trend line model fitted using the least squares method is as follows:
[0031] y t =β0+β1t+ε t ;
[0032] Among them, y t ε represents the observed value at time t; t represents the position of the time series; β0 represents the intercept term of the trend line model; β1 represents the slope term of the model; ε t Represents the residual.
[0033] As a further improvement to this technical solution, the specific steps in S4 for constructing the prediction model based on the linear regression algorithm are as follows:
[0034] S4.1 Set the prediction model parameters: intercept α, slope γ;
[0035] S4.2 Define a loss function to measure the difference between the model's predicted value and the actual value;
[0036] S4.3 Find the optimal parameters intercept α and slope γ by minimizing the loss function;
[0037] S4.4 Output prediction model based on linear regression algorithm.
[0038] As a further improvement to this technical solution, in S4.4, the prediction model is specifically as follows:
[0039] y = α + γ1f1 + γ2f2 + ... + γ m f m +∈;
[0040] Where y represents CPU utilization and memory usage; f1...f m Representing relevant characteristic variables: ∈ represents the error term.
[0041] As a further improvement to this technical solution, the specific steps in S5 for training the prediction model using historical data and evaluating the performance of the prediction model are as follows:
[0042] S5.1 Divide historical usage data into training and testing sets;
[0043] S5.2 Training the prediction model using the training set:
[0044] S5.3. Evaluate the predictive performance of the prediction model using test set data based on mean squared error.
[0045] As a further improvement to this technical solution, in step S6, the specific steps for using the trained prediction model to predict resource usage at future points in time are as follows:
[0046] S6.1 Construct a vector containing the resource usage at future time points;
[0047] Specifically, the vector of resource usage at future points in time is as follows:
[0048] f future =[1 f 1future f 2future … f mfuture ];
[0049] Among them, f 1future ...f mfuture This represents the values of relevant characteristic variables at future points in time, i.e., the input data used to predict resource usage; f future A vector representing resource usage at future points in time; 1 represents the intercept term α.
[0050] S6.2 Use the trained model parameters to predict resource usage at future points in time;
[0051] Specifically, the predicted resource usage at future points in time is as follows:
[0052]
[0053] in, This indicates the amount of resources used at a future point in time. This represents the intercept obtained during training; This represents the slope obtained during training.
[0054] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0055] 1. This big data-based cloud platform resource usage prediction method collects historical usage data of cloud platform resources through the cloud platform's built-in monitoring and log recording modules, improving the comprehensiveness, accuracy, and real-time nature of the data. By removing outliers and filling in missing values, the quality and usability of the data are improved. Only features closely related to the prediction task are retained, reducing interference from irrelevant variables; the selected features are generally easier to understand and interpret, contributing to improved model interpretability.
[0056] 2. In this big data-based cloud platform resource usage prediction method, the prediction model built using the linear regression algorithm is easy to understand and implement, and convenient for debugging and maintenance; by dividing the training set and the test set, the generalization ability of the model can be evaluated; by predicting resource demand, the stability and reliability of the service can be ensured; the prediction results can help the cloud platform to make resource scheduling and expansion plans in advance. Attached Figure Description
[0057] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation
[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0059] Example:
[0060] Please see Figure 1 As shown, this embodiment provides a cloud platform resource usage prediction method based on big data, including the following steps:
[0061] S1. Collect historical usage data of cloud platform resources through the built-in monitoring and log recording modules of the cloud platform;
[0062] Specifically, the monitoring module is used to provide real-time monitoring of cloud resource usage and collect key performance indicators; the logging module is used to record all user operations, API calls and other important events, as well as system-level events.
[0063] In this example, the historical usage data of cloud platform resources includes CPU utilization and memory usage.
[0064] S2. Preprocess the collected historical usage data through cleaning and transformation;
[0065] In this example, the specific steps of data preprocessing are as follows:
[0066] S2.1 Check if there are null values and non-numeric values in the dataset, and remove invalid and missing data;
[0067] S2.2 By standardizing and normalizing the data range, different historical data are transformed into a unified scale to avoid bias caused by scale differences;
[0068] Specifically, the standardization is as follows:
[0069]
[0070] Where x represents the historical data value; μ represents the mean; σ represents the standard deviation; and z represents the standardized value.
[0071] Normalized to:
[0072]
[0073] Where, x min Indicates the minimum value; x max x represents the maximum value; x′ represents the normalized value;
[0074] Standardization and normalization processes ensure that historical data from different sources remain consistent on numerical scales, thus enabling them to be better applied to subsequent analysis and modeling tasks.
[0075] S2.3 Identify and process outliers that deviate from the normal range based on the IQR algorithm.
[0076] In this example, the specific steps for identifying and processing outliers that deviate from the normal range based on the IQR algorithm are as follows:
[0077] Let Q1 be the first quartile (lower quartile) and Q2 be the third quartile (upper quartile), then IQR is Q3 - Q1;
[0078] The lower bound L and upper bound U of outliers can be expressed as:
[0079] L = Q1 - 1.5 × IQR;
[0080] U = Q³ + 1.5 × IQR;
[0081] Values less than the lower limit L or greater than the upper limit U are considered outliers and removed from the dataset;
[0082] Replace values less than the lower limit L with L, and replace values greater than the upper limit U with U.
[0083] S3. Extract key features from the collected historical usage data that are helpful for prediction;
[0084] In this example, the specific steps for extracting key features that are helpful for prediction are as follows:
[0085] S3.1. Determine relevant characteristic variables based on domain knowledge;
[0086] Among them, the relevant characteristic variables include CPU load, utilization, and memory usage;
[0087] S3.2 Calculate the basic statistical characteristics of key variables;
[0088] Specifically, the basic statistical characteristics of key variables are calculated based on the mean, median, and standard deviation;
[0089] The mean is:
[0090]
[0091] Where, x i This represents the i-th observation; N represents the total number of observations.
[0092] Median:
[0093]
[0094] Where Median represents the median;
[0095] Standard deviation:
[0096]
[0097] S3.3. Use the least squares method to fit the trend line model to extract the time series features of key features.
[0098] Among them, the time series features of key features represent features that can reflect data trends.
[0099] In this example, the specific expression for the trendline model fitted using the least squares method is:
[0100] y t =β0+β1t+ε t ;
[0101] Among them, y t ε represents the observed value at time t; t represents the position of the time series; β0 represents the intercept term of the trend line model, indicating the expected resource usage at t=0, i.e., the starting point of the time series, representing the baseline level of the model; β1 represents the slope term of the model, indicating the expected rate of change of resource usage as time t increases; if β1>0, it indicates that CPU utilization and memory usage are increasing over time; if β1<0, it indicates that CPU utilization and memory usage are decreasing over time; if β1=0, it indicates that CPU utilization and memory usage have no obvious time trend; t Represents the residual.
[0102] S4. Construct a prediction model based on the linear regression algorithm;
[0103] In this example, the specific steps for building a prediction model based on the linear regression algorithm are as follows:
[0104] S4.1 Set the prediction model parameters: intercept α, slope γ;
[0105] S4.2 Define a loss function to measure the difference between the model's predicted value and the actual value;
[0106] Specifically, if the loss function is the mean squared error algorithm, then the mean squared error algorithm is as follows:
[0107]
[0108] Where n represents the number of observations; y i Indicates the actual amount of resources used;
[0109] MSE provides a quantitative metric for evaluating the performance of a predictive model, reflecting the gap between model predictions and actual observations.
[0110] S4.3 Find the optimal parameters intercept α and slope γ by minimizing the loss function;
[0111] Specifically, if the gradient descent method is chosen to minimize the loss function, then the gradient descent method is as follows:
[0112]
[0113]
[0114] Where η represents the learning rate; α new Represents the optimal intercept parameter; γ new Indicates the optimal slope parameter;
[0115] Optimal parameters allow the model's predictions to more closely approximate actual observations. By adjusting the model parameters to minimize the loss function, the model's prediction accuracy can be improved.
[0116] S4.4 Output prediction model based on linear regression algorithm.
[0117] In this example, the prediction model is specifically as follows:
[0118] y = α + γ1f1 + γ2f2 + ... + γ m f m +∈;
[0119] Where y represents CPU utilization and memory usage; f1...f m ∈ represents the relevant feature variables; ∈ represents the error term.
[0120] S5. Train the prediction model using historical data and evaluate the performance of the prediction model;
[0121] In this example, the specific steps for training a prediction model using historical data and evaluating its performance are as follows:
[0122] S5.1 Divide historical usage data into training and testing sets;
[0123] S5.2. Train the prediction model using the training set;
[0124] Specifically, the steps involved in training the prediction model are as follows:
[0125] First, design a matrix X;
[0126]
[0127] Each row represents a feature vector of an observation, and the first column is a vector of all 1s, used to represent the intercept term.
[0128] Construct a vector to provide the actual observed values for the target variable so that they can be compared with the model's predicted values;
[0129]
[0130] Where z represents the target vector;
[0131] Finally, the least squares estimation is used;
[0132]
[0133] in, Represents the estimated parameter vector; (X) T X) -1 X represents the inverse matrix of the transpose of the design matrix X multiplied by itself; T This represents the transpose of the design matrix X;
[0134] S5.3. Evaluate the predictive performance of the prediction model using test set data based on mean squared error.
[0135] Specifically, the mean squared error is:
[0136]
[0137] Among them, y i Indicates the actual resource usage in the test set; f 1i ...f mi This represents the values of the feature variables in the test set; This represents the intercept obtained during training; N represents the slope obtained during training. test Indicates the size of the test set.
[0138] S6. Apply the trained prediction model to predict resource usage at future points in time.
[0139] In this example, the specific steps for using a trained prediction model to predict resource usage at future points in time are as follows:
[0140] S6.1 Construct a vector containing the resource usage at future time points;
[0141] Specifically, the vector of resource usage at future points in time is as follows:
[0142] f future =[1 f 1future f 2future … f mfuture ];
[0143] Among them, f 1future ...f mfuture This represents the values of relevant characteristic variables at future points in time, i.e., the input data used to predict resource usage; f future A vector representing resource usage at future points in time; 1 represents the intercept term α.
[0144] S6.2 Use the trained model parameters to predict resource usage at future points in time;
[0145] Specifically, the predicted resource usage at future points in time is as follows:
[0146]
[0147] in, This indicates the amount of resources used at a future point in time. This represents the intercept obtained during training; This represents the slope obtained during training.
[0148] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.
Claims
1. A cloud platform resource usage prediction method based on big data, characterized in that: Includes the following steps: S1. Collect historical usage data of cloud platform resources; S2. Preprocess the collected historical usage data through cleaning and transformation; S3. Extract key features from the collected historical usage data that are helpful for prediction; S4. Construct a prediction model based on the linear regression algorithm; S5. Train the prediction model using historical data and evaluate the performance of the prediction model; S6. Apply the trained prediction model to predict resource usage at future points in time.
2. The cloud platform resource usage prediction method based on big data according to claim 1, characterized in that: In S1, the historical usage data of cloud platform resources includes CPU utilization and memory usage.
3. The cloud platform resource usage prediction method based on big data according to claim 1, characterized in that: In step S2, the specific steps of data preprocessing are as follows: S2.1 Check if there are null values and non-numeric values in the dataset, and remove invalid and missing data; S2.2 By standardizing and normalizing the data range, different historical data are transformed into a unified scale to avoid bias caused by scale differences; S2.3 Identify and handle outliers that deviate from the normal range.
4. The cloud platform resource usage prediction method based on big data according to claim 1, characterized in that: In S2.3, the specific steps for identifying and processing outliers that deviate from the normal range based on the IQR algorithm are as follows: Let Q1 be the first quartile (lower quartile) and Q2 be the third quartile (upper quartile), then IQR is Q3 - Q1; The lower bound L and upper bound U of outliers can be expressed as: L = Q1 - 1.5 × IQR; U = Q³ + 1.5 × IQR; Values less than the lower limit L or greater than the upper limit U are considered outliers and removed from the dataset; Replace values less than the lower limit L with L, and replace values greater than the upper limit U with U.
5. The cloud platform resource usage prediction method based on big data according to claim 1, characterized in that: In step S3, the specific steps for extracting key features that aid in prediction are as follows: S3.
1. Determine relevant characteristic variables based on domain knowledge; Among them, the relevant characteristic variables include CPU load, utilization, and memory usage; S3.2 Calculate the basic statistical characteristics of key variables; S3.
3. Use the least squares method to fit the trend line model to extract the time series features of key features; Among them, the time series features of key features represent features that can reflect data trends.
6. The cloud platform resource usage prediction method based on big data according to claim 1, characterized in that: In S3.3, the specific expression of the trendline model fitted using the least squares method is as follows: yt=β0+β1t+εt; Among them, y t ε represents the observed value at time t; t represents the position of the time series; β0 represents the intercept term of the trend line model; β1 represents the slope term of the model; ε t Represents the residual.
7. The cloud platform resource usage prediction method based on big data according to claim 1, characterized in that: In step S4, the specific steps for constructing the prediction model based on the linear regression algorithm are as follows: S4.1 Set the prediction model parameters: intercept α, slope γ; S4.2 Define a loss function to measure the difference between the model's predicted value and the actual value; S4.3 Find the optimal parameters intercept α and slope γ by minimizing the loss function; S4.4 Output prediction model based on linear regression algorithm.
8. The cloud platform resource usage prediction method based on big data according to claim 1, characterized in that: In S4.4, the prediction model is specifically as follows: y=α+γ1f1+γ2f2+...+γ m f m +∈; Where y represents CPU utilization and memory usage; f1...f m ∈ represents the relevant feature variables; ∈ represents the error term.
9. The cloud platform resource usage prediction method based on big data according to claim 1, characterized in that: In step S5, the specific steps for training the prediction model using historical data and evaluating the performance of the prediction model are as follows: S5.1 Divide historical usage data into training and testing sets; S5.
2. Train the prediction model using the training set; S5.
3. Evaluate the predictive performance of the prediction model using test set data based on mean squared error.
10. The cloud platform resource usage prediction method based on big data according to claim 1, characterized in that: In step S6, the specific steps for using the trained prediction model to predict resource usage at future points in time are as follows: S6.1 Construct a vector containing the resource usage at future time points; Specifically, the vector of resource usage at future points in time is as follows: f future =[1 f 1future f 2future …f mfuture ]; Among them, f 1future ...f mfuture This represents the values of relevant characteristic variables at future points in time, i.e., the input data used to predict resource usage; f future A vector representing resource usage at future points in time; 1 represents the intercept term α. S6.2 Use the trained model parameters to predict resource usage at future points in time; Specifically, the predicted resource usage at future points in time is as follows: in, This indicates the amount of resources used at a future point in time. This represents the intercept obtained during training; This represents the slope obtained during training.