Private cloud resource usage prediction method and system based on combinatorial optimization algorithm

Abstract

The invention discloses a private cloud resource usage prediction method and system based on a combinatorial optimization algorithm, and the method comprises the steps: collecting historical memory related data, and carrying out the data processing of the historical memory related data, and obtaining the processed memory related data; performing feature screening on the processed memory related data by adopting an SVM-RFE algorithm to obtain optimal feature data; based on the optimal feature data, obtaining an optimal parameter combination through a grid search mode; inputting the optimal parameter combination into a resource consumption prediction model constructed based on an XGBoost algorithm, and training the resource consumption prediction model by using the processed memory related data; and utilizing the trained resource consumption prediction model to predict the memory occupancy of a future predetermined number of days. According to the invention, the accuracy of private cloud environment resource usage configuration is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for predicting private cloud resource usage based on a combinatorial optimization algorithm. Background Art

[0002] In recent years, cloud platforms have garnered widespread attention for their numerous advantages, including affordable and scalable computing resources, stable and high-capacity storage, and flexible, anytime, anywhere access. To improve resource utilization on cloud platforms, researchers have proposed numerous scheduling algorithms. These algorithms use prediction models based on historical data to predict future resource usage. However, as demand increases, the demand for accurate predictions of future resource usage is becoming increasingly stringent. Greater prediction errors mean greater resource waste. Reducing resource waste in private clouds has become a major challenge, and accurately predicting resource usage has become a key research focus in the private cloud field.

[0003] There are currently three main methods for predicting cloud platform resource usage:

[0004] ① Offline workload analysis: First, let the task run for a period of time, use the recommendation algorithm to determine its possible resource utilization, and then reschedule. In practice, offline analysis is not feasible because in actual production, the workload input is usually not available before the task runs.

[0005] ② A detailed analysis of resource usage in the cloud platform shows that the cloud platform has low resource utilization and predictability of cloud platform resource utilization, and a random forest regression model is used to predict the resource usage of virtual machines, but the performance is poor.

[0006] ③ The resource utilization of tasks in the cloud platform is divided into three levels: high, medium and low. Classification technology is used to predict the resource utilization level of tasks. However, this classification method is based on coarse granularity and cannot effectively help the scheduler allocate resources.

[0007] Most of the above methods are aimed at public resources, using data analysis to study the usage of public cloud resources or using other models to predict resource occupancy. However, neither the granularity of the prediction nor the accuracy of the prediction results can meet the needs of private cloud environments for resource usage prediction.

[0008] Furthermore, in the prior art, resource allocation in a private cloud environment often uses an experience-based approach to determine resource usage, which often leads to problems such as over-allocation of resources and low resource utilization. Summary of the Invention

[0009] The present invention aims to provide a private cloud resource usage prediction method and system based on a combinatorial optimization algorithm to improve the accuracy of resource usage configuration in a private cloud environment.

[0010] To achieve the above objectives, the present invention provides a private cloud resource usage prediction method based on a combinatorial optimization algorithm, comprising:

[0011] Collecting historical memory-related data, and performing data processing on the historical memory-related data to obtain processed memory-related data;

[0012] The SVM-RFE algorithm is used to perform feature screening on the processed memory-related data to obtain the optimal feature data;

[0013] Based on the optimal feature data, an optimal parameter combination is obtained through a grid search method;

[0014] Input the optimal parameter combination into the resource usage prediction model built based on the XGBoost algorithm, and use the processed memory-related data to train the resource usage prediction model;

[0015] The trained resource usage prediction model is used to predict memory usage for a predetermined number of days in the future.

[0016] Furthermore, the historical memory-related data is processed to obtain processed memory-related data, including:

[0017] The historical memory-related data is the index data related to the memory usage of the computing node group nodes;

[0018] Data processing methods include: data analysis, and handling of missing values ​​and outliers;

[0019] The data analysis of the historical memory related data includes:

[0020] Batch parsing the historical memory-related data into a recognizable fixed format; wherein the recognizable fixed format includes: converting timestamps and times, separating data, filtering redundant information, and rearranging data;

[0021] Processing missing values ​​and outliers on the historical memory-related data includes:

[0022] Calculate the mean of a predetermined number of data around the abnormal data in the historical memory related data, and replace the abnormal data with the calculated mean; and fill in a predetermined number of data around the missing data in the historical memory related data.

[0023] Furthermore, the SVM-RFE algorithm is used to perform feature screening on the processed memory-related data to obtain the optimal feature data, including:

[0024] Step a: Divide the processed memory-related data into training set data and test set data;

[0025] Step b: Set the current feature subset and the optimal feature subset; wherein, in the initial state, the current feature subset contains all training set data, the optimal feature subset is empty, the initial accuracy w=0.0, the number of features deleted in each iteration is set to K, and the number of features retained is set to M;

[0026] Step c: construct an SVM model based on the current feature subset, calculate the accuracy value w of each feature data in the current feature subset, and sort each feature data in the current feature subset in descending order according to the accuracy value w, and delete the K feature data with the lowest accuracy value in the current feature subset.

[0027] Step d: Repeat step c until the current feature subset cannot be iterated further, that is, when the number of features in the current feature subset is M, return the optimal feature subset;

[0028] The optimal feature data in the returned optimal feature subset are the M feature data remaining in the current feature subset after the iteration is completed.

[0029] Furthermore, the prediction method further includes calculating an objective function of a resource usage prediction model and sorting the objective function of the resource usage prediction model to improve the accuracy of the resource usage prediction model, including:

[0030] Calculate the objective function of the resource usage prediction model; where the objective function of the resource usage prediction model is the objective function of the base learner plus the objective function of the new learner;

[0031] Set a penalty term to limit the number of leaf nodes in the decision tree of the XGBoost algorithm;

[0032] Based on the objective function of the resource usage prediction model, a penalty term is added to obtain a total objective function including the penalty term, and the total objective function is sorted.

[0033] Furthermore, the objective function of the resource usage prediction model is:

[0034]

[0035] Where, Represents the prediction result of the i-th sample at the t-th iteration, whose value is the sum of the previous prediction result and the new learner function f t (xi )The sum of the prediction results; f t (x i ) represents the prediction result of the t-th base learner; k represents the k-th learner; i represents the number of samples, x i represents the i-th sample, and t represents the number of iterations, that is, the number of decision trees added;

[0036] The penalty terms are:

[0037]

[0038] Where, Ω(f t ) represents the penalty term after adding the t-th base learner, γ represents the penalty intensity, λ represents the parameter; ω represents the weight of each leaf node in the decision tree; T represents the number of leaf nodes in the decision tree;

[0039] The overall objective function is:

[0040]

[0041] Where, Obj (t) represents the overall objective function of the resource usage prediction model, represents the loss function of resource usage prediction model training, n represents the total number of samples, i represents the number of samples, y i represents the true value of the i-th sample, represents the prediction result of the i-th sample in the first t iterations, and C represents a constant.

[0042] Furthermore, the overall objective function is organized, including:

[0043] The total objective function is approximated using a second-order Taylor expansion, and the Taylor expansion form of the total objective function is:

[0044]

[0045] Where f(x+Δx) represents the Taylor expansion of the total objective function, where x corresponds to Δx corresponds to f t (x i ), represents the first-order derivative of f(x), represents the second-order derivative of f(x);

[0046] According to the total objective function formula and the Taylor expansion form of the total objective function, the following expression is defined:

[0047]

[0048] Where, f t (x i) represents the prediction result of the t-th base learner, represents the loss function of resource usage prediction model training; g i represents the first-order derivative of the Taylor expansion, h i represents the second derivative of the Taylor expansion; Express request The first-order derivative of , t represents the number of iterations, that is, the number of decision trees added, y i represents the true value of the i-th sample, represents the prediction result of the i-th sample in the first t iterations, Express request The second derivative of

[0049] Based on the total objective function formula, the Taylor expansion form of the total objective function and the defined expression (5), the final expression of the total objective function is:

[0050]

[0051] Where, Obj (t) represents the overall objective function of the resource usage prediction model, Ω(f t ) represents the penalty term after adding the t-th base learner.

[0052] Based on the same inventive concept, the present invention also provides a private cloud resource usage prediction system based on a combinatorial optimization algorithm, comprising: a collection unit, a feature screening unit, a model building unit, and a prediction unit.

[0053] The collecting unit is used to collect historical memory related data and perform data processing on the historical memory related data to obtain processed memory related data;

[0054] The feature screening unit is used to perform feature screening on the processed memory-related data using the SVM-RFE algorithm to obtain optimal feature data;

[0055] The model building unit is used to obtain an optimal parameter combination through a grid search method based on the optimal feature data; input the optimal parameter combination into a resource usage prediction model built based on the XGBoost algorithm, and train the resource usage prediction model using the processed memory-related data;

[0056] The prediction unit is used to use the trained resource usage prediction model to predict the memory usage for a predetermined number of days in the future.

[0057] Furthermore, the prediction system further includes a calculation unit,

[0058] The calculation unit is used to calculate the objective function of the resource usage prediction model and sort out the objective function of the resource usage prediction model to improve the accuracy of the resource usage prediction model;

[0059] Among them, the computing unit is specifically used to: calculate the objective function of the resource usage prediction model; set a penalty term to limit the number of leaf nodes in the decision tree of the XGBoost algorithm; based on the objective function of the resource usage prediction model, add a penalty term to obtain the total objective function including the penalty term, and organize the total objective function.

[0060] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including: a memory and a processor; the processor is used to read and execute the computer program stored in the memory to implement the aforementioned private cloud resource usage prediction system based on a combinatorial optimization algorithm.

[0061] Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, which, when executed, implement the aforementioned private cloud resource usage prediction system based on a combinatorial optimization algorithm.

[0062] The technical effects and advantages of the present invention are as follows: The present invention proposes a usage prediction model for private cloud resources based on a combinatorial optimization algorithm. The method performs feature screening on the collected indicator data related to memory usage, and uses a combination of recursive feature elimination (SVM-RFE) and extreme gradient boosting (XGBoost) algorithms to perform feature screening on the existing feature data to screen out features closely related to resource usage. Then, a resource usage prediction model based on the XGBoost algorithm is built for the screened features, and the load of computing resources is predicted in combination with the actual resource requirements of users. The present invention has higher prediction accuracy and smaller prediction error, and adopts a data-driven approach to realize intelligent prediction of resource usage in a private cloud environment, thereby improving resource utilization and providing decision support for optimal resource allocation.

[0063] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0065] Figure 1 This is a flowchart of the steps of a method for predicting private cloud resource usage based on a combined optimization algorithm according to an embodiment of the present invention;

[0066] Figure 2 A flow chart for model construction of an embodiment of the present invention;

[0067] Figure 3 A schematic diagram of an algorithm flow for performing data screening on feature data according to an embodiment of the present invention;

[0068] Figure 4 This is a schematic diagram of the structure of a private cloud resource usage prediction system based on a combinatorial optimization algorithm according to an embodiment of the present invention;

[0069] Figure 5 The figure is a schematic structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0071] To address the deficiencies of the prior art, the present invention discloses a private cloud resource usage prediction method based on a combined optimization algorithm. Figure 1 As shown, the following steps are included:

[0072] Step 1: Collect historical memory-related data from the private cloud platform and process the historical memory-related data to obtain processed memory-related data;

[0073] Among them, the historical memory-related data is the indicator data related to the memory usage of the computing node group node, a total of 47 indicator data, and the specific related indicator data are shown in Table 1:

[0074] Table 1

[0075]

[0076]

[0077] Data processing methods for historical memory related data include: data analysis, missing value and outlier processing, etc.

[0078] The data parsing of the historical memory related data includes: batch parsing the historical memory related data into a recognizable fixed format; wherein the recognizable fixed format includes: conversion of timestamp and time, separation of data, filtering of redundant information and data rearrangement.

[0079] Processing missing values ​​and outliers in historical memory data includes: calculating the mean of four data points surrounding the outlier in the historical memory data, replacing the outlier with the mean; and filling in the four data points surrounding the missing data in the historical memory data. The present invention uses the four data points surrounding the missing data for filling in because the data collection interval is 15 seconds, and analysis of the data morphology reveals that the data does not undergo significant changes within 1 minute.

[0080] Step 2: To reduce the number of features input into the resource usage prediction model, the SVM-RFE algorithm is used to perform feature screening on the processed memory-related data to obtain the optimal feature data; Figure 2 and Figure 3 Shown, including:

[0081] Step a: Divide the processed memory-related data into a ratio of 7:3 to generate training set data and test set data;

[0082] Step b: Set the current feature subset and the optimal feature subset; wherein, in the initial state, the current feature subset contains all training set data, the optimal feature subset is empty, the initial accuracy w=0.0, the number of features deleted in each iteration is set to K, and the number of features retained is set to M;

[0083] Step c: construct an SVM model based on the current feature subset, calculate the accuracy value w of each feature data in the current feature subset, and sort each feature data in the current feature subset in descending order according to the accuracy value w, and delete the K feature data with the lowest accuracy value in the current feature subset.

[0084] Step d: Repeat step c until the current feature subset cannot be iterated further, that is, when the number of features in the current feature subset is M, return the optimal feature subset;

[0085] The optimal feature data in the returned optimal feature subset are the M feature data remaining in the current feature subset after the iteration is completed.

[0086] In summary, the core of the SVM-RFE algorithm is to sort all feature data according to their influence on the resource usage prediction model. Feature data with smaller influence will be deleted sooner, and then the remaining features will be remodeled. This algorithm is repeated until no iteration is possible (that is, the number of features reaches the set retention number).

[0087] Among them, the target data memory usage is:

[0088]

[0089] Here, z represents the memory usage, memory_MemAvailable_bytes represents the number of available memory bytes, and memory_MemTotal_bytes represents the total number of memory bytes.

[0090] After feature screening of all relevant indicator data in Table 1, 9 optimal feature data were finally obtained. These 9 optimal feature data were consistent with the feature data selected based on expert experience, indicating that the feature screening of the present invention is effective. The 9 optimal feature data are shown in Table 2:

[0091] Table 2 Optimal feature data after feature screening

[0092] Eigenvalue memory_Buffers_bytes memory_Inactive_anon_bytes memory_inactive_bytes memory_inactive_file_bytes memory_MemAvailable_bytes memory_Shmem_bytes memory_slab_bytes memory_SReclaimable_bytes memory_SUnreclaim_bytes .

[0093] Here, the feature screening process is illustrated by example:

[0094] If there are 5 features (ABCDE) in total, and the number of features deleted each time is K = 2, then in the first iteration, the current combination of all features is 5, that is, combination 1 = (BCDE), combination 2 = (ACDE), combination 2 = (ABDE), combination 4 = (ABCE), and combination 5 = (ABCD).

[0095] Each combination is input into the SVM model to obtain an accuracy rate, which is W1, W2, W3, W4, and W5 respectively. Assuming W1 = 0.9, W2 = 0.8, W3 = 0.7, W4 = 0.6, and W5 = 0.5, the accuracy rates are sorted in descending order: W1, W2, W3, W4, and W5.

[0096] Generally, a higher accuracy indicates that removing that feature data has less impact on the SVM model's performance, meaning that the feature data is less important to the model. Therefore, the order of influence of each feature data on the SVM model based on accuracy is EDCBA. Since K = 2, the features removed in the first iteration are AB, and the current feature subset is = (EDC). The above cycle is repeated for EDC until the current number of features is empty or the set number of features is retained.

[0097] The number of features finally selected can be set to 0.2 times the number of original features. The final number of preferred features is 1. In this way, the remaining feature data is the most important, that is, the optimal feature data to be retained, and the optimal feature subset can be obtained.

[0098] Step 3: Based on the optimal feature data, the optimal parameter combination is obtained through grid search, wherein the parameters after grid search are learning_rate=0.1, max_depth=10, gamma=1, n_estimators=100, min_child_weight=6.

[0099] Among them, learning_rate (represents the learning rate): the learning rate controls the step size of each update; a smaller learning rate can make the resource usage prediction model more stable, but may require more iterations; a larger learning rate can speed up the learning speed of the resource usage prediction model, but may cause instability and oscillation. The parameter range set in this embodiment is:

[0100] [0.001,0.01,0.1,0.5].

[0101] max_depth (indicates the maximum depth of the tree): The maximum depth of the decision tree; controls the complexity of the decision tree and the capacity of the resource usage prediction model. Increasing the maximum depth can improve the fitting ability of the resource usage prediction model, but may also lead to overfitting. In this example, the parameter range is set to [3, 5, 10, 12, 20, 30].

[0102] Gamma (gamma value): controls the condition for splitting leaf nodes in the tree. When the change in the loss function of a leaf node after splitting is less than gamma, the split is stopped. In this example, the parameter range is set to [0, 0.5, 0.8, 1].

[0103] n_estimators (indicates the number of trees): the number of decision trees, also known as the number of iterations; increasing the number of decision trees can improve the fitting ability of the resource usage prediction model, but it will also increase the calculation time of resource usage prediction. The parameter range set in this embodiment is:

[0104] [60,70,80,100,110,120].

[0105] min_child_weight (indicates the minimum weight of a leaf node): the minimum sum of weights of samples on a leaf node; if the sum of the weights of samples on a leaf node in the decision tree is less than min_child_weight, the split is stopped. The parameter range set in this embodiment is:

[0106] [4,5,6,7,8].

[0107] Step 4: Input the optimal parameter combination into the resource usage prediction model built based on the XGBoost algorithm, and use the processed memory-related data to train the resource usage prediction model; including:

[0108] Step 4.1: Calculate the objective function of the resource usage prediction model; specifically,

[0109] Since the XGBoost algorithm is an improved method based on gradient boosting decision trees, it improves the accuracy of the algorithm by generating new trees through iterative training to fit the residuals of the previous tree; the algorithm idea is to build a base learner, add new learners on the basis of this base learner, and calculate the objective function value once each learner is added to ensure that the objective function value gradually decreases during the iteration process.

[0110] Then the objective function of the resource usage prediction model is the objective function of the base learner plus the objective function of the new learner. The objective function of the resource usage prediction model is:

[0111]

[0112]

[0113] Where, Represents the prediction result of the i-th sample at the t-th iteration, whose value is the sum of the previous prediction result and the new learner function f t (x i )The sum of the prediction results; f t (x i ) represents the prediction result of the t-th base learner; k represents the k-th learner; i represents the number of samples, x i represents the i-th sample; t represents the number of iterations, that is, the number of decision trees added.

[0114] Step 4.2: Since there are too many leaf nodes in the decision tree of the XGBoost algorithm, the resource usage prediction model may be at risk of overfitting, and the risk increases accordingly. Set a penalty term to limit the number of leaf nodes in the decision tree of the XGBoost algorithm.

[0115] The penalty items are:

[0116]

[0117] Where, Ω(f t) represents the penalty term after adding the t-th base learner, γ represents the penalty intensity, λ represents the parameter; ω represents the weight of each leaf node in the decision tree; T represents the number of leaf nodes in the decision tree.

[0118] Step 4.3: Based on the objective function of the resource usage prediction model, a penalty term is added to obtain a total objective function including the penalty term, and the total objective function is sorted;

[0119] Among them, the overall objective function is:

[0120]

[0121] Where, Obj (t) represents the overall objective function of the resource usage prediction model, represents the loss function of resource usage prediction model training, n represents the total number of samples, i represents the number of samples, and t represents the number of iterations, that is, the number of decision trees added; y i represents the true value of the i-th sample, represents the prediction result of the i-th sample in the first t iterations, and C represents a constant.

[0122] The overall objective function is organized, including:

[0123] The total objective function is approximated using a second-order Taylor expansion, and the Taylor expansion form of the total objective function is:

[0124]

[0125] Where f(x+Δx) represents the Taylor expansion of the total objective function, where x corresponds to Δx corresponds to f t (x i ); represents the first-order derivative of f(x), represents the second-order derivative of f(x);

[0126] According to the total objective function formula and the Taylor expansion form of the total objective function, the following expression is defined:

[0127]

[0128] Where, f t (x i ) represents the prediction result of the t-th base learner, Represents the loss function of resource usage prediction model training, since The value is fixed and can be regarded as a constant; g i represents the first-order derivative of the Taylor expansion, h i represents the second derivative of the Taylor expansion; Express request The first-order derivative of , t represents the number of iterations, that is, the number of decision trees added, y i represents the true value of the i-th sample, represents the prediction result of the i-th sample in the first t iterations, Express request The second derivative of .

[0129] Based on the total objective function formula, the Taylor expansion form of the total objective function and the defined expression (5), the final expression of the total objective function is:

[0130]

[0131] Where, Obj (t) represents the overall objective function of the resource usage prediction model, Ω(f t ) represents the penalty term after adding the t-th base learner, t represents the number of iterations, g i represents the first-order derivative of the Taylor expansion, f t (x i ) represents the prediction result of the t-th base learner.

[0132] Step 5: using the trained resource usage prediction model to predict memory usage for a predetermined number of days in the future; including:

[0133] Determine whether the accuracy of the trained resource usage prediction model meets the requirements;

[0134] If the resource usage prediction model meets the accuracy requirement, the trained model is tested using the test set data to predict the data memory usage for the next week, and the test results are then output.

[0135] If the accuracy of the resource usage prediction model does not meet the requirements, the training set data is used to train the resource usage prediction model.

[0136] Based on the same inventive concept, the present invention also provides a private cloud resource usage prediction system based on a combination optimization algorithm, such as Figure 4 As shown, it includes: acquisition unit, feature screening unit, model building unit and prediction unit,

[0137] an acquisition unit, configured to acquire historical memory-related data and perform data processing on the historical memory-related data to obtain processed memory-related data;

[0138] A feature screening unit is used to perform feature screening on the processed memory-related data using the SVM-RFE algorithm to obtain optimal feature data;

[0139] A model building unit is configured to obtain an optimal parameter combination through a grid search method based on the optimal feature data; the optimal parameter combination is input into a resource usage prediction model built based on the XGBoost algorithm, and the resource usage prediction model is trained using the processed memory-related data;

[0140] The prediction unit is used to use the trained resource usage prediction model to predict the memory usage for a predetermined number of days in the future.

[0141] In some specific embodiments, the prediction system further includes a calculation unit, which is used to calculate the objective function of the resource usage prediction model and sort the objective function of the resource usage prediction model to improve the accuracy of the resource usage prediction model;

[0142] Among them, the computing unit is specifically used to: calculate the objective function of the resource usage prediction model; set the penalty term to limit the number of leaf nodes in the decision tree of the XGBoost algorithm; based on the objective function of the resource usage prediction model, add the penalty term to obtain the total objective function including the penalty term, and organize the total objective function.

[0143] Regarding the system in the above embodiment, the specific manner in which each unit module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0144] Based on the same inventive concept, an embodiment of the present invention further provides an electronic device, the structure of which is as follows: Figure 5 As shown, it includes: a memory and a processor, and the processor is used to read and execute the computer program stored in the memory to implement the aforementioned private cloud resource usage prediction method based on the combined optimization algorithm.

[0145] Based on the same inventive concept, an embodiment of the present invention further provides a computer storage medium, wherein the computer storage medium stores computer executable instructions, which, when executed, implement the aforementioned private cloud resource usage prediction method based on the combinatorial optimization algorithm.

[0146] The present invention proposes a private cloud resource usage prediction model based on a combinatorial optimization algorithm. The method performs feature screening on 47 indicator data related to memory usage collected. By combining recursive feature elimination (SVM-RFE) and extreme gradient boosting (XGBoost) algorithms, the existing feature data is subjected to feature screening to screen out features closely related to resource usage. Then, a resource usage prediction model based on the XGBoost algorithm is built for the screened features to predict the computing resource load in combination with the user's actual resource demand.

[0147] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A private cloud resource usage prediction method based on a combinatorial optimization algorithm, characterized in that: include: Collecting historical memory-related data, and performing data processing on the historical memory-related data to obtain processed memory-related data; The SVM-RFE algorithm is used to perform feature screening on the processed memory-related data to obtain the optimal feature data; Based on the optimal feature data, an optimal parameter combination is obtained through a grid search method; Input the optimal parameter combination into the resource usage prediction model built based on the XGBoost algorithm, and use the processed memory-related data to train the resource usage prediction model; The trained resource usage prediction model is used to predict memory usage for a predetermined number of days in the future.

2. The private cloud resource usage prediction method based on the combinatorial optimization algorithm according to claim 1 is characterized in that: Processing the historical memory-related data to obtain processed memory-related data includes: The historical memory-related data is the index data related to the memory usage of the computing node group nodes; Data processing methods include: data analysis, and handling of missing values ​​and outliers; The data analysis of the historical memory related data includes: Batch parsing the historical memory-related data into a recognizable fixed format; wherein the recognizable fixed format includes: converting timestamps and times, separating data, filtering redundant information, and rearranging data; Processing missing values ​​and outliers on the historical memory-related data includes: Calculate the mean of a predetermined number of data around the abnormal data in the historical memory related data, and replace the abnormal data with the calculated mean; and fill in a predetermined number of data around the missing data in the historical memory related data.

3. The private cloud resource usage prediction method based on the combinatorial optimization algorithm according to claim 1 is characterized in that: The SVM-RFE algorithm is used to perform feature screening on the processed memory-related data to obtain the optimal feature data, including: Step a: Divide the processed memory-related data into training set data and test set data; Step b: Set the current feature subset and the optimal feature subset; wherein, in the initial state, the current feature subset contains all training set data, the optimal feature subset is empty, the initial accuracy w=0.0, the number of features deleted in each iteration is set to K, and the number of features retained is set to M; Step c: construct an SVM model based on the current feature subset, calculate the accuracy value w of each feature data in the current feature subset, and sort each feature data in the current feature subset in descending order according to the accuracy value w, and delete the K feature data with the lowest accuracy value in the current feature subset. Step d: Repeat step c until the current feature subset cannot be iterated further, that is, when the number of features in the current feature subset is M, return the optimal feature subset; The optimal feature data in the returned optimal feature subset are the M feature data remaining in the current feature subset after the iteration is completed.

4. The private cloud resource usage prediction method based on the combinatorial optimization algorithm according to claim 1 or 3, characterized in that: The prediction method further includes calculating an objective function of a resource usage prediction model and sorting the objective function of the resource usage prediction model to improve the accuracy of the resource usage prediction model, including: Calculate the objective function of the resource usage prediction model; where the objective function of the resource usage prediction model is the objective function of the base learner plus the objective function of the new learner; Set a penalty term to limit the number of leaf nodes in the decision tree of the XGBoost algorithm; Based on the objective function of the resource usage prediction model, a penalty term is added to obtain a total objective function including the penalty term, and the total objective function is sorted.

5. The private cloud resource usage prediction method based on the combined optimization algorithm according to claim 4, The objective function of the resource usage prediction model is: Where, Represents the prediction result of the i-th sample at the t-th iteration, whose value is the sum of the previous prediction result and the new learner function f t (x i )The sum of the prediction results; f t (x i ) represents the prediction result of the t-th base learner; k represents the k-th learner; i represents the number of samples, x i represents the i-th sample, and t represents the number of iterations, that is, the number of decision trees added; The penalty terms are: Where, Ω(f t ) represents the penalty term after adding the t-th base learner, γ represents the penalty intensity, λ represents the parameter; ω represents the weight of each leaf node in the decision tree; T represents the number of leaf nodes in the decision tree; The overall objective function is: Where, Obj (t) represents the overall objective function of the resource usage prediction model, represents the loss function of resource usage prediction model training, n represents the total number of samples, i represents the number of samples, y i represents the true value of the i-th sample, represents the prediction result of the i-th sample in the first t iterations, and C represents a constant.

6. The private cloud resource usage prediction method based on the combinatorial optimization algorithm according to claim 4 is characterized in that: The overall objective function is organized, including: The total objective function is approximated using a second-order Taylor expansion, and the Taylor expansion form of the total objective function is: Where f(x+Δx) represents the Taylor expansion of the total objective function, where x corresponds to Δx corresponds to f t (x i ), represents the first-order derivative of f(x), represents the second-order derivative of f(x); According to the total objective function formula and the Taylor expansion form of the total objective function, the following expression is defined: Where, f t (x i ) represents the prediction result of the t-th base learner, represents the loss function of resource usage prediction model training; g i represents the first-order derivative of the Taylor expansion, h i represents the second derivative of the Taylor expansion; Express request The first-order derivative of , t represents the number of iterations, that is, the number of decision trees added, y i represents the true value of the i-th sample, represents the prediction result of the i-th sample in the first t iterations, Express request The second derivative of Based on the total objective function formula, the Taylor expansion form of the total objective function and the defined expression (5), the final expression of the total objective function is: Where, Obj (t) represents the overall objective function of the resource usage prediction model, Ω(f t ) represents the penalty term after adding the t-th base learner.

7. A private cloud resource usage prediction system based on a combinatorial optimization algorithm is characterized by: Said include: Acquisition unit, feature screening unit, model building unit and prediction unit, The collecting unit is used to collect historical memory related data and perform data processing on the historical memory related data to obtain processed memory related data; The feature screening unit is used to perform feature screening on the processed memory-related data using the SVM-RFE algorithm to obtain optimal feature data; The model building unit is used to obtain an optimal parameter combination through a grid search method based on the optimal feature data; input the optimal parameter combination into a resource usage prediction model built based on the XGBoost algorithm, and train the resource usage prediction model using the processed memory-related data; The prediction unit is used to use the trained resource usage prediction model to predict the memory usage for a predetermined number of days in the future.

8. The private cloud resource usage prediction system based on the combinatorial optimization algorithm according to claim 7 is characterized in that: The prediction system further includes a calculation unit, The calculation unit is used to calculate the objective function of the resource usage prediction model and sort out the objective function of the resource usage prediction model to improve the accuracy of the resource usage prediction model; Among them, the computing unit is specifically used to: calculate the objective function of the resource usage prediction model; set a penalty term to limit the number of leaf nodes in the decision tree of the XGBoost algorithm; based on the objective function of the resource usage prediction model, add a penalty term to obtain the total objective function including the penalty term, and organize the total objective function.

9. An electronic device, characterized in that: include: Memory, processor; The processor is used to read and execute the computer program stored in the memory to implement the private cloud resource usage prediction system based on the combinatorial optimization algorithm described in any one of claims 1-6.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed, implement the private cloud resource usage prediction system based on the combinatorial optimization algorithm described in any one of claims 1-6.

Images