Cloud computing resource allocation method based on dynamic optimization, computer device and storage medium

Abstract

The invention discloses a cloud computing resource allocation method based on dynamic optimization, a computer device and a storage medium, and the method comprises the steps: collecting resource use data of each virtual machine in a cloud computing environment when an update period is reached; calculating the activeness of the resource nodes, and generating a demand expected value and a corrected expected value of each resource according to the activeness; according to a resource state evaluation result, dynamically adjusting resources by using a first resource allocation strategy, and actually allocating the resources; monitoring resource use data, and carrying out dynamic migration and resource adjustment on the virtual machine; and monitoring the access popularity of each data set, and dynamically adjusting the resource allocation strategy according to the second resource allocation strategy. According to the invention, through a multi-level resource monitoring and optimization mechanism, the resource management level in a cloud computing environment is effectively improved, and the system performance and the resource utilization rate are improved through accurate resource allocation, adaptive resource adjustment, dynamic data set management and an optimized migration strategy.

Description

[Technical field]

[0001] The present invention relates to the field of cloud computing technology, and in particular to a cloud computing resource allocation method based on dynamic optimization, a computer device and a storage medium. [Background technology]

[0002] With the widespread application of cloud computing technology, more and more enterprises and organizations are migrating their businesses to the cloud to improve business flexibility and scalability.

[0003] However, the resource allocation problem in cloud computing environment has gradually become prominent. The existing resource allocation methods are mainly divided into two types: static allocation and dynamic allocation. The static allocation method easily leads to resource waste, while the dynamic allocation method may bring large computing overhead due to frequent resource allocation and release.

[0004] To this end, there is an urgent need for an allocation method that can comprehensively consider multi-dimensional resource usage to optimize resource utilization, reduce computing overhead, and dynamically adjust resource configuration to adapt to changing workloads.

[0005] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. [Summary of the invention]

[0006] In view of this, an embodiment of the present invention provides a cloud computing resource allocation method based on dynamic optimization, a computer device and a storage medium.

[0007] In a first aspect, an embodiment of the present invention provides a cloud computing resource allocation method based on dynamic optimization, the method comprising:

[0008] S1. When the update cycle is reached, resource usage data of each virtual machine in the cloud computing environment is collected;

[0009] S2. Calculate the activity of resource nodes, and generate the expected value and revised expected value of each resource based on the activity;

[0010] S3. According to the resource status evaluation result, dynamically adjust the resources using the first resource allocation strategy and actually allocate the resources;

[0011] S4, monitoring resource usage data, and when identifying resource usage data changes that reach preset conditions, dynamically migrating virtual machines and adjusting resources;

[0012] S5. Monitor the access popularity of each data set, calculate the access popularity value of the data set, and dynamically adjust the resource allocation strategy according to the second resource allocation strategy.

[0013] According to the above aspects and any possible implementation, an implementation is further provided, wherein S1 specifically includes:

[0014] Deploy a monitoring tool on each virtual machine in the cloud computing environment, and use the monitoring tool to collect usage data of target resources at a preset collection frequency, wherein the target resources include but are not limited to CPU usage, memory usage, storage space occupancy, and network bandwidth usage;

[0015] The collected resource usage data is stored in a centralized data storage system.

[0016] According to the above aspects and any possible implementation, a further implementation is provided, wherein the activity A of the resource node s The calculation formula is as follows:

[0017] A s =w c *(f c -o c )+w r *(n r -o r )+w t *(n t -o t )*e -kt ,

[0018] Among them, A s Indicates the activity of the resource node, w c 、w r 、w t Respectively represent the weights of CPU usage, memory usage and network traffic, f c 、n r 、n t Indicates the actual CPU usage, memory usage, and network bandwidth usage of the virtual machine over a period of time. c , o r , o t They represent the corresponding offsets, e represents the natural constant, k represents the time decay rate, and t represents the interval between the last acquisition time and the current time;

[0019] The calculation formula of the expected demand value E is as follows:

[0020]

[0021] Among them, E represents the expected value of demand, α represents the adjustment coefficient of activity, and A si represents the activity of the i-th resource node, n represents the number of resource nodes involved in the calculation, and r i Indicates the resource usage of the i-th resource node, mi represents the weight of the i-th resource node associated with the resource usage, and M represents the normalized reference value;

[0022] The calculation formula of the modified expected value E′ is as follows:

[0023]

[0024] Among them, E′ represents the corrected expected value, E represents the expected value of demand, β and γ represent the adjustment coefficients, and Δ max Indicates the maximum fluctuation value of resource usage in a certain period of time in the past. Indicates the average utilization of resources.

[0025] According to the above aspects and any possible implementation, an implementation is further provided, wherein S3 specifically includes:

[0026] Apply a first resource allocation strategy to adjust resource allocation for all virtual machines, wherein the first resource allocation strategy:

[0027] The corrected expected value of each virtual machine is obtained, and the resource allocation priority of each virtual machine is calculated. The calculation formula of the priority is as follows:

[0028] P i =a*(E′ i +kU i )+b*A si +c*SLA i -d*C i ,

[0029]

[0030] Among them, P i represents the resource allocation priority of the ith virtual machine, a, b, c, d represent the adjustment coefficients, E′ represents the corrected expected value of the ith virtual machine, k represents the demand compensation factor, and U i A represents the unmet resource demand of the i-th virtual machine in the resource allocation during the last update cycle. si Indicates the activity of the i-th virtual machine, SLA i Represents the service level agreement, C i represents the migration cost, T i represents the migration time, R i Indicates the consumption rate of network bandwidth, D 1i Indicates the memory data volume of the virtual machine, D 2i Indicates the amount of storage data of the virtual machine, B i represents the actual available network bandwidth, P′ represents the impact factor on the virtual machine performance during the migration process, and O iIndicates disk I / O overhead;

[0031] Based on the priority calculation results, a priority sorting list is generated to sort the virtual machines according to their priorities;

[0032] Allocate resources in the order of the sorted list, from high to low;

[0033] Update the resource allocation status of each virtual machine and record the current resource allocation results.

[0034] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein S4 specifically includes:

[0035] Monitor virtual machine resource usage data, summarize resource usage data, and perform data cleaning and normalization;

[0036] Based on the historical resource usage data of each virtual machine, the ARIMA model in the time series analysis method is used to establish a resource usage baseline model, select and optimize the model parameters, and train and verify the ARIMA model so that its accuracy meets the preset requirements;

[0037] The monitored resource usage data is compared with the baseline model to detect whether there is an abnormal usage pattern. The detection criteria of the abnormal usage pattern meet any of the following criteria, namely:

[0038] Criteria 1: CPU usage exceeds the first threshold and memory usage exceeds the second threshold;

[0039] Standard 2: The resource usage fluctuation within the first preset time exceeds the third threshold;

[0040] Criterion 3: The difference between current usage data and historical baseline is below the fourth threshold;

[0041] Classify the detected abnormal usage patterns into resource overload according to standard one, short-term fluctuation according to standard two, and resource waste according to standard three;

[0042] Dynamically migrate virtual machines and adjust resources based on different classification results:

[0043] If the current resource usage rate is lower than the preset ratio of the baseline model and lasts longer than the preset time, the number of CPU cores, memory quota and network bandwidth quota of the virtual machine are reduced according to the preset ratio, and the resource allocation status of each virtual machine is updated;

[0044] For the detected high-load virtual machines, if the CPU usage exceeds the preset threshold, an additional preset number of CPU cores will be allocated to the virtual machine; if the memory usage exceeds the preset threshold, the memory quota of the preset virtual machine will be increased; if the network bandwidth usage exceeds the preset threshold, the network bandwidth limit of the preset virtual machine will be increased;

[0045] For virtual machines with short-term fluctuations, evaluate the migration priority of virtual machines and sort them. Select the virtual machines with high sorting as the priority migration targets. Monitor the duration of short-term fluctuations. If the fluctuation lasts for more than a certain period of time, calculate the migration cost. If the migration cost is higher than the preset cost threshold, do not migrate. If the migration cost is not higher than the preset cost threshold, determine whether the migration target is within the preset "cooling time". If so, replace the migration target according to the sorting. If not, migrate the target virtual machine to a computing node with lower load according to the sorting.

[0046] Update the resource allocation status of each virtual machine and record the current resource configuration.

[0047] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein evaluating the migration priority of the virtual machine and sorting the virtual machine specifically includes:

[0048] Collect basic data of each virtual machine, obtain the business priority, SLA level and historical migration information of each virtual machine;

[0049] Assigning a business importance score to each virtual machine, the score being based on the business priority and SLA level of the virtual machine, ranging from 1 to 10;

[0050] Assign a score to each virtual machine's resource usage, the score being calculated based on the current resource usage, with a score range of 1 to 10;

[0051] Assigning a score to the virtual machine's load fluctuation frequency, the score being calculated based on the frequency and magnitude of resource usage fluctuations over a period of time, with a score range of 1 to 10;

[0052] Assigning a score to the historical migration record of the virtual machine, wherein the score is calculated based on the historical migration record and the score ranges from 1 to 10;

[0053] The above scores are summed up to calculate the migration priority of each virtual machine. The final migration priority score ranges from 1 to 100.

[0054] Sort all virtual machines according to migration priority.

[0055] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein the calculation formula of the migration cost is as follows:

[0056] C total =C cpu +C memory +C network +C time +C disk ,

[0057] C cpu =W cpu *C l ,

[0058]

[0059]

[0060] C disk =W disk *D o ,

[0061] Among them, C total represents the total migration cost of the virtual machine, C cpu represents the CPU load cost, C memory represents the memory resource cost, C network represents the network bandwidth cost, C time represents the migration time cost, C disk represents disk I / O overhead, W cpu Indicates the weight of CPU load, C L Indicates the current CPU usage of the virtual machine, W memory represents the weight of memory migration, M s Indicates the memory size of the virtual machine, A b Indicates the currently available network bandwidth, W network Represents the weight of network bandwidth cost, N u Indicates the current network bandwidth usage of the virtual machine, N c Represents the total network capacity of the system, W time represents the weight of the time cost, M′ represents the migration factor related to the CPU load, and W disk represents the weight of disk I / O cost, D o Indicates the frequency of disk read and write operations of the virtual machine.

[0062] According to the above aspects and any possible implementation manner, an implementation manner is further provided, wherein S5 specifically includes:

[0063] Collect access data of each data set, wherein the access data includes the number of accesses, access frequency, data read volume, data written volume, and access source;

[0064] The collected access data is cleaned up, specifically: values ​​exceeding 3 times the standard deviation are treated as outliers, and missing values ​​are filled with the previous and next values, converting all access data to the range of 0 to 1;

[0065] The access heat value of each data set is calculated according to the access data of the data set. The calculation formula of the access heat value is as follows:

[0066] H date =w count *A count +w freq *A freq +w read *A read +w write *A write ,

[0067] Among them, H date Indicates the access popularity value of the dataset, w c 、w r 、w t They represent the weight coefficients of access times, access frequency, read volume, and write volume, respectively. count Indicates the number of visits to the data set, A freq Indicates the access frequency of the data set, A read Indicates the amount of data set read, A write Indicates the amount of data written to the dataset;

[0068] Dynamically adjust the resource allocation strategy according to the second resource allocation strategy, specifically:

[0069] The sliding window technology is used to calculate the heat change rate of the data set within the preset time, where the window size is 5 minutes and the step size is 1 minute;

[0070] If the heat value change does not reach the preset 5% change rate, it is recognized that the access demand is stable and the current resource allocation remains unchanged;

[0071] If the moving average of the heat value continues to rise by more than 10%, it is recognized as an increase in access demand, and the memory allocation of the data set is increased to 1.2 times the original allocation, the network bandwidth is increased to 1.3 times the original bandwidth, and the storage space is increased to 1.2 times the original storage space;

[0072] If the moving average value of the heat value continues to drop by more than 10%, it is recognized that the access demand is weakening, and the memory allocation is gradually reduced to 80% of the original allocation, the network bandwidth is gradually reduced to 70% of the original bandwidth, and the storage space is reduced to 80% of the original storage space;

[0073] Record the data before and after each resource adjustment.

[0074] In a second aspect, an embodiment of the present invention provides a computer device, including:

[0075] at least one processor; and a memory in communication with the processor;

[0076] The memory stores instructions executable by the processor, and the instructions are executed by at least one of the processors so that the processor is configured to execute the above-mentioned cloud computing resource allocation method based on dynamic optimization.

[0077] In a third aspect, an embodiment of the present invention provides a computer-readable storage medium, in which computer execution instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the above-mentioned cloud computing resource allocation method based on dynamic optimization.

[0078] One of the above technical solutions has the following beneficial effects:

[0079] In the method of the embodiment of the present invention, a cloud computing resource allocation method based on dynamic optimization is proposed. Through a multi-level resource monitoring and optimization mechanism, the resource management level in the cloud computing environment is effectively improved. Its precise resource allocation, adaptive resource adjustment, dynamic data set management and optimized migration strategy not only improve the system performance and resource utilization, but also reduce the system's operating costs, and enhance the system's reliability and scalability, which can better meet the complex and changeable cloud computing business needs.

Brief Description of the Drawings

[0080] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0081] Figure 1 It is a flow chart of a cloud computing resource allocation method based on dynamic optimization provided by an embodiment of the present invention. [Specific implementation method]

[0082] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] Please refer to Figure 1 , which is a flow chart of a cloud computing resource allocation method based on dynamic optimization provided by an embodiment of the present invention, such as Figure 1 As shown, the method comprises the following steps:

[0084] S1. When the update cycle is reached, resource usage data of each virtual machine in the cloud computing environment is collected;

[0085] S2. Calculate the activity of resource nodes, and generate the expected value and revised expected value of each resource based on the activity;

[0086] S3. According to the resource status evaluation result, dynamically adjust the resources using the first resource allocation strategy and actually allocate the resources;

[0087] S4, monitoring resource usage data, and when identifying resource usage data changes that reach preset conditions, dynamically migrating virtual machines and adjusting resources;

[0088] S5. Monitor the access popularity of each data set, calculate the access popularity value of the data set, and dynamically adjust the resource allocation strategy according to the second resource allocation strategy.

[0089] The present invention collects resource usage data of virtual machines to understand the resource usage of each virtual machine in real time. By using the calculated resource node activity and the generated demand expectation value and the corrected expectation value, the present invention can more accurately identify resource demand and dynamically adjust the resource allocation strategy. This real-time and accuracy ensure that resources can be efficiently allocated to the virtual machines that need them most, reducing resource waste and over-allocation. Under the continuous monitoring of resource usage data, when it is identified that the resource usage data change reaches the preset conditions, the virtual machine can be dynamically migrated and resources adjusted in time. The adaptive mechanism can respond to changes in resource demand more quickly, avoid performance bottlenecks caused by improper resource allocation, and enhance the ability to cope with complex scenarios and emergencies. By monitoring the access popularity of data sets, the access demand of each data set can be accurately understood, and the resource allocation strategy can be dynamically adjusted. When it is detected that the access popularity of a data set increases, the allocation of corresponding resources will be increased, and when the access popularity decreases, the allocation of corresponding resources will be increased. Gradually reduce the allocation of resources. Designing this dynamic optimization strategy for resource allocation not only improves the utilization efficiency of resources, but also avoids performance problems caused by excessive or insufficient resource allocation; by comprehensively evaluating the activity of resource nodes, the expected value of resource demand and the resource allocation priority of virtual machines, ensure that each decision of resource allocation and virtual machine migration is optimized. In particular, through monitoring and adjustment, unnecessary frequent migration and resource reallocation are avoided, which not only reduces resource waste, but also reduces performance loss during the migration process, thereby reducing the overall operating cost of the system; the dynamically optimized resource allocation strategy and real-time monitoring mechanism enable the system to respond quickly when resource usage changes. Through reasonable resource allocation and migration strategies, the system can maintain stable performance under high load and complex operating environments, reducing the risk of service interruption and performance degradation. The reliability of the present invention is particularly important when facing sudden resource demands in large-scale cloud computing scenarios.

[0090] Therefore, the embodiments of the present invention significantly improve the intelligence level of resource allocation in the cloud computing environment. It can not only dynamically adjust resource allocation and migration strategies according to actual resource usage, but also optimize resource utilization efficiency, reduce system operating costs, enhance system stability and reliability, and meet the complex and changing needs of cloud computing applications.

[0091] In a preferred embodiment of the present invention, the S1 specifically includes:

[0092] Deploy a monitoring tool on each virtual machine in the cloud computing environment, and use the monitoring tool to collect usage data of target resources at a preset collection frequency, wherein the target resources include but are not limited to CPU usage, memory usage, storage space occupancy, and network bandwidth usage;

[0093] The collected resource usage data is stored in a centralized data storage system.

[0094] In a preferred embodiment of the present invention, the activity level A of the resource node s The calculation formula is as follows:

[0095] A s =w c *(f c -o c )+w r *(n r -o r )+w t *(n t -o t )*e -kt ,

[0096] Among them, A s Indicates the activity of the resource node, w c 、w r 、w t Respectively represent the weights of CPU usage, memory usage and network traffic, f c 、n r 、n t Indicates the actual CPU usage, memory usage, and network bandwidth usage of the virtual machine over a period of time. c , o r , o t They represent the corresponding offsets, e represents the natural constant, k represents the time decay rate, and t represents the interval between the last acquisition time and the current time;

[0097] The calculation formula of the expected demand value E is as follows:

[0098]

[0099] Among them, E represents the expected value of demand, α represents the adjustment coefficient of activity, and A si represents the activity of the i-th resource node, n represents the number of resource nodes involved in the calculation, and r i Indicates the resource usage of the i-th resource node, m i represents the weight of the i-th resource node associated with the resource usage, and M represents the normalized reference value;

[0100] The calculation formula of the modified expected value E′ is as follows:

[0101]

[0102] Among them, E′ represents the corrected expected value, E represents the expected value of demand, β and γ represent the adjustment coefficients, and Δ max Indicates the maximum fluctuation value of resource usage in a certain period of time in the past. Indicates the average utilization of resources.

[0103] The embodiment of the present invention introduces three key indicators, namely CPU usage, memory usage and network traffic usage, and uses the time-weighted average method to calculate the activity of resource nodes. It can more accurately evaluate the real-time status of each resource node, taking into account the multi-dimensional indicators of resource usage, so that the system can fully and accurately understand the load of each resource node. The calculated activity is directly used to generate the expected value of demand for each resource, and the allocation of resources is further refined through correction. The demand forecast based on activity can better reflect the actual demand of each resource node, effectively avoid the problem of excess or insufficient resources, and ensure that the allocation of system resources can match the actual demand, thereby improving resource utilization efficiency. Based on accurate activity calculation, the resource allocation strategy is dynamically adjusted to optimize resource utilization efficiency, reduce system operation risks, and enhance the scalability of the system.

[0104] In a preferred embodiment of the present invention, S3 specifically includes:

[0105] Apply a first resource allocation strategy to adjust resource allocation for all virtual machines, wherein the first resource allocation strategy:

[0106] The corrected expected value of each virtual machine is obtained, and the resource allocation priority of each virtual machine is calculated. The calculation formula of the priority is as follows:

[0107] P i =a*(E′ i +kU i )+b*A si +c*SLA i -d*C i ,

[0108]

[0109] Among them, P i represents the resource allocation priority of the ith virtual machine, a, b, c, d represent the adjustment coefficients, E′ represents the corrected expected value of the ith virtual machine, k represents the demand compensation factor, and U i A represents the unmet resource demand of the i-th virtual machine in the resource allocation during the last update cycle. si Indicates the activity of the i-th virtual machine, SLA i Represents the service level agreement, C i represents the migration cost, T i represents the migration time, R i Indicates the consumption rate of network bandwidth, D 1i Indicates the memory data volume of the virtual machine, D 2iIndicates the amount of storage data of the virtual machine, B i represents the actual available network bandwidth, P′ represents the impact factor on the virtual machine performance during the migration process, and O i Indicates disk I / O overhead;

[0110] Based on the priority calculation results, a priority sorting list is generated to sort the virtual machines according to their priorities;

[0111] Allocate resources in the order of the sorted list, from high to low;

[0112] Update the resource allocation status of each virtual machine and record the current resource allocation results.

[0113] The embodiment of the present invention calculates the resource allocation priority of each virtual machine by comprehensively considering multiple key factors such as the corrected expected value, demand compensation factor, activity, service level agreement, migration cost, etc., thereby ensuring the accuracy of resource allocation decisions, so that resources can be allocated first to virtual machines with the most urgent needs and the highest priority, effectively improving resource utilization efficiency and avoiding unnecessary resource migration and adjustment. By adopting a priority sorting method, the system can quickly determine the order of resource allocation, ensuring that resources are allocated first to the virtual machines that need them most in the event of resource shortage or sudden demand, thereby improving the overall response speed of the system. By incorporating the service level agreement (SLA) into the calculation of resource allocation priority, the system can ensure that high-priority and critical business virtual machines obtain sufficient resource support, thereby improving the system's ability to ensure service quality.

[0114] In a preferred embodiment of the present invention, S4 specifically includes:

[0115] Monitor virtual machine resource usage data, summarize resource usage data, and perform data cleaning and normalization;

[0116] Based on the historical resource usage data of each virtual machine, the ARIMA model in the time series analysis method is used to establish a resource usage baseline model, select and optimize the model parameters, and train and verify the ARIMA model so that its accuracy meets the preset requirements;

[0117] The monitored resource usage data is compared with the baseline model to detect whether there is an abnormal usage pattern. The detection criteria of the abnormal usage pattern meet any of the following criteria, namely:

[0118] Criteria 1: CPU usage exceeds the first threshold and memory usage exceeds the second threshold;

[0119] Standard 2: The resource usage fluctuation within the first preset time exceeds the third threshold;

[0120] Criterion 3: The difference between current usage data and historical baseline is below the fourth threshold;

[0121] Classify the detected abnormal usage patterns into resource overload according to standard one, short-term fluctuation according to standard two, and resource waste according to standard three;

[0122] Dynamically migrate virtual machines and adjust resources based on different classification results:

[0123] If the current resource usage rate is lower than the preset ratio of the baseline model and lasts longer than the preset time, the number of CPU cores, memory quota and network bandwidth quota of the virtual machine are reduced according to the preset ratio, and the resource allocation status of each virtual machine is updated;

[0124] For the detected high-load virtual machines, if the CPU usage exceeds the preset threshold, an additional preset number of CPU cores will be allocated to the virtual machine; if the memory usage exceeds the preset threshold, the memory quota of the preset virtual machine will be increased; if the network bandwidth usage exceeds the preset threshold, the network bandwidth limit of the preset virtual machine will be increased;

[0125] For virtual machines with short-term fluctuations, evaluate the migration priority of virtual machines and sort them. Select the virtual machines with high sorting as the priority migration targets. Monitor the duration of short-term fluctuations. If the fluctuation lasts for more than a certain period of time, calculate the migration cost. If the migration cost is higher than the preset cost threshold, do not migrate. If the migration cost is not higher than the preset cost threshold, determine whether the migration target is within the preset "cooling time". If so, replace the migration target according to the sorting. If not, migrate the target virtual machine to a computing node with lower load according to the sorting.

[0126] Update the resource allocation status of each virtual machine and record the current resource configuration.

[0127] The embodiment of the present invention can eliminate noise and outliers in the data by real-time monitoring and summarizing the resource usage data of the virtual machine, and combine data cleaning and normalization operations to ensure the accuracy and consistency of the monitoring data. By establishing a resource usage baseline model based on a time series analysis method, the present invention can accurately capture the historical patterns and trends of virtual machine resource usage, and compare the current resource usage data with the baseline model, so as to timely discover abnormal situations in resource usage, thereby improving the sensitivity and detection accuracy of the system to abnormal patterns. The present invention classifies virtual machines according to the detected abnormal patterns and takes different resource adjustment measures. This classification adjustment strategy enables the system to deal with different types of abnormalities in a targeted manner, such as resource overload, short-term fluctuations and resource waste, so as to quickly and effectively respond to changes in resource demand. Through the detection and classification adjustment of resource usage anomalies, problems in resource configuration can be discovered and corrected in a timely manner, avoiding the situation of over-allocation or under-allocation of resources. For virtual machines with short-term fluctuations, delayed migration or resource adjustment can be adopted to avoid unnecessary resource waste, further improving the resource utilization rate of the system. The present invention can effectively prevent system performance degradation or service interruption caused by abnormal resource usage through timely detection and processing of abnormal resource usage patterns. Classification and adjustment measures for high-load virtual machines and resource waste ensure the stable operation of the system under high-load conditions, and maintain the overall balance and reliability of the system through resource recovery and reallocation.

[0128] Among them, based on the historical resource usage data of each virtual machine, the ARIMA model in the time series analysis method is used to establish a resource usage baseline model, select and optimize the model parameters, and train and verify the ARIMA model so that its accuracy meets the preset requirements. The specific method is as follows:

[0129] Collect historical CPU usage data of virtual machines from the monitoring system of the cloud computing environment. The data collection interval is once every minute and the collection period is the past 30 days.

[0130] The collected data is stored in a CSV file in the following format: Date and time: record the timestamp of each minute; CPU usage: record the CPU usage of each minute, expressed as a percentage; Memory usage: record the memory usage of each minute, in GB; Network bandwidth usage: record the network bandwidth usage of each minute, in Mbps; Storage space usage: record the storage space usage of each minute, in GB;

[0131] Clean the collected resource usage data and process outliers and missing values. Outliers are defined as points that exceed 3 times the standard deviation. Missing values ​​are filled by interpolation. Use Python's Pandas library to clean each data indicator.

[0132] Normalize the data and convert each resource usage data into a value between 0 and 1 for model analysis. This is implemented using Python code.

[0133] ARIMA model parameter selection and model establishment:

[0134] By analyzing the autocorrelation function (ACF) and partial autocorrelation function (PACF) graphs of various resource usage data, the parameters p, d, and q of the ARIMA model are determined; the ACF and PACF graphs of various resource usage data are generated using Python's statsmodels library;

[0135] By observing the graph, select p (AR term order) and q (MA term order) of each indicator; use the unit root test (ADF test) to determine the d value (number of differences) to ensure that the series is stable;

[0136] Use the selected parameters p, d, and q to train the ARIMA model for each resource indicator, and use Python's statsmodels library to train the ARIMA model for each resource usage data;

[0137] Use the model to predict resource usage data for a period of time in the future, and compare it with the actual data to verify the accuracy of the model; specifically, the data set can be divided into a training set and a test set (the data of the last 7 days is used for testing), use the training set to train the model, and the test set for verification; calculate the mean square error (MSE) and mean absolute error (MAE) between the predicted value and the actual value of each resource indicator as model verification indicators; if the MAE and MSE of the model are within an acceptable range (MAE is less than 0.05), the model accuracy is considered to meet the requirements, otherwise, adjust the p, d, q parameters and retrain the model;

[0138] Use the trained ARIMA model to predict various resource usage indicators for the next 7 days as the predicted values ​​of the baseline model; compare the actual monitoring data with the predicted values ​​of the baseline model. If the difference exceeds the set threshold, it is marked as an anomaly and classified as resource overload, short-term fluctuation, or resource waste. Set the threshold for anomaly detection: if the CPU usage exceeds the preset first threshold (such as 90%) and the memory usage exceeds the second threshold (such as 95%), it is marked as resource overload; within the first preset time (such as 5 minutes), any resource usage fluctuation exceeds the third threshold (such as 50%), it is marked as short-term fluctuation; the difference between the current usage data and the historical baseline is lower than the fourth threshold (such as 10%), it is marked as resource waste.

[0139] In a preferred embodiment of the present invention, evaluating the migration priorities of virtual machines and ranking them specifically includes:

[0140] Collect basic data of each virtual machine, obtain the business priority, SLA level and historical migration information of each virtual machine;

[0141] Assigning a business importance score to each virtual machine, the score being based on the business priority and SLA level of the virtual machine, ranging from 1 to 10;

[0142] Assign a score to each virtual machine's resource usage, the score being calculated based on the current resource usage, with a score range of 1 to 10;

[0143] Assigning a score to the virtual machine's load fluctuation frequency, the score being calculated based on the frequency and magnitude of resource usage fluctuations over a period of time, with a score range of 1 to 10;

[0144] Assigning a score to the historical migration record of the virtual machine, wherein the score is calculated based on the historical migration record and the score ranges from 1 to 10;

[0145] The above scores are summed up to calculate the migration priority of each virtual machine. The final migration priority score ranges from 1 to 100.

[0146] Sort all virtual machines according to migration priority.

[0147] This embodiment introduces a multi-dimensional scoring system to comprehensively evaluate and rank the migration priorities of virtual machines. This solution effectively improves the accuracy of resource allocation decisions, reduces unnecessary resource migrations, optimizes the response efficiency of the system and the flexibility of resource scheduling, and by reasonably ranking the migration priorities of virtual machines, the present invention can ensure the efficient use of system resources and the stable operation of key businesses.

[0148] In a preferred embodiment of the present invention, the calculation formula of the migration cost is as follows:

[0149] C total =C cpu +C memory +C network +C time +C disk ,

[0150] C cpu =W cpu *C l ,

[0151]

[0152] C disk =W disk *D o ,

[0153] Among them, C totalrepresents the total migration cost of the virtual machine, C cpu represents the CPU load cost, C memory represents the memory resource cost, C network represents the network bandwidth cost, C time represents the migration time cost, C disk represents disk I / O overhead, W cpu Indicates the weight of CPU load, C L Indicates the current CPU usage of the virtual machine, W memory represents the weight of memory migration, M s Indicates the memory size of the virtual machine, A b Indicates the currently available network bandwidth, W network Represents the weight of network bandwidth cost, N u Indicates the current network bandwidth usage of the virtual machine, N c Represents the total network capacity of the system, W time represents the weight of the time cost, M′ represents the migration factor related to the CPU load, and W disk represents the weight of disk I / O cost, D o Indicates the frequency of disk read and write operations of the virtual machine.

[0154] In a preferred embodiment of the present invention, S5 specifically includes:

[0155] Collect access data of each data set, wherein the access data includes the number of accesses, access frequency, data read volume, data written volume, and access source;

[0156] The collected access data is cleaned up, specifically: values ​​exceeding 3 times the standard deviation are treated as outliers, and missing values ​​are filled with the previous and next values, converting all access data to the range of 0 to 1;

[0157] The access heat value of each data set is calculated according to the access data of the data set. The calculation formula of the access heat value is as follows:

[0158] H date =w count *A count +w freq *A freq +w read *A read +w write *A write ,

[0159] Among them, H date Indicates the access popularity value of the dataset, w c 、w r 、w tThey represent the weight coefficients of access times, access frequency, read volume, and write volume, respectively. count Indicates the number of visits to the data set, A freq Indicates the access frequency of the data set, A read Indicates the amount of data set read, A write Indicates the amount of data written to the dataset;

[0160] Dynamically adjust the resource allocation strategy according to the second resource allocation strategy, specifically:

[0161] The sliding window technology is used to calculate the heat change rate of the data set within the preset time, where the window size is 5 minutes and the step size is 1 minute;

[0162] If the heat value change does not reach the preset 5% change rate, it is recognized that the access demand is stable and the current resource allocation remains unchanged;

[0163] If the moving average of the heat value continues to rise by more than 10%, it is recognized as an increase in access demand, and the memory allocation of the data set is increased to 1.2 times the original allocation, the network bandwidth is increased to 1.3 times the original bandwidth, and the storage space is increased to 1.2 times the original storage space;

[0164] If the moving average value of the heat value continues to drop by more than 10%, it is recognized that the access demand is weakening, and the memory allocation is gradually reduced to 80% of the original allocation, the network bandwidth is gradually reduced to 70% of the original bandwidth, and the storage space is reduced to 80% of the original storage space;

[0165] Record the data before and after each resource adjustment.

[0166] The embodiment of the present invention calculates the heat change rate through the sliding window technology, which can timely capture the dynamic changes of the access demand of the data set. Whether the access demand increases suddenly or gradually decreases, the system can quickly adjust the resource allocation strategy to ensure the system's sensitivity and response speed to changes, and improve the system's adaptability in complex environments. By dynamically adjusting resource allocation according to the access heat of the data set, the system can effectively guarantee the performance of high-access frequency and high-read volume data sets, avoid access delays or performance bottlenecks caused by insufficient resources, and at the same time, timely reduce the resource usage of low-heat data sets, which also helps the system release more resources for other key tasks without affecting the overall performance.

[0167] The embodiments of the present invention further provide device embodiments for implementing the steps and methods in the above method embodiments.

[0168] An embodiment of the present invention provides a computer device, including:

[0169] at least one processor; and a memory in communication with the processor;

[0170] The memory stores instructions executable by the processor, and the instructions are executed by at least one of the processors so that the processor is configured to execute the above-mentioned cloud computing resource allocation method based on dynamic optimization.

[0171] For the convenience of description, the above device is described as various units or modules according to their functions. Of course, when implementing the present invention, the functions of each unit or module can be implemented in the same or multiple software and / or hardware.

[0172] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0174] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0175] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1The steps for the functions specified in one or more boxes.

[0176] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0177] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0178] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0179] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0180] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0181] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0182] Each embodiment of the present invention is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0183] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A cloud computing resource allocation method based on dynamic optimization, It is characterized in that The method comprises: S1. When the update cycle is reached, resource usage data of each virtual machine in the cloud computing environment is collected; S2. Calculate the activity of resource nodes, and generate the expected value and revised expected value of each resource based on the activity; S3. According to the resource status evaluation result, dynamically adjust the resources using the first resource allocation strategy and actually allocate the resources; S4, monitoring resource usage data, and when identifying resource usage data changes that reach preset conditions, dynamically migrating virtual machines and adjusting resources; S5. Monitor the access popularity of each data set, calculate the access popularity value of the data set, and dynamically adjust the resource allocation strategy according to the second resource allocation strategy.

2. The method according to claim 1, It is characterized in that The S1 specifically includes: Deploy a monitoring tool on each virtual machine in the cloud computing environment, and use the monitoring tool to collect usage data of target resources at a preset collection frequency, wherein the target resources include but are not limited to CPU usage, memory usage, storage space occupancy, and network bandwidth usage; The collected resource usage data is stored in a centralized data storage system.

3. The method according to claim 2, It is characterized in that The activity of the resource node A s The calculation formula is as follows: A s =w c *(f c -o c )+w r *(n r -o r )+w t *(n t -o t )*have been -kt , Among them, A s Indicates the activity of the resource node, w c 、w r 、w t Respectively represent the weights of CPU usage, memory usage and network traffic, f c 、n r 、n t Indicates the actual CPU usage, memory usage, and network bandwidth usage of the virtual machine over a period of time. c , o r , o t They represent the corresponding offsets, e represents the natural constant, k represents the time decay rate, and t represents the interval between the last acquisition time and the current time; The calculation formula of the expected demand value E is as follows: Among them, E represents the expected value of demand, α represents the adjustment coefficient of activity, and A si represents the activity of the i-th resource node, n represents the number of resource nodes involved in the calculation, and r i Indicates the resource usage of the i-th resource node, m i represents the weight of the i-th resource node associated with the resource usage, and M represents the normalized reference value; The calculation formula of the modified expected value E′ is as follows: Among them, E′ represents the corrected expected value, E represents the expected value of demand, β and γ represent the adjustment coefficients, and Δ max Indicates the maximum fluctuation value of resource usage in a certain period of time in the past. Indicates the average utilization of resources.

4. The method according to claim 1, It is characterized in that The S3 specifically includes: Apply a first resource allocation strategy to adjust resource allocation for all virtual machines, wherein the first resource allocation strategy: The corrected expected value of each virtual machine is obtained, and the resource allocation priority of each virtual machine is calculated. The calculation formula of the priority is as follows: P i =a*(E i ′+kU i )+b*A si +c*SLA i -d*C i , Among them, P i represents the resource allocation priority of the ith virtual machine, a, b, c, d represent the adjustment coefficients, E′ represents the corrected expected value of the ith virtual machine, k represents the demand compensation factor, and U i A represents the unmet resource demand of the i-th virtual machine in the resource allocation during the last update cycle. si Indicates the activity of the i-th virtual machine, SLA i Represents the service level agreement, C i represents the migration cost, T i represents the migration time, R i Indicates the consumption rate of network bandwidth, D 1i Indicates the memory data volume of the virtual machine, D 2i Indicates the amount of storage data of the virtual machine, B i represents the actual available network bandwidth, P′ represents the impact factor on the virtual machine performance during the migration process, and O i Indicates disk I / O overhead; Based on the priority calculation results, a priority sorting list is generated to sort the virtual machines according to their priorities; Allocate resources in the order of the sorted list, from high to low; Update the resource allocation status of each virtual machine and record the current resource allocation results.

5. The method according to claim 4, It is characterized in that The S4 specifically includes: Monitor virtual machine resource usage data, summarize resource usage data, and perform data cleaning and normalization; Based on the historical resource usage data of each virtual machine, the ARIMA model in the time series analysis method is used to establish a resource usage baseline model, select and optimize the model parameters, and train and verify the ARIMA model so that its accuracy meets the preset requirements; The monitored resource usage data is compared with the baseline model to detect whether there is an abnormal usage pattern. The detection criteria of the abnormal usage pattern meet any of the following criteria, namely: Criteria 1: CPU usage exceeds the first threshold and memory usage exceeds the second threshold; Standard 2: The resource usage fluctuation within the first preset time exceeds the third threshold; Criterion 3: The difference between current usage data and historical baseline is below the fourth threshold; Classify the detected abnormal usage patterns into resource overload according to standard one, short-term fluctuation according to standard two, and resource waste according to standard three; Dynamically migrate virtual machines and adjust resources based on different classification results: If the current resource usage rate is lower than the preset ratio of the baseline model and lasts longer than the preset time, the number of CPU cores, memory quota and network bandwidth quota of the virtual machine are reduced according to the preset ratio, and the resource allocation status of each virtual machine is updated; For the detected high-load virtual machines, if the CPU usage exceeds the preset threshold, an additional preset number of CPU cores will be allocated to the virtual machine; if the memory usage exceeds the preset threshold, the memory quota of the preset virtual machine will be increased; if the network bandwidth usage exceeds the preset threshold, the network bandwidth limit of the preset virtual machine will be increased; For virtual machines with short-term fluctuations, evaluate the migration priority of virtual machines and sort them, take the virtual machines with high sorting as the priority migration targets, monitor the duration of short-term fluctuations, and calculate the migration cost if the fluctuation lasts for more than a certain period of time. If the migration cost is higher than the preset cost threshold, do not migrate. If the migration cost is not higher than the preset cost threshold, determine whether the migration target is within the preset "cooling time". If so, replace the migration target according to the sorting. If not, migrate the target virtual machine to a computing node with lower load according to the sorting. Update the resource allocation status of each virtual machine and record the current resource configuration.

6. The method according to claim 5, It is characterized in that The evaluating and sorting the migration priorities of the virtual machines specifically includes: Collect basic data of each virtual machine, obtain the business priority, SLA level and historical migration information of each virtual machine; Assigning a business importance score to each virtual machine, the score being based on the business priority and SLA level of the virtual machine, ranging from 1 to 10; Assign a score to each virtual machine's resource usage, the score being calculated based on the current resource usage, with a score range of 1 to 10; Assigning a score to the virtual machine's load fluctuation frequency, the score being calculated based on the frequency and magnitude of resource usage fluctuations over a period of time, with a score range of 1 to 10; Assigning a score to the historical migration record of the virtual machine, wherein the score is calculated based on the historical migration record and the score ranges from 1 to 10; The above scores are summed up to calculate the migration priority of each virtual machine. The final migration priority score ranges from 1 to 100. Sort all virtual machines according to migration priority.

7. The method according to claim 5, It is characterized in that The calculation formula of the migration cost is as follows: C total =C cpu +C memory +C network +C time +C disk , C cpu =W cpu *C l , C disk =W disk *D o , Among them, C total represents the total migration cost of the virtual machine, C cpu represents the CPU load cost, C memory represents the memory resource cost, C network represents the network bandwidth cost, C time represents the migration time cost, C disk Represents disk I / O overhead, W cpu Indicates the weight of CPU load, C L Indicates the current CPU usage of the virtual machine, W memory represents the weight of memory migration, M s Indicates the memory size of the virtual machine, A b Indicates the currently available network bandwidth, W network Represents the weight of network bandwidth cost, N u Indicates the current network bandwidth usage of the virtual machine, N c Represents the total network capacity of the system, W time represents the weight of the time cost, M′ represents the migration factor related to the CPU load, and W disk represents the weight of disk I / O cost, D o Indicates the frequency of disk read and write operations of the virtual machine.

8. The method according to claim 1, It is characterized in that The S5 specifically includes: Collect access data of each data set, wherein the access data includes the number of accesses, access frequency, data read volume, data written volume, and access source; The collected access data is cleaned up, specifically: values ​​exceeding 3 times the standard deviation are treated as outliers, and missing values ​​are filled with the previous and next values, converting all access data to the range of 0 to 1; The access heat value of each data set is calculated according to the access data of the data set. The calculation formula of the access heat value is as follows: A date =w count *A count +w freq *A freq +w read *A read +w write *A write , Among them, H date Indicates the access popularity value of the dataset, w c 、w r 、w t They represent the weight coefficients of access times, access frequency, read volume, and write volume, respectively. count Indicates the number of visits to the data set, A freq Indicates the access frequency of the data set, A read Indicates the amount of data set read, A write Indicates the amount of data written to the dataset; Dynamically adjust the resource allocation strategy according to the second resource allocation strategy, specifically: The sliding window technology is used to calculate the heat change rate of the data set within the preset time, where the window size is 5 minutes and the step size is 1 minute; If the heat value change does not reach the preset 5% change rate, it is recognized that the access demand is stable and the current resource allocation remains unchanged; If the moving average of the heat value continues to rise by more than 10%, it is recognized as an increase in access demand, and the memory allocation of the data set is increased to 1.2 times the original allocation, the network bandwidth is increased to 1.3 times the original bandwidth, and the storage space is increased to 1.2 times the original storage space; If the moving average value of the heat value continues to drop by more than 10%, it is recognized that the access demand is weakening, and the memory allocation is gradually reduced to 80% of the original allocation, the network bandwidth is gradually reduced to 70% of the original bandwidth, and the storage space is reduced to 80% of the original storage space; Record the data before and after each resource adjustment.

9. A computer device, It is characterized in that include: at least one processor; and a memory communicatively connected to the processor; The memory stores instructions executable by the processor, and the instructions are executed by at least one of the processors so that the processor is configured to execute the cloud computing resource allocation method based on dynamic optimization described in any one of claims 1 to 8.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the cloud computing resource allocation method based on dynamic optimization as described in any one of claims 1 to 8.

Images