Computing power resource scheduling method and system

Abstract

The invention relates to the technical field of data processing, in particular to a computing power resource scheduling method and system, and the method comprises the steps: collecting computing power demand data of a target node in multiple consecutive days, taking the data size of each data segment obtained through division as a data point, and constructing a historical demand sequence; aRIMA prediction is carried out based on the historical demand sequence, in the prediction process, the moving average coefficient of the corresponding reference data obtained through maximum likelihood estimation is weighted and normalized by using the weight of each reference data, the actual moving average coefficient of the corresponding reference data is obtained, and the moving average coefficient of the corresponding reference data is calculated based on the actual moving average coefficient of each reference data. And obtaining a moving average value of all reference data to carry out ARIMA prediction, and carrying out computing power scheduling based on a prediction result. According to the method, the data volume processed by the target node in the specific time period can be accurately predicted, so that accurate scheduling of computing power resources can be realized.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a computing resource scheduling method and system. Background Art

[0002] With the rapid development of technologies such as cloud computing, big data, and artificial intelligence, computing resources have become one of the core competitiveness of modern enterprises and scientific research institutions. However, the efficient scheduling and management of computing resources faces many challenges, mainly involving resource allocation under dynamic loads, which increases the difficulty of computing resource management. Therefore, it is urgent to find an accurate computing resource scheduling method to achieve efficient allocation and utilization of computing resources.

[0003] In related technologies, time series prediction algorithms are usually used to predict the computing resource requirements of each node, and dynamically allocate computing resources based on the computing resource demand prediction results. The ARIMA model consists of three parts: AutoRegressive (AR), Integrated (I), and Moving Average (MA). By modeling the historical data of the time series, future trends can be predicted, and the computing resource requirements of the node can be predicted.

[0004] However, in the process of using the ARIMA model to predict the computing power demand of the node, the MA model in the model captures the dependency between error terms, and the moving average coefficient of each error term is usually calculated directly based on the maximum likelihood function. However, since the maximum likelihood estimation is based on the probability distribution, it is impossible to perform targeted processing on the abnormal data in the error term, resulting in inaccurate calculation results of the traditional MA model, affecting the accuracy of the prediction effect and making it impossible to accurately schedule computing power resources. Summary of the invention

[0005] In order to solve the problem that when predicting the computing power demand of a node based on the ARIMA model, the accuracy of the prediction result is poor, resulting in the inability to accurately schedule the computing power, the present invention provides a computing power resource scheduling method and system.

[0006] According to a first aspect of the present invention, a computing resource scheduling method is provided, comprising: Collect the computing power demand data of the target node for multiple consecutive days and divide it into several data segments of the same length. Take the data volume of each data segment as a data point to build a historical demand sequence. ARIMA forecasting is performed based on the historical demand sequence. During the forecasting process, the previous several error values ​​of the current error value are used as reference data. The moving average coefficient of the corresponding reference data obtained by maximum likelihood estimation is weighted and normalized using the weight of each reference data to obtain the actual moving average coefficient of the corresponding reference data. Based on the actual moving average coefficient of each reference data, the moving average of all reference data is obtained for ARIMA forecasting, and computing power scheduling is performed based on the forecast results. The method for obtaining the weight includes: calculating the credibility of any reference data, wherein the credibility is negatively correlated with the difference between the any reference data and the average value of all reference data, and the difference between the any reference data and the average error value of all data points in the historical demand sequence; All reference data are fitted, and based on the fitting error of any reference data and the fitting errors of all reference data, the trend reference of any reference data is obtained to correct the credibility. The correction value is positively correlated with the trend reference, and the weight of any reference data is obtained.

[0007] When the present invention uses the ARIMA model to predict the computing power demand of the node, it can focus on the interference of abnormal data in the reference data of the current error value, and can determine the weight by evaluating the possibility that each reference data is abnormal data, and weight the moving average coefficient of the corresponding reference data obtained by maximum likelihood estimation, so that the determined moving average can reduce the impact of the abnormal data in the reference data, thereby improving the accuracy of the computing power demand prediction results and realizing the precise scheduling of computing power resources.

[0008] Preferably, the number of reference data of the current error value is the same as the moving average order of the pre-built ARIMA model.

[0009] Preferably, the credibility of any reference data satisfies the following relationship: ; In the formula, is the current error value The credibility of the reference data; is the current error value The value of reference data; is the average value of the error values ​​of all data points in the historical demand sequence; is the average value of all reference data; is the absolute value symbol; is the standard deviation of the error values ​​of all data points in the historical demand sequence; is the standard deviation of all reference data for the current error value.

[0010] The present invention can accurately measure the possibility of each reference data being abnormal data by comparing and analyzing the values ​​of each reference data with all error data, and using the standard deviation as a reference standard can eliminate the influence of differences in different dimensions, thereby ensuring the accuracy of the determined credibility.

[0011] Preferably, the method for obtaining the error value of any data point in the historical demand sequence includes: For any data point in the historical demand sequence, the difference between the value of the data point and the predicted value corresponding to the data point is calculated to obtain the error value of the data point.

[0012] Preferably, based on the fitting error of any reference data and the fitting errors of all reference data, the trend reference of any reference data is obtained, satisfying the following relationship: ; In the formula, is the current error value The trend reference of the reference data; is the current error value The value of reference data; is the current error value The fitted values ​​of reference data; is the absolute value symbol; The number of reference data for the current error value; is a natural exponential function.

[0013] The present invention can evaluate the possibility of each reference data being abnormal data from the dimension of change trend, so as to adjust the credibility of each reference data, thereby being able to more accurately evaluate the degree of abnormality of each reference data.

[0014] Preferably, the process of correcting the credibility of any reference data by using the trend reference of the reference data includes: The trend reference of any reference data and the credibility of the reference data are multiplied to obtain a correction value of the credibility of the reference data, and the correction value is used as the weight of the reference data.

[0015] The present invention can set a larger weight for reference data with a lower degree of abnormality, thereby reducing the impact of abnormal data existing in the reference data.

[0016] Preferably, a method for obtaining the actual moving average coefficient of any reference data includes: Obtain the weighted value of the moving average coefficient of each reference data and sum them up, calculate the weighted value of the moving average coefficient of any reference data, and the ratio of the weighted value to the obtained cumulative sum to obtain the normalized value of the weighted value, and use the normalized value as the actual moving average coefficient of any reference data.

[0017] Preferably, the process of obtaining the moving average value of all reference data based on the actual moving average coefficient of each reference data includes: The actual moving average coefficient of each reference data is used to perform weighted summation on all reference data to obtain the moving average of all reference data.

[0018] The present invention can ensure the accuracy of the determined moving average value.

[0019] Preferably, the target node is a computer device.

[0020] According to a second aspect of the present invention, a computing resource scheduling system is provided. The system includes a memory and a processor. A computer program is stored in the memory. The processor executes the computer program to implement the steps of the first aspect of the present invention.

[0021] The present invention has the following effects: The method of determining the actual moving average coefficient of each reference data in the present invention can reduce the impact of abnormal data existing in the reference data on the moving average of all reference data, thereby ensuring the accuracy of the moving average. Therefore, based on the moving average with higher accuracy, the computing power demand of the target node can be accurately predicted, thereby realizing accurate scheduling of computing power resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 It is a schematic diagram of the steps of a computing resource scheduling method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0025] Reference Figure 1 , a computing resource scheduling method, including steps S1 to S3, specifically as follows: S1: Collect the computing power demand data of the target node for several consecutive days and divide it into several data segments of the same length. Take the data volume of each data segment as a data point to build a historical demand sequence.

[0026] In an exemplary embodiment of the present invention, the target node refers to a randomly selected node, and the node is a computing unit or device in the network, responsible for executing computing tasks, storing data or participating in network communications. The node can be a physical device (such as a server, a computer) or a virtual instance (such as a virtual machine). This embodiment does not specifically limit the type of the node.

[0027] Computing power demand data refers to the quantitative indicator of the computing power actually required by a node to complete its computing tasks (such as data processing, algorithm execution, model training, transaction verification, etc.) within a specific time period. It reflects the degree of demand for computing resources (such as CPU, GPU, memory, etc.) of the node during operation.

[0028] In an exemplary embodiment of the present invention, the target node is a computer device.

[0029] Furthermore, after the target node is selected, the computing power demand data of the target node for multiple consecutive days, such as 7 consecutive days, can be obtained, and the computing power demand data every ten minutes can be used as a data segment, thereby dividing the computing power demand data within the 7 days into several data segments; then, the data volume of each data segment can be used as a data point, thereby obtaining a historical demand sequence composed of all data points. This embodiment does not specifically limit the length of the data segment.

[0030] S2: Perform ARIMA forecasting based on historical demand sequences. During the forecasting process, the previous several error values ​​of the current error value are used as reference data. The moving average coefficient of the corresponding reference data obtained by maximum likelihood estimation is weighted and normalized using the weight of each reference data to obtain the actual moving average coefficient of the corresponding reference data.

[0031] It should be noted that the present invention improves the traditional MA model to address the problem of inaccurate calculation results obtained by the traditional MA model. The specific improvements are: by evaluating several error values ​​before the current error value, that is, the performance of the reference data, the weights of the corresponding reference data are determined, and the moving average coefficients of the corresponding reference data obtained by maximum likelihood estimation are corrected using the weights, thereby obtaining the actual moving average coefficients of each reference data, and subsequent operations are performed based on the actual moving average coefficients of each reference data to improve the accuracy of the calculation results of the traditional MA model.

[0032] It should be further explained that the data expression of the MA model is: ,in, is the current observation value; is the current error value; is the moving average coefficient; is the order of the MA model. The present invention only improves the determination of the moving average coefficient in the traditional MA model, that is, only improves , ..., The way the value is determined does not improve the MA model and other content of trend forecasting using the ARIMA model.

[0033] In an exemplary embodiment of the present invention, the number of reference data of the current error value is the same as the moving average order of the pre-built ARIMA model.

[0034] Among them, the moving average order of the ARIMA model refers to the order of the MA model. The order of the MA model can be determined by the ACF diagram (autocorrelation diagram) or the PACF diagram (partial correlation diagram). This embodiment does not specifically limit the method for determining the order of the MA model. It should be noted that the process of determining the order of the MA model using the ACF diagram and the PACF diagram is a prior art and is not described in detail in this embodiment.

[0035] Furthermore, after the reference data of the current error value is determined, the weight of each reference data can be determined by the following steps: Step 1: Calculate the credibility of any reference data. The credibility is negatively correlated with the difference between the reference data and the average value of all reference data, and the difference between the reference data and the average error value of all data points in the historical demand sequence. The credibility refers to the possibility that any reference data is normal data. If any reference data is likely to be abnormal data, the credibility of the reference data is relatively low; otherwise, the credibility of the reference data is relatively high.

[0036] In an exemplary embodiment of the present invention, the error value of any data point in the historical demand sequence can be determined by the following steps: For any data point in the historical demand sequence, the difference between the value of the data point and the predicted value corresponding to the data point is calculated to obtain the error value of the data point.

[0037] It should be noted that since each data point in the historical demand sequence is the data volume of a data segment, the predicted value corresponding to each data point is the data volume of the corresponding data segment predicted by the ARIMA model, so that the error value of the corresponding data point can be determined based on the actual value and predicted value of each data point.

[0038] Furthermore, after determining the error value of each data point in the historical demand sequence, the credibility of each reference data can be calculated. Specifically, the credibility of any reference data satisfies the following relationship: ; In the formula, is the current error value The credibility of the reference data; is the current error value The value of reference data; is the average value of the error values ​​of all data points in the historical demand sequence; is the average value of all reference data; is the absolute value symbol; It is the standard deviation of the error values ​​of all data points in the historical demand sequence, which is used as a comparison standard to eliminate the impact of differences in different dimensions; The standard deviation of all reference data for the current error value is used as a comparison standard to eliminate the impact of differences in different dimensions.

[0039] in, Reflects the The smaller the value is, the greater the deviation between the reference data and all the error values, which means that the value of the reference data is more likely to be abnormal, and the corresponding reference data is less reliable.

[0040] Reflects the The smaller the value is, the greater the deviation between the reference data and all the reference data is, which means that the reference data has local prominent features. The more likely it is that the value of the reference data is abnormal, and the lower the credibility of the corresponding reference data is.

[0041] Step 2: Fit all reference data, and obtain the trend reference of any reference data based on the fitting error of any reference data and the fitting errors of all reference data; It should be noted that, since the ARIMA model can analyze and predict the changing trend of the time series, so that the data error gradually becomes smaller or remains unchanged, the present invention uses this feature to evaluate the changing trend of each reference data, and uses the evaluation results to correct the credibility of each reference data to further ensure the accuracy of the determined credibility.

[0042] Optionally, all reference data may be fitted using a least squares method, polynomial fitting, or the like. This embodiment does not specifically limit the selected fitting method.

[0043] Furthermore, after fitting all reference data, the trend reference of any reference data can be determined based on the fitting error of any reference data and the fitting errors of all reference data. Specifically, the trend reference of any reference data satisfies the following relationship: ; In the formula, is the current error value The trend reference of the reference data; is the current error value The value of reference data; is the current error value The fitted values ​​of reference data; is the absolute value symbol; The number of reference data for the current error value; is a natural exponential function.

[0044] in, Reflects the current error value The larger the value is, the worse the fitting effect of the reference data is, which further indicates that the change trend of the reference data may be abnormal, and the corresponding trend of the reference data is less referenceable.

[0045] It reflects the cumulative sum of the fitting errors of all reference data. The larger the value, the worse the function fitting effect is, and the subsequent error value cannot be accurately predicted. At this time, if the current error value is When the fitting error of a reference data is also large, it means that the changing trend of the reference data is relatively consistent with the changing trend of all reference data, which further indicates that the changing trend of the reference data is a normal changing trend, and the corresponding trend of the reference data is more referenceable.

[0046] Step 3: Use trend reference to correct the credibility. The correction value is positively correlated with the trend reference to obtain the weight of any reference data.

[0047] In an exemplary embodiment of the present invention, the credibility of any reference data may be corrected by the following steps: The trend reference of any reference data and the credibility of the reference data are multiplied to obtain a correction value of the credibility of the reference data, and the correction value is used as the weight of the reference data.

[0048] In another embodiment, linear correction or logarithmic correction can be selected according to specific circumstances. This embodiment does not specifically limit the selected correction credibility, as long as the correction value is positively correlated with the trend reference of the corresponding reference data.

[0049] Further, after the weight of each reference data is determined, the weight of each reference data can be used to weight the moving average coefficient of the corresponding reference data obtained by maximum likelihood estimation, thereby obtaining a weighted value of the moving average coefficient of each reference data. It should be noted that the process of determining the moving average coefficient of each error value by maximum likelihood estimation is a prior art and is not described in detail in this embodiment.

[0050] In an exemplary embodiment of the present invention, the determination of the actual moving average coefficient of each reference data can be achieved by the following steps: Obtain the weighted value of the moving average coefficient of each reference data and sum them up, calculate the weighted value of the moving average coefficient of any reference data, and the ratio of the weighted value to the obtained cumulative sum to obtain the normalized value of the weighted value, and use the normalized value as the actual moving average coefficient of any reference data.

[0051] In another implementation, a linear normalization function may also be used, such as The weighted values ​​of the moving average coefficients of the reference data are normalized by a function, so as to obtain the actual moving average coefficients of the reference data. This embodiment does not specifically limit the selected normalization method.

[0052] S3: Based on the actual moving average coefficient of each reference data, the moving average of the error is obtained for ARIMA prediction, and computing power is scheduled based on the prediction results.

[0053] In an exemplary embodiment of the present invention, the determination of the moving average of all reference data can be achieved by the following steps: The actual moving average coefficient of each reference data is used to perform weighted summation on all reference data to obtain a moving average.

[0054] It should be noted that obtaining the moving average value of the error based on the moving average coefficient of each error term is a prior art in the MA model, which will not be described in detail in this embodiment.

[0055] Furthermore, after obtaining the moving average of all reference data, ARIMA prediction can be performed based on the moving average to obtain the predicted value of the amount of data processed by the target node in the next data segment, so that real-time computing power resource scheduling can be performed based on the prediction results to maximize the utilization rate of computing power resources. It should be noted that the process of performing ARIMA prediction based on a known moving average is a prior art and is not described in detail in this embodiment.

[0056] The present invention also provides a computing power resource scheduling system, the system includes a memory and a processor, and a computer program is stored in the memory, the computer program integrates the function of a computing power resource scheduling method, when the computer program is executed, a computing power resource scheduling method can be used to accurately predict the amount of data processed by the target node within a specific time period, thereby realizing accurate scheduling of computing power resources.

[0057] In the description of this specification, "plurality" or "several" means at least two, such as two, three or more, etc., unless otherwise clearly and specifically defined.

[0058] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

Claims

1. A computing resource scheduling method, characterized in that: include: Collect the computing power demand data of the target node for multiple consecutive days and divide it into several data segments of the same length. Take the data volume of each data segment as a data point to build a historical demand sequence. ARIMA forecasting is performed based on the historical demand sequence. During the forecasting process, the previous several error values ​​of the current error value are used as reference data. The moving average coefficient of the corresponding reference data obtained by maximum likelihood estimation is weighted and normalized using the weight of each reference data to obtain the actual moving average coefficient of the corresponding reference data. Based on the actual moving average coefficient of each reference data, the moving average of all reference data is obtained for ARIMA forecasting, and computing power scheduling is performed based on the forecast results. The method for obtaining the weight includes: calculating the credibility of any reference data, wherein the credibility is negatively correlated with the difference between the any reference data and the average value of all reference data, and the difference between the any reference data and the average error value of all data points in the historical demand sequence; All reference data are fitted, and based on the fitting error of any reference data and the fitting errors of all reference data, the trend reference of any reference data is obtained to correct the credibility, and the correction value is positively correlated with the trend reference, so as to obtain the weight of any reference data.

2. A computing resource scheduling method according to claim 1, characterized in that: The number of reference data of the current error value is the same as the moving average order of the pre-built ARIMA model.

3. A computing resource scheduling method according to claim 1, characterized in that: The credibility of any reference data satisfies the following relationship: ； In the formula, is the current error value The credibility of the reference data; is the current error value The value of reference data; is the average value of the error values ​​of all data points in the historical demand sequence; is the average value of all reference data; is the absolute value symbol; is the standard deviation of the error values ​​of all data points in the historical demand sequence; is the standard deviation of all reference data for the current error value.

4. A computing resource scheduling method according to claim 3, characterized in that: The method for obtaining the error value of any data point in the historical demand sequence includes: For any data point in the historical demand sequence, the difference between the value of the data point and the predicted value corresponding to the data point is calculated to obtain the error value of the data point.

5. A computing resource scheduling method according to claim 1, characterized in that: The trend reference of any reference data is obtained based on the fitting error of any reference data and the fitting errors of all reference data, and satisfies the following relationship: ； In the formula, is the current error value The trend reference of the reference data; is the current error value The value of reference data; is the current error value The fitted values ​​of reference data; is the absolute value symbol; The number of reference data for the current error value; is a natural exponential function.

6. A computing resource scheduling method according to claim 5, characterized in that: The process of correcting the credibility of any reference data by using its trend reference includes: The trend reference of any reference data and the credibility of the reference data are multiplied to obtain a correction value of the credibility of the reference data, and the correction value is used as the weight of the reference data.

7. A computing resource scheduling method according to claim 1, characterized in that: A method for obtaining the actual moving average coefficient of any reference data includes: Obtain the weighted values ​​of the moving average coefficients of each reference data and sum them up, calculate the weighted value of the moving average coefficient of any reference data, and the ratio of the weighted value to the obtained cumulative sum to obtain the normalized value of the weighted value, and use the normalized value as the actual moving average coefficient of any reference data.

8. A computing resource scheduling method according to claim 1, characterized in that: The process of obtaining the moving average of all reference data based on the actual moving average coefficient of each reference data includes: The actual moving average coefficient of each reference data is used to perform weighted summation on all reference data to obtain the moving average of all reference data.

9. A computing resource scheduling method according to claim 1, characterized in that: The target node is a computer device.

10. A computing resource scheduling system, characterized in that: The computing power resource scheduling system includes a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps of the computing power resource scheduling method as described in any one of claims 1-9.

Images