Storage space cruising ability evaluation method and system based on database capacity prediction

Abstract

The invention provides a storage space cruising ability evaluation method and system based on database capacity prediction, and the method comprises the steps: data collection, data management, data calculation, cruising ability evaluation and user interaction, during data calculation, a difference sequence is obtained through calculation according to historical database capacity observation value data, and the cruising ability of a storage space is evaluated based on the difference sequence. And executing an adaptive weighted multi-period smoothing algorithm, and completing an iterative calculation process according to the algorithm. A self-adaptive weighted multi-period smoothing method is adopted, and the window size is dynamically adjusted to better adapt to the time sequence change of the capacity, so that the data processing is more flexible and effective; by introducing a weight attenuation factor, a higher weight is distributed for a recent observation value, and meanwhile, the influence of relatively far historical data is gradually reduced, so that the latest trend of capacity change can be tracked in time.

Description

Technical Field

[0001] The invention relates to a storage space endurance evaluation method and system based on database capacity prediction, belonging to the field of database operation and maintenance. Background Art

[0002] With the rapid development of Internet technology, especially the large-scale application of self-media, all industries have accumulated a huge amount of data in a short period of time. Although great progress has been made in storage methods in recent years, databases are still the core system for storing, managing and retrieving data. Therefore, under the premise of the ever-increasing scale of data, it is crucial to maintain a healthy and efficient database system.

[0003] There are many factors that affect the database. Database administrators have to pay attention to the key performance indicators of the database at all times, timely discover and troubleshoot hidden dangers, and try their best to avoid database anomalies. Among the many data performance indicators, monitoring and predicting database capacity is very necessary to ensure the stable operation of the system. Existing database performance monitoring systems often only focus on the current storage status, and there are relatively few predictions and assessments of future storage needs. This results in system database administrators often only being able to take emergency capacity expansion, data migration and other measures when facing insufficient storage space, and being unable to plan and optimize resource allocation in advance, which is very passive. Especially in scenarios such as cloud computing, big data and the Internet of Things, if sufficient storage space cannot be predicted and prepared in advance, system performance may decline, and even data loss and service interruption may occur in severe cases. Therefore, developing an accurate and effective database capacity prediction algorithm is of great significance to improving the storage space endurance of the system.

[0004] At present, the research on database capacity prediction has made some progress, but there are still some shortcomings. Although some advanced prediction models have high prediction accuracy, they have high computational complexity and are difficult to deploy and apply on a large scale in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1 This is a system structure diagram of the present invention.

[0006] Figure 2 It is a flow chart of the present invention. Summary of the invention

[0007] In order to solve the problems existing in the prior art, the present invention proposes a storage space endurance evaluation method based on database capacity prediction, and the specific steps of the method include:

[0008] (1) Data collection: Collect database capacity data from the database or other data sources and store it in the system's historical database capacity observation database;

[0009] (2) Data management: This module stores the collected database capacity data in the system database and can delete and back up the data;

[0010] (3) Data calculation: Read the historical database capacity observation data and calculate the difference sequence. Based on the difference sequence, execute the adaptive weighted multi-period smoothing algorithm and complete the iterative calculation process according to the algorithm;

[0011] (4) Endurance evaluation: Based on the smoothed value prediction results output by the data calculation module and the current remaining storage space, the endurance of the remaining storage space is calculated;

[0012] (5) User interaction: Provide a user interface to display the endurance evaluation results and allow users to adjust algorithm parameters.

[0013] Furthermore, step (3) specifically includes:

[0014] (1) Weight calculation: Calculate the weight w of the data at the i-th time point i =α·e -b(t-i) , where n is the current window size. In this window, there are n data from the current data to the previous data, and the weights corresponding to these data are the weight sequence w t ,w t-1 ,...,w t-n+1 , where i = t, t-1, …, t-n+1, ​​b is the decay rate; α is the weight decay factor, 0<α<1;

[0015] (2) Smoothing value calculation: When a new observation value is reached, the smoothing value is calculated S i is the smoothed value at the i-th time point, D t is the element in the difference sequence D of the database capacity time series C, D t =C t -C t-1， C t is the element in the database capacity time series C, C t is the database capacity at the tth time point;

[0016] (3) Prediction and mean square error calculation: Use the current smoothing value S t+1 As the predicted value, calculate S t With D t The mean square error between where k = t, t-1, ..., t-n+1, ​​D k is the change in database capacity from the k-1th time point to the kth time point, S kis the smoothed value at the kth time point.

[0017] (4) Dynamically adjust the window: If MSE t >MSE upper , the window size after enlargement is min(n+step,n max ), where n max Indicates the maximum value of the window; if MSE t <MSE lower , the window size is reduced to max(n-step,n min ), where n min Indicates the minimum value of the window, step is the adjustment step of the initial window size, MSE upper is the upper threshold of mean square error, MSE lower is the lower threshold of mean square error;

[0018] (5) Iterative update: When new observations arrive, repeat steps (1) to (4) to optimize the prediction performance by adjusting the window size.

[0019] (6) Output result: When all data are processed, the latest predicted value ΔD of the differential sequence D is obtained, and the endurance time of the remaining storage space is calculated = remaining storage space × time unit / ΔD.

[0020] Based on the above method, the present invention further proposes a storage space endurance evaluation system based on database capacity prediction, which includes the following modules:

[0021] (1) Data collection module: collects database capacity data from the database or other data sources and stores it in the system's historical database capacity observation database.

[0022] (2) Data management module: This module stores the collected database capacity data in the system database and can delete and back up the data.

[0023] (3) Data calculation module: read the historical database capacity observation data and calculate the difference sequence. Based on the difference sequence, execute the adaptive weighted multi-period smoothing algorithm and complete the iterative calculation process according to the algorithm.

[0024] (4) Battery life evaluation module: Based on the smoothed value prediction result output by the data calculation module and the current remaining storage space, the battery life of the remaining storage space is calculated.

[0025] (5) User interaction module: provides a user interface, displays the endurance evaluation results, and allows users to adjust algorithm parameters.

[0026] Furthermore, the data calculation module includes the following steps:

[0027] 3.1 Weight calculation: Calculate the weight w of the data at the i-th time point i =α·e -b(t-i) , where n is the current window size. In this window, there are n data from the current data to the previous data, and the weights corresponding to these data are the weight sequence w t ,w t-1 ,...,w t-n+1 , where i = t, t-1, …, t-n+1, ​​b is the decay rate; α is the weight decay factor, 0<α<1;

[0028] 3.2 Smoothing value calculation: When a new observation value arrives, calculate the smoothing value S i is the smoothed value at the i-th time point, D t is the element in the difference sequence D of the database capacity time series C, D t =C t -C t-1， C t is the element in the database capacity time series C, C t is the database capacity at the tth time point;

[0029] 3.3 Prediction and mean square error calculation: Using the current smoothing value S t+1 As the predicted value, calculate S t With D t The mean square error between where k = t, t-1, ..., t-n+1, ​​D k is the change in database capacity from the k-1th time point to the kth time point, S k is the smoothed value at the kth time point.

[0030] 3.4 Dynamically adjust the window: If MSE t >MSE upper , the window size after enlargement is min(n+step,n max ), where n max Indicates the maximum value of the window; if MSE t <MSE lower , the window size is reduced to max(n-step,n min ), where n min Indicates the minimum value of the window, step is the adjustment step of the initial window size, MSE upper is the upper threshold of mean square error, MSE lower is the lower threshold of mean square error;

[0031] 3.5 Iterative update: When subsequent new observations arrive, repeat steps 3.1 to 3.4 and optimize the prediction performance by adjusting the window size.

[0032] Furthermore, in the battery life evaluation module, after all data are processed, the latest predicted value ΔD of the differential sequence D is obtained, and the battery life of the remaining storage space is calculated as follows: the remaining storage space×time unit / ΔD.

[0033] The present invention adopts an adaptive weighted multi-period smoothing method, which has the following advantages: (1) by dynamically adjusting the window size, it can better adapt to the temporal changes in capacity, making data processing more flexible and effective; (2) by introducing a weight decay factor, a higher weight is assigned to recent observations, while gradually reducing the impact of more distant historical data, which helps to track the latest trends in capacity changes in a timely manner. DETAILED DESCRIPTION

[0034] The present invention designs an adaptive weighted multi-period smoothing algorithm, and accordingly designs a storage space endurance evaluation system based on database capacity prediction ( Figure 1 ), the flowchart of the adaptive weighted multi-period smoothing algorithm is as follows Figure 2 As shown by Figure 1 It can be known that the specific process of the present invention is:

[0035] (1) Initialization

[0036] 1) The initial window size n, which represents the number of past observations considered at the beginning of the algorithm.

[0037] 2) Weight decay factor α, 0<α<1.

[0038] 3) Initialize the first n smoothed values, using the actual historical observations.

[0039] 4) Initial mean square error threshold, including the upper mean square error threshold MSE upper and mean square error lower threshold MSE lower .

[0040] 5) The step size of the window size adjustment.

[0041] 6) Set the maximum window size n max and the minimum value n min .

[0042] (2) Data preparation: Assume that there is a database capacity time series C, whose elements are represented by C1, C2, …, C t , C t represents the database capacity at the tth time point; according to the database capacity time series C, the difference sequence D is calculated, and its elements are represented by D t, where D t =C t -C t-1 , represents the change in database capacity from the t-1th time point to the tth time point. For the differential sequence D, starting from the nth element of the sequence, execute the following algorithm steps.

[0043] (3) Weight calculation

[0044] Assume the current window size is n, then in this window, the weights corresponding to the n data from the current data to the previous data are the weight sequence w t ,w t-1 ,...,w t-n+1 , the value of each weight in the sequence is calculated by the following formula:

[0045] w i =α·e -b(t-i)

[0046] Where i = t, t-1, ..., t-n+1, ​​b represents the decay rate, which determines the decay speed. The larger the value of b, the faster the decay. For the first n elements of the differential sequence D, to simplify the calculation process, b is taken as a constant; for the subsequent elements, b is taken as the mean square error corresponding to the element.

[0047] (4) Smoothing value calculation

[0048] When a new observation arrives, the new smoothed value S is calculated using the observation and smoothed value in the current window and the weight sequence. t+1 , and its calculation formula is:

[0049]

[0050] (5) Prediction and mean square error calculation

[0051] Use the current smoothing value S t+1 As the predicted value. Then, calculate S t With D t The mean square error MSE between t , the calculation method is:

[0052]

[0053] (6) Dynamically adjust window size

[0054] According to MSE t Dynamically adjust the window size n:

[0055] 1) If MSE t >MSE upper, indicating that the data fluctuates greatly, at this time, the window size is increased to include more historical information and improve the stability of the data. The size of the window after the increase is n = min (n + step, n max ), where n max Indicates the maximum value of the window.

[0056] 2) If MSE t <MSE lower , indicating that the data fluctuation is small, at this time, the window size is reduced to reduce the dependence on historical data, and the size of the window after reduction is n = max (n-step, n min ), where n min Indicates the minimum value of the window.

[0057] (7) Iterative Update

[0058] When new observations arrive, repeating steps (3) to (6) can dynamically adapt to changes in time series data and optimize prediction performance by adjusting the window size.

[0059] (8) Result output

[0060] When all data are processed, the latest predicted value ΔD of the differential sequence D is obtained, which indicates the change in database capacity within a time unit (such as from the t-1th time point to the tth time point) under the current environmental conditions. Under normal circumstances, the value of ΔD is positive, indicating that the storage capacity of the database is increasing. At this time, the remaining storage space life can be calculated based on the remaining storage space, the value of ΔD and the time unit:

[0061] The remaining storage space life = remaining storage space × time unit / ΔD

[0062] In special circumstances, such as freeing up space after data cleanup, the storage capacity will decrease. In this case, the ΔD value may be negative. In this case, the user needs to be clearly reminded that the storage capacity is on a downward trend and the battery life will not be evaluated for the time being.

[0063] Figure 2 In the example, the information of each module is as follows:

[0064] (1) Data collection module: collects database capacity data from the database or other data sources and stores it in the system's historical database capacity observation database for subsequent processing and use.

[0065] (2) Data management module: This module stores the collected database capacity data in the system database and can implement management operations such as data deletion and backup.

[0066] (3) Data calculation module: read the historical database capacity observation data and calculate the difference sequence. Based on the difference sequence, execute the adaptive weighted multi-period smoothing algorithm and complete the iterative calculation process according to the algorithm.

[0067] (4) Battery life evaluation module: Based on the smoothed value prediction result output by the data calculation module and the current remaining storage space, the battery life of the remaining storage space is calculated.

[0068] (5) User interaction module: provides a user interface for displaying the endurance evaluation results and allows users to adjust algorithm parameters.

[0069] A more specific embodiment:

[0070] In the storage space endurance evaluation system based on database capacity prediction designed by the present invention, the data calculation module is its core module. This module adopts an adaptive weighted multi-period smoothing algorithm. The embodiment of the present invention part explains the specific process of the adaptive weighted multi-period smoothing algorithm.

[0071] The steps of the algorithm mainly include:

[0072] (1) Initialization

[0073] Initialize the parameters in the algorithm. In this embodiment, the following parameters are set:

[0074] 1) Initial window size n = 5

[0075] 2) Weight decay factor α = 0.8

[0076] 3) Initial mean square error threshold MSE upper =0.1,MSE lower =0.01

[0077] 4) Window size adjustment step step = 1

[0078] 5) Maximum window size n max =10, minimum value n min =3

[0079] 6) Initial value of decay rate b=0.1 (for simplicity, it is assumed in the embodiment that b is a constant for all elements of the differential sequence D)

[0080] (2) Data preparation: Assume that we have collected a time series C = [100, 105, 110, 112, 113, 115, 117, 116, 114, 112] of the database capacity. Based on the sequence C, we can calculate the differential sequence D = [5, 5, 2, 1, 2, 2, -1, -2, -2].

[0081] (3) Weight calculation

[0082] For the 5 data in the window, namely D4 to D0, it can be calculated:

[0083] w4=α·e -b(4-4) =0.8·e 0 =0.8·1=0.8

[0084] w3=α·e -b(4-3) =0.8·e -0.1 ≈0.8·0.9048=0.7238

[0085] w2=α·e -b(4-2) =0.8·e -0.2 ≈0.8·0.8187=0.6549

[0086] w1=α·e -b(4-1) =0.8·e -0.3 ≈0.8·0.7408=0.5926

[0087] w0=α·e -b(4-0) =0.8·e -0.4 ≈0.8·0.6703=0.5362

[0088] (4) Smoothing value calculation

[0089] Assume S0=5, S1=5, S2=2, S3=1, S4=2, then according to the formula:

[0090]

[0091] It can be calculated that S5≈2.16.

[0092] (5) Prediction and mean square error calculation

[0093] If no new observations arrive, the above S5 can be used as the final prediction value; if there are new observations, the calculation continues.

[0094] Use the following formula to calculate the predicted value S t Compared with the actual observed value D t The mean square error MSE between t :

[0095]

[0096] In this simplified example, since the first n predicted values ​​S t Compared with the actual observed value D t The same, so the calculated MSE t =0.

[0097] (6) Dynamically adjust window size

[0098] Because MSE t =0, so MSE t <MSE lower , thus reducing the size of the window, so according to:

[0099] n=max(n-step,n min )

[0100] So n=4.

[0101] (7) Iterative Update

[0102] When new observations arrive, repeating steps (3) to (6) can dynamically adapt to changes in time series data and optimize prediction performance by adjusting the window size.

[0103] (8) Result output

[0104] Assume that after processing all differential data, the last predicted value is 1.9, which means that based on historical data, the predicted differential at the next time point will be 1.9, that is, the change in database capacity is 1.9GB. (Here, it is assumed that the unit is GB, and the collection interval of the original database capacity time series C is 1 day)

[0105] Assuming that the remaining capacity of the current database service system is 100 GB, if the database capacity changes according to the smoothing value every day, the estimated life of the remaining storage space is: 100 / 1.9=52.63 days.

[0106] It can be seen from this embodiment that during the calculation process, the algorithm only uses historical data for calculation and does not adopt complex model methods such as neural networks. Therefore, it is very convenient in system design and ensures high real-time performance of the algorithm.

Claims

1. A method for evaluating storage space endurance based on database capacity prediction, the method comprising the following specific steps: (1) Data collection: Collect database capacity data from the database or other data sources and store it in the system's historical database capacity observation database; (2) Data management: This module stores the collected database capacity data in the system database and can delete and back up the data; (3) Data calculation: read the historical database capacity observation data and complete the iterative calculation process according to the algorithm; (4) Endurance evaluation: Based on the smoothed value prediction results output by the data calculation module and the current remaining storage space, the endurance of the remaining storage space is calculated; (5) User interaction: Provides a user interface to display the endurance evaluation results and allow users to adjust algorithm parameters; the features are: When calculating data, the difference sequence is calculated based on the historical database capacity observation data. Based on the difference sequence, the adaptive weighted multi-period smoothing algorithm is executed, and the iterative calculation process is completed according to the algorithm.

2. The method for evaluating storage space endurance based on database capacity prediction according to claim 1, characterized in that: Step (3) specifically includes: (1) Weight calculation: Calculate the weight w of the data at the i-th time point i =α·e -b(t-i) , where n is the current window size. In this window, there are n data from the current data to the previous data, and the weights corresponding to these data are the weight sequence w t ,w t-1 ,...,w t-n+1 , where i = t, t-1, …, t-n+1, ​​b is the decay rate; α is the weight decay factor, 0<α<1; (2) Smoothing value calculation: When a new observation value is reached, the smoothing value is calculated S i is the smoothed value at the i-th time point, D t is the element in the difference sequence D of the database capacity time series C, D t =C t -C t-1 , C t is the element in the database capacity time series C, C t is the database capacity at the tth time point; (3) Prediction and mean square error calculation: Use the current smoothing value S t+1 As the predicted value, calculate S t With D t The mean square error between where k = t, t-1, ..., t-n+1, ​​D k is the change in database capacity from the k-1th time point to the kth time point, S k is the smoothed value at the kth time point. (4) Dynamically adjust the window: If MSE t >MSE upper , the window size after enlargement is min(n+step,n max ), where n max Indicates the maximum value of the window; if MSE t <MSE lower , the window size is reduced to max(n-step,n min ), where n min Indicates the minimum value of the window, step is the adjustment step of the initial window size, MSE upper is the upper threshold of mean square error, MSE lower is the lower threshold of mean square error; (5) Iterative update: When new observations arrive, repeat steps (1) to (4) to optimize the prediction performance by adjusting the window size.

3. The method for evaluating storage space endurance based on database capacity prediction according to claim 1, characterized in that: During user interaction, when all data are processed, the latest predicted value ΔD of the differential sequence D is obtained, and the endurance time of the remaining storage space is calculated = remaining storage space × time unit / ΔD.

4. A storage space endurance assessment system based on database capacity prediction, the system comprising the following modules: (1) Data collection module: collects database capacity data from the database or other data sources and stores it in the system's historical database capacity observation database; (2) Data management module: This module stores the collected database capacity data in the system database and can delete and back up the data; (3) Data calculation module: reads the historical database capacity observation data and completes the iterative calculation process according to the algorithm; (4) Endurance evaluation module: Calculates the endurance of the remaining storage space based on the smoothed value prediction result output by the data calculation module and the current remaining storage space; (5) User interaction module: provides a user interface to display the battery life evaluation results and allows the user to adjust the algorithm parameters; its features are: The data calculation module calculates the difference sequence based on the historical database capacity observation data, executes the adaptive weighted multi-period smoothing algorithm based on the difference sequence, and completes the iterative calculation process according to the algorithm.

5. The storage space endurance evaluation system based on database capacity prediction according to claim 4, characterized in that: The data calculation module includes the following steps: 3.1 Weight calculation: Calculate the weight w of the data at the i-th time point i =α·e -b(t-i) , where n is the current window size. In this window, there are n data from the current data to the previous data, and the weights corresponding to these data are the weight sequence w t ,w t-1 ,...,w t-n+1 , where i = t, t-1, …, t-n+1, ​​b is the decay rate; α is the weight decay factor, 0<α<1; 3.2 Smoothing value calculation: When a new observation value arrives, calculate the smoothing value S i is the smoothed value at the i-th time point, D t is the element in the difference sequence D of the database capacity time series C, D t =C t -C t-1， C t is the element in the database capacity time series C, C t is the database capacity at the tth time point; 3.3 Prediction and mean square error calculation: Using the current smoothing value S t+1 As the predicted value, calculate S t With D t The mean square error between where k = t, t-1, ..., t-n+1, ​​D k is the change in database capacity from the k-1th time point to the kth time point, S k is the smoothed value at the kth time point. 3.4 Dynamically adjust the window: If MSE t >MSE upper , the window size after enlargement is min(n+step,n max ), where n max Indicates the maximum value of the window; if MSE t <MSE lower , the window size is reduced to max(n-step,n min ), where n min Indicates the minimum value of the window, step is the adjustment step of the initial window size, MSE upper is the upper threshold of mean square error, MSE lower is the lower threshold of mean square error; 3.5 Iterative update: When subsequent new observations arrive, repeat steps 3.1 to 3.4 and optimize the prediction performance by adjusting the window size.

6. The storage space endurance evaluation system based on database capacity prediction according to claim 4, characterized in that: In the battery life evaluation module, after all data are processed, the latest predicted value ΔD of the differential sequence D is obtained, and the battery life of the remaining storage space is calculated as remaining storage space × time unit / ΔD.

Images