File system capacity acquisition and early warning method, device, equipment and medium

Abstract

The invention provides a file system capacity acquisition and early warning method and device, equipment and a medium, and the method comprises the steps: obtaining acquisition data corresponding to all file systems in tested equipment according to a set time interval, the acquisition data comprising used capacity, total capacity and acquisition time; according to a set period length, extracting corresponding collection data to obtain a trained file capacity prediction model; setting an alarm threshold level, acquiring real-time acquired data, performing prediction by adopting the file capacity prediction model to obtain a predicted capacity, and if the predicted capacity is greater than a set alarm threshold, triggering an early warning notification; if the predicted capacity is smaller than or equal to the set alarm threshold value and the predicted capacity is kept stable, the early warning notification frequency is reduced; if the predicted capacity is smaller than or equal to the set alarm threshold value and the predicted capacity is not kept stable, the early warning notification frequency is increased; the alarm accuracy is improved, invalid alarms are reduced, and the pressure of operation and maintenance personnel is relieved.

Description

Technical Field

[0001] The present invention relates to the field of computer early warning technology, and in particular to a method, device, equipment and medium for file system capacity acquisition and early warning. Background Art

[0002] Computer systems and their applications usually read and write to the file system during operation. If the file system usage has reached the upper limit, the system and applications will not be able to provide normal services. Therefore, in order to discover and avoid file system capacity problems in the shortest possible time, the industry has designed various monitoring and alarm devices for file system usage. The current traditional method is to check the file system in real time. When the usage reaches the set usage threshold, an alarm is triggered to remind the operation and maintenance personnel to take early action. This method has the following disadvantages:

[0003] 1. Fixed thresholds are inflexible and it is difficult to find a suitable universal threshold. If you want to use threshold alarms to prevent the file system usage from reaching the upper limit, you need to set a reasonable threshold. Setting a lower alarm threshold can reduce the risk of space exhaustion, but it will lead to frequent alarms and easily waste storage resources.

[0004] 2. It is difficult to cope with the capacity surge caused by business peaks. By simply using the threshold, it is difficult for operation and maintenance personnel to predict whether the capacity will surge. Once an alarm is generated, the file system capacity may already be in the process of surge.

[0005] 3. Collecting and transmitting data takes up a lot of system resources. In order to ensure the real-time and reliability of monitoring data, protocols such as TCP and WebSocket are generally used, or message queue transmission tools are used. When there are many targets to be measured, the performance requirements of the network and the receiving server are relatively high.

[0006] In short, traditional file system capacity alarms have the disadvantages of many invalid alarms and failure to detect problems in a timely manner. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method, device, equipment and medium for file system capacity collection and early warning, which is lightweight, can improve the accuracy of alarms, detect potential risks early, reduce invalid alarms, and alleviate the pressure on operation and maintenance personnel.

[0008] In a first aspect, the present invention provides a method for file system capacity collection and early warning, comprising:

[0009] Step 1: Acquire the collected data corresponding to all file systems in the device under test at a set time interval, wherein the collected data includes: used capacity, total capacity and collection time;

[0010] Step 2: According to the set cycle length, the corresponding collected data is extracted, and the STL seasonal trend decomposition algorithm is used to seasonally decompose the used capacity sequence of the file system to determine whether there is seasonality. If there is seasonality, the SARIMA model is used for training; if there is no seasonality, the ARIMA model is used for training to obtain the trained SARIMA file capacity prediction model and ARIMA file capacity prediction model respectively;

[0011] Step 3: Set the alarm threshold level, obtain real-time collected data, use the file capacity prediction model to predict, and obtain the predicted capacity. If the predicted capacity is greater than the set alarm threshold, trigger an early warning notification;

[0012] If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity remains stable within the set number of days after the current period, the frequency of early warning notifications is reduced;

[0013] If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity does not remain stable within the set number of days after the current period, the frequency of early warning notifications is increased.

[0014] In a second aspect, the present invention provides a device for collecting and warning file system capacity, comprising:

[0015] The data acquisition module acquires the collected data corresponding to all file systems in the device under test at a set time interval, wherein the collected data includes: used capacity, total capacity and acquisition time;

[0016] The training model module extracts the corresponding collected data according to the set cycle length, and uses the STL seasonal trend decomposition algorithm to seasonally decompose the used capacity sequence of the file system to determine whether there is seasonality. If there is seasonality, the SARIMA model is used for training; if there is no seasonality, the ARIMA model is used for training to obtain the trained SARIMA file capacity prediction model and ARIMA file capacity prediction model respectively;

[0017] The prediction module sets the alarm threshold level, obtains the real-time collected data, uses the file capacity prediction model to predict, and obtains the predicted capacity. If the predicted capacity is greater than the set alarm threshold, an early warning notification is triggered;

[0018] If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity remains stable within the set number of days after the current period, the frequency of early warning notifications is reduced;

[0019] If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity does not remain stable within the set number of days after the current period, the frequency of early warning notifications is increased.

[0020] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0022] 1. Compared with traditional monitoring methods, the present invention can predict problems with file system capacity in advance, greatly reducing the risk of excessive file system capacity utilization and preventing problems before they occur.

[0023] 2. The present invention trains multiple models to make predictions for a file system object, thereby improving the reliability of the predictions.

[0024] 3. The present invention can solve the problem of too many low-level alarms in traditional monitoring and reduce unimportant information.

[0025] 4. The present invention adopts a lightweight data transmission method, which has small demands on network resources and program running resources.

[0026] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0028] Figure 1 It is a schematic diagram of the framework of the system of the present invention;

[0029] Figure 2 is a flow chart of the method in Embodiment 1 of the present invention;

[0030] Figure 3 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0031] The embodiments of the present application provide a method, device, equipment and medium for collecting and warning the capacity of a file system, so as to solve the technical problems that a traditional file system capacity alarm has many invalid alarms and cannot detect problems in a timely manner.

[0032] The technical solution in the embodiment of the present application has the following general idea:

[0033] In view of the problems of traditional monitoring methods such as inability to make early forecasts, many invalid alarms, and high usage of transmission resources, Figure 1 As shown, the present invention includes a collection layer, a receiving layer, a prediction and analysis layer, and an alarm processing layer.

[0034] 1. The acquisition layer will obtain monitoring data including file system capacity usage and total capacity information, and splice it with the acquisition code of this data as monitoring content. The monitoring content is encrypted and spliced ​​with salt value, and the monitoring data is sent using UDP protocol. The sender sends data regularly, and the same data is sent multiple times.

[0035] 2. After decryption at the receiving layer, duplicate messages are removed according to the identity code to obtain the collected data once, and the data is persisted in the database.

[0036] 3. The prediction model layer analyzes and processes the collected data, and the data is input into the fixed alarm rule analyzer and the prediction model analyzer respectively. The prediction model can identify file systems with different change types, such as stable trend type, periodic type, etc.

[0037] 4. The alarm processing layer comprehensively processes the analysis results of fixed alarm rules and the prediction analysis results, and automatically upgrades and downgrades the alarms generated by fixed alarm rules, making the alarm notifications flexible and forward-looking.

[0038] Specifically:

[0039] 1. Collect the used capacity and total capacity of all file systems in the system under test at fixed intervals, and generate the acquisition code in combination with time and device name. The agreed reversible function y=f(x) is used to encrypt the acquisition code as the salt value, that is, the salt value=f(acquisition code). The collected content is encrypted with salt using an encryption algorithm, and finally the salt value is spliced ​​at the front of the message data as the UDP message data. The encryption algorithm can select a public algorithm, such as 3DES and other encryption algorithms. According to the network environment, the collected data is sent n times at a time, and n is greater than or equal to 1.

[0040] 2. The receiving end obtains the message and obtains the acquisition code = f based on the salt value at the front of the message data -1 (Salt value), discard the message with illegal collection code, remove the duplicate message, use the encryption algorithm to decrypt and get the collection data. Use the device ID, collection time, and file system ID as the primary key to store capacity data in the database.

[0041] 3. Take a device as a whole and process the data collected from each file system in it. Follow the steps below to train three models for each file system in the device: 30-day model, 7-day model, and 24-hour model.

[0042] 3.1 Data extraction and calculation: Select week, day, and hour as the cycle length, extract data: select the used capacity data of the past 30 days, calculate the daily average used capacity of these 30 days, and use it to determine the seasonality of the cycle of 7. For 7 days, take the average of each hour of the past seven days to determine the seasonality of the cycle of one day. For 24 hours, take the measurement points in minutes in each hour of the past 24 hours at fixed time intervals to determine the seasonality of the cycle of one hour; finally, for a file system, three used capacity sequences will be obtained with time windows of 30 days, 7 days, and 24 hours respectively.

[0043] 3.2 Determine whether there is seasonality: Use the STL seasonal trend decomposition algorithm to seasonally decompose the used capacity sequence. YT, YS, YR = STL (Y) (that is, the output value of the STL function is three sequences YT, YS, YR; Y is the used capacity sequence), where YT = [dt1, dt2...dti] is the trend term, YS = [ds1, ds2...dsi] is the seasonal term, and YR = [dr1, dr2...dri] is the random term. Calculate the seasonal strength SS = max(0,1-(Var(YR) / Var(YS+YR))), Var represents the variance calculation formula. If the cycle strength is greater than the preset threshold, the capacity series data is marked as seasonal; this step is to select the parameter s of the SARIMA model, that is, the cycle length. If there is periodicity, s is the cycle length; if there is no periodicity, s = 0, and the ARIMA model is used for prediction.

[0044] 3.3 Model parameter selection and training: When the capacity series data is marked as seasonal, perform an ADF test on the sequence Y processed in 3.1 to obtain whether it is stable or not. If it is not stable, perform a first or second difference on the sequence Y processed in 3.1 to obtain Ydiff. Perform seasonal difference according to the cycle length s to obtain the sequence Y(d1,s)=Ydiff t -Ydiff t-s , which describes the calculation method of difference, which means the value at time t in the sequence minus the value at time ts, and processes each value in the sequence to obtain a new sequence; repeat the ADF test and difference order adjustment, and if a stable sequence can be obtained, proceed with the subsequent operation; the stable sequence is represented by Y d1,s, where s is the cycle length and d1 is the difference order. The seasonal autoregressive integrated moving average model SARIMA is used. The model is expressed as SARIMA(p,d1,q)(P,D,Q)s, where the non-seasonal part of the model parameters: p is the autoregressive order, d1 is the difference order, q is the moving average order; seasonal part parameters: P is the autoregressive order, D is the difference order, Q is the moving average order, and s is the cycle length. The data is the stationary time series data Y after difference. d1,s , Use the grid search method to determine the remaining SARIMA parameters (i.e., s, d1, and D have been determined, and p, q, P, and Q need to be calculated) to obtain the parameter values ​​with the best AIC value. Use the file system's used capacity sequence (the sequence obtained in 3.1) to fit the SARIMA model and obtain the trained SARIMA file capacity prediction model;

[0045] Similarly, when the capacity series data is not seasonal, the sequence Y processed in 3.1 is differentiated once or twice to obtain Y(d2)=Ydiff; the ADF test and difference order adjustment are repeated. If a stationary sequence can be obtained, the subsequent operation is performed; the stationary sequence is expressed as Y d2 , where d2 is the difference order. The ARIMA model is used, expressed as ARIMA(p,d2,q), where the ARIMA model parameters are: p is the autoregressive order, d2 is the difference order, and q is the moving average order; the data is the differenced stationary time series data Y d2 , Use the grid search method to determine the remaining ARIMA parameters and obtain the parameter value with the optimal AIC value. Use the file system's used capacity sequence (the sequence obtained in 3.1) to fit the ARIMA model and obtain the trained ARIMA file capacity prediction model.

[0046] 3.4 Model rolling update: As time goes by, when the collected file capacity data is sufficient to calculate the capacity data of the next observation point of the sequence in 3.1, the earliest element of the sequence should be removed and the latest data should be added as a new training sequence. Use the new sequence to refit the model in 3.3.

[0047] 3.5 Model monitoring and re-identification: Due to changes in business behavior and other reasons, the law of file system capacity changes has changed, and the model needs to be re-identified and the model parameters adjusted. The method to determine whether adjustment is needed is as follows: For each prediction, record the predicted value. Use the predicted value and the actual value obtained in the future to calculate the mean square error (MSE) value. Set a threshold k. If the error increases for k consecutive times, it indicates that the model prediction accuracy has decreased. It is necessary to perform 3.2 and 3.3 to reselect the prediction model and calculate the model parameters.

[0048] 4. Alarm processing layer:

[0049] 4.1. In the fixed alarm rules, set a series of alarm threshold levels: a -i ...a -1 ,a 0 ,a 1 ...a i Here, the larger the i is, the higher the alarm level is. Assuming i>=0 is a serious alarm, i<0 is a minor alarm, and the minor alarm will be sent repeatedly at regular intervals.

[0050] 4.2. In the prediction analysis model, three models are used: 30-day model, 7-day model, and 24-hour model. For each file system, the trained 30-day model is used to predict the capacity of the time 1 to 5 days after the current time, the trained 7-day model is used to predict the capacity of the time 1 to 2 days after the current time, and the 24-hour model is used to predict the time 1 hour after the current time. Define the current time as T, and the prediction from T to T+1 days is the short-term prediction; the prediction from T+2 to T+5 days is the long-term prediction.

[0051] 4.3. Perform the following processing for warning notifications and fixed rule notifications: 1. If the short-term forecast capacity or real-time capacity is greater than a 0 1. If the short-term forecast capacity is less than a0, the long-term forecast capacity remains stable within 5 days after the current period (the stability means that the corresponding forecast capacity obtained by the 30-day model, the 7-day model, and the 24-hour model are all less than a set threshold), and the fixed rule notification of the current real-time capacity is in the secondary interval, then the notification frequency is reduced. 0 , if the long-term forecast capacity does not remain stable within 5 days after the current period (the stability means that the corresponding forecast capacity obtained by the 30-day model, the 7-day model, and the 24-hour model is greater than a set threshold), and the fixed rule notification of the current real-time capacity is in the secondary interval, then the notification frequency is increased. 4. If there is no forecast model, and the current real-time used capacity is less than a 0 , then set the alarm notification frequency according to the real-time capacity threshold level.

[0052] Embodiment 1

[0053] like Figure 2 As shown, this embodiment provides a method for collecting and warning the capacity of a file system, including:

[0054] Step 1: Acquire the collected data corresponding to all file systems in the device under test at a set time interval, wherein the collected data includes: used capacity, total capacity and collection time;

[0055] Step 2: According to the set cycle length, the corresponding collected data is extracted, and the STL seasonal trend decomposition algorithm is used to seasonally decompose the used capacity sequence of the file system to determine whether there is seasonality. If there is seasonality, the SARIMA model is used for training; if there is no seasonality, the ARIMA model is used for training to obtain the trained SARIMA file capacity prediction model and ARIMA file capacity prediction model respectively;

[0056] Step 3: Set the alarm threshold level, obtain real-time collected data, use the file capacity prediction model to predict, and obtain the predicted capacity. If the predicted capacity is greater than the set alarm threshold, trigger an early warning notification;

[0057] If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity remains stable within the set number of days after the current period, the frequency of early warning notifications is reduced;

[0058] If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity does not remain stable within the set number of days after the current period, the frequency of early warning notifications is increased.

[0059] In this embodiment, preferably, the step 2 is specifically:

[0060] The file capacity prediction model includes a 30-day prediction model, a 7-day prediction model and a 24-hour prediction model;

[0061] Select the collected data of the past 30 days, calculate the daily average used capacity of these 30 days, and obtain the used capacity sequence of 30 days;

[0062] Select the collected data of the past 7 days, calculate the average hourly used capacity in these 7 days, and obtain the 7-day used capacity sequence;

[0063] Select the collected data of the past 24 hours, calculate the average used capacity at fixed time intervals in minutes within each hour, and obtain the used capacity sequence of 24 hours;

[0064] Use the STL seasonal trend decomposition algorithm to seasonally decompose the used capacity series of 30 days, 7 days and 24 hours; YT, YS, YR = STL (Y), where YT = [dt1, dt2...dti] is the trend term, YS = [ds1, ds2...dsi] is the seasonal term, and YR = [dr1, dr2...dri] is the random term; calculate the seasonal strength SS = max(0,1-(Var(YR) / Var(YS+YR))), Var represents the variance calculation formula; if the seasonal strength SS is greater than the preset threshold, the used capacity series of 30 days, 7 days or 24 hours is marked as seasonal, otherwise it is not seasonal, and s is 0;

[0065] When the used capacity series of 30 days, 7 days or 24 hours is marked as seasonal, the ADF test is performed on the used capacity series of 30 days, 7 days and 24 hours; if it is not stationary, the used capacity series of 30 days, 7 days and 24 hours is differentiated once or twice to obtain Ydiff; seasonal differentiation is performed to obtain the sequence Y(d1,s)=Ydiff t -Ydiff t-s Repeat the ADF test and difference order adjustment to obtain the corresponding new used capacity sequence. If a stable sequence can be obtained, the stable sequence is expressed as Y d1,s , where s is the cycle length and d1 is the difference order; the seasonal autoregressive integrated moving average model SARIMA is used. The SARIMA model is expressed as SARIMA(p,d1,q)(P,D,Q)s, where the non-seasonal part of the model parameters: p is the autoregressive order, d1 is the difference order, q is the moving average order; seasonal part parameters: P is the autoregressive order, D is the difference order, Q is the moving average order, s is the cycle length; the data is the differenced stationary time series data Y d1,s ,Use the grid search method to determine the remaining SARIMA parameters and obtain the parameter values ​​with the best AIC value; Use the file system's used capacity sequence to fit the SARIMA model and obtain the trained SARIMA file capacity prediction model;

[0066] When the used capacity series of 30 days, 7 days or 24 hours is marked as non-seasonal, the ADF test is performed on the used capacity series of 30 days, 7 days and 24 hours; if it is not stable, the used capacity series of 30 days, 7 days and 24 hours is differentiated once or twice to obtain the sequence Y(d2)=Ydiff; repeat the ADF test and difference order adjustment to obtain the corresponding new used capacity series. If a stable series can be obtained, the stable series is expressed as Y d2 , where d2 is the difference order, the ARIMA model is used, and the ARIMA model is expressed as ARIMA(p,d2,q), where the model parameters are: p is the autoregressive order, d2 is the difference order, and q is the moving average order; the data is the stationary time series data Y after difference d2 , use the grid search method to determine the remaining ARIMA parameters and obtain the parameter values ​​with the optimal AIC value; use the file system's used capacity sequence to fit the ARIMA model and obtain the trained ARIMA file capacity prediction model;

[0067] If a stationary sequence cannot be obtained, proceed directly to step 3.

[0068] In this embodiment, preferably, step 3 is specifically:

[0069] Use the trained 30-day forecast model to forecast capacity for 1 to 5 days after the current time, and obtain forecast values ​​of multiple 30-day forecast models;

[0070] Use the trained 7-day forecast model to forecast capacity for 1 to 2 days after the current time, and obtain forecast values ​​of multiple 7-day forecast models;

[0071] Use the 24-hour prediction model to predict 1 hour after the current time and obtain the prediction values ​​of multiple 24-hour prediction models;

[0072] And obtain the real-time used capacity; define the current time as T, the forecast value from T to T+1 is the short-term forecast; the forecast value from T+2 to T+5 is the long-term forecast;

[0073] Set the alarm threshold level: a -i ...a -1 ,a 0 ,a 1 ...a i , the larger i is, the higher the alarm level is; set i>=0 for severe alarm, i<0 for minor alarm, minor warning will send warning notification repeatedly according to the frequency set by the level, severe alarm will send warning notification directly, i is a positive integer;

[0074] If the short-term forecast or real-time used capacity is greater than a 0 Then send a warning notification directly;

[0075] If the short-term forecast is less than a 0 , the long-term forecast remains stable within 5 days after the current period, which means that the corresponding forecast values ​​obtained by the 30-day forecast model, the 7-day forecast model, and the 24-hour forecast model are all less than the set threshold, and the current real-time used capacity is less than a 0 , and when the alarm threshold level of the short-term forecast is less than or equal to the alarm threshold level of the real-time used capacity, the set frequency is lowered by a set value, otherwise the corresponding notification frequency is set according to the alarm threshold level of the short-term forecast;

[0076] If the short-term forecast is less than a 0 , the long-term forecast will not remain stable within 5 days after the current period. The instability means that the corresponding forecast values ​​obtained by the 30-day forecast model, the 7-day forecast model, and the 24-hour forecast model are all greater than or equal to the set threshold, and the current real-time used capacity is less than a 0 , then according to the short-term forecast at the alarm threshold level, the set frequency is increased by a set value to obtain the adjusted frequency, and an early warning notification is sent according to the adjusted frequency;

[0077] If there is no predicted value, and the current real-time used capacity is less than a 0 , then set the alarm notification frequency according to the capacity threshold level.

[0078] In this embodiment, preferably, step 2 also includes: when the newly collected data is sufficient to calculate the set observation points, processing the newly collected data as a new training sequence; using the new training sequence to refit the 30-day prediction model, the 7-day prediction model and the 24-hour prediction model; and obtaining the corresponding latest prediction model.

[0079] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0080] Embodiment 2

[0081] like Figure 3 As shown, in this embodiment, a device for collecting and warning the capacity of a file system is provided, including:

[0082] The data acquisition module acquires the collected data corresponding to all file systems in the device under test at a set time interval, wherein the collected data includes: used capacity, total capacity and acquisition time;

[0083] The training model module extracts the corresponding collected data according to the set cycle length, and uses the STL seasonal trend decomposition algorithm to seasonally decompose the used capacity sequence of the file system to determine whether there is seasonality. If there is seasonality, the SARIMA model is used for training; if there is no seasonality, the ARIMA model is used for training to obtain the trained SARIMA file capacity prediction model and ARIMA file capacity prediction model respectively;

[0084] The prediction module sets the alarm threshold level, obtains the real-time collected data, uses the file capacity prediction model to predict, and obtains the predicted capacity. If the predicted capacity is greater than the set alarm threshold, an early warning notification is triggered;

[0085] If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity remains stable within the set number of days after the current period, the frequency of early warning notifications is reduced;

[0086] If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity does not remain stable within the set number of days after the current period, the frequency of early warning notifications is increased.

[0087] In this embodiment, preferably, the training model module is specifically:

[0088] The file capacity prediction model includes a 30-day prediction model, a 7-day prediction model and a 24-hour prediction model;

[0089] Select the collected data of the past 30 days, calculate the daily average used capacity of these 30 days, and obtain the used capacity sequence of 30 days;

[0090] Select the collected data of the past 7 days, calculate the average hourly used capacity in these 7 days, and obtain the 7-day used capacity sequence;

[0091] Select the collected data of the past 24 hours, calculate the average used capacity at fixed time intervals in minutes within each hour, and obtain the used capacity sequence of 24 hours;

[0092] Use the STL seasonal trend decomposition algorithm to seasonally decompose the used capacity series of 30 days, 7 days and 24 hours; YT, YS, YR = STL (Y), where YT = [dt1, dt2...dti] is the trend term, YS = [ds1, ds2...dsi] is the seasonal term, and YR = [dr1, dr2...dri] is the random term; calculate the seasonal strength SS = max(0,1-(Var(YR) / Var(YS+YR))), Var represents the variance calculation formula; if the seasonal strength SS is greater than the preset threshold, the used capacity series of 30 days, 7 days or 24 hours is marked as seasonal, otherwise it is not seasonal, and s is 0;

[0093] When the used capacity series of 30 days, 7 days or 24 hours is marked as seasonal, the ADF test is performed on the used capacity series of 30 days, 7 days and 24 hours; if it is not stationary, the used capacity series of 30 days, 7 days and 24 hours is differentiated once or twice to obtain Ydiff; seasonal differentiation is performed to obtain the sequence Y(d1,s)=Ydiff t -Ydiff t-s Repeat the ADF test and difference order adjustment to obtain the corresponding new used capacity sequence. If a stable sequence can be obtained, the stable sequence is expressed as Y d1,s , where s is the cycle length and d1 is the difference order; the seasonal autoregressive integrated moving average model SARIMA is used. The SARIMA model is expressed as SARIMA(p,d1,q)(P,D,Q)s, where the non-seasonal part of the model parameters: p is the autoregressive order, d1 is the difference order, q is the moving average order; seasonal part parameters: P is the autoregressive order, D is the difference order, Q is the moving average order, s is the cycle length; the data is the differenced stationary time series data Y d1,s ,Use the grid search method to determine the remaining SARIMA parameters and obtain the parameter values ​​with the best AIC value; Use the file system's used capacity sequence to fit the SARIMA model and obtain the trained SARIMA file capacity prediction model;

[0094] When the used capacity series of 30 days, 7 days or 24 hours is marked as non-seasonal, the ADF test is performed on the used capacity series of 30 days, 7 days and 24 hours; if it is not stable, the used capacity series of 30 days, 7 days and 24 hours is differentiated once or twice to obtain the sequence Y(d2)=Ydiff; repeat the ADF test and difference order adjustment to obtain the corresponding new used capacity series. If a stable series can be obtained, the stable series is expressed as Y d2 , where d2 is the difference order, the ARIMA model is used, and the ARIMA model is expressed as ARIMA(p,d2,q), where the model parameters are: p is the autoregressive order, d2 is the difference order, and q is the moving average order; the data is the stationary time series data Y after difference d2 , where d2 has been determined; use the grid search method to determine the remaining ARIMA parameters and obtain the parameter values ​​with the optimal AIC value; use the file system's used capacity sequence to fit the ARIMA model and obtain a trained ARIMA file capacity prediction model;

[0095] If a stationary sequence cannot be obtained, it directly enters the prediction module.

[0096] In this embodiment, preferably, the prediction module is specifically:

[0097] Use the trained 30-day forecast model to forecast capacity for 1 to 5 days after the current time, and obtain forecast values ​​of multiple 30-day forecast models;

[0098] Use the trained 7-day forecast model to forecast capacity for 1 to 2 days after the current time, and obtain forecast values ​​of multiple 7-day forecast models;

[0099] Use the 24-hour prediction model to predict 1 hour after the current time and obtain the prediction values ​​of multiple 24-hour prediction models;

[0100] And obtain the real-time used capacity; define the current time as T, the forecast value from T to T+1 is the short-term forecast; the forecast value from T+2 to T+5 is the long-term forecast;

[0101] Set the alarm threshold level: a -i ...a -1 ,a 0 ,a 1 ...a i , the larger i is, the higher the alarm level is; set i>=0 for severe alarm, i<0 for minor alarm, minor warning will send warning notification repeatedly according to the frequency set by the level, severe alarm will send warning notification directly, i is a positive integer;

[0102] If the short-term forecast or real-time used capacity is greater than a 0 Then send a warning notification directly;

[0103] If the short-term forecast is less than a 0 , the long-term forecast remains stable within 5 days after the current period, which means that the corresponding forecast values ​​obtained by the 30-day forecast model, the 7-day forecast model, and the 24-hour forecast model are all less than the set threshold, and the current real-time used capacity is less than a 0 , and when the alarm threshold level of the short-term forecast is less than or equal to the alarm threshold level of the real-time used capacity, the set frequency is lowered by a set value, otherwise the corresponding notification frequency is set according to the alarm threshold level of the short-term forecast;

[0104] If the short-term forecast is less than a 0 , the long-term forecast will not remain stable within 5 days after the current period. The instability means that the corresponding forecast values ​​obtained by the 30-day forecast model, the 7-day forecast model, and the 24-hour forecast model are all greater than or equal to the set threshold, and the current real-time used capacity is less than a 0 , then according to the short-term forecast at the alarm threshold level, the set frequency is increased by a set value to obtain the adjusted frequency, and an early warning notification is sent according to the adjusted frequency;

[0105] If there is no predicted value, and the current real-time used capacity is less than a 0 , then set the alarm notification frequency according to the capacity threshold level.

[0106] In this embodiment, preferably, the training model module also includes: when the newly collected data is sufficient to calculate the set observation points, the newly collected data is processed as a new training sequence; the 30-day prediction model, the 7-day prediction model and the 24-hour prediction model are refitted using the new training sequence; and the corresponding latest prediction model is obtained.

[0107] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0108] Based on the same inventive concept, the present application provides an electronic device embodiment corresponding to the first embodiment, see the third embodiment for details.

[0109] Embodiment 3

[0110] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any implementation method in the first embodiment can be implemented.

[0111] Since the electronic device introduced in this embodiment is a device used to implement the method in the first embodiment of the present application, based on the method introduced in the first embodiment of the present application, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as a person skilled in the art implements the device used by the method in the embodiment of the present application, it belongs to the scope of protection of the present application.

[0112] Based on the same inventive concept, the present application provides a storage medium corresponding to the first embodiment, see the fourth embodiment for details.

[0113] Embodiment 4

[0114] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any implementation method in the first embodiment can be implemented.

[0115] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages: the method, device, equipment and medium provided in the embodiments of the present application can predict the problems existing in the file system capacity in advance, greatly reducing the risk of excessive utilization of the file system capacity and preventing problems before they occur; for a file system object, multiple models are trained for prediction, thereby improving the reliability of the prediction; the problem of too many low-level alarms in traditional monitoring is solved, and unimportant information is reduced; a lightweight data transmission method is adopted, with low demand on network resources and program running resources.

[0116] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0118] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0119] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0120] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for collecting and warning file system capacity, characterized in that: include: Step 1: Acquire the collected data corresponding to all file systems in the device under test at a set time interval, wherein the collected data includes: used capacity, total capacity and collection time; Step 2: According to the set cycle length, the corresponding collected data is extracted, and the STL seasonal trend decomposition algorithm is used to seasonally decompose the used capacity sequence of the file system to determine whether there is seasonality. If there is seasonality, the SARIMA model is used for training; if there is no seasonality, the ARIMA model is used for training to obtain the trained SARIMA file capacity prediction model and ARIMA file capacity prediction model respectively; Step 3: Set the alarm threshold level, obtain real-time collected data, use the file capacity prediction model to predict, and obtain the predicted capacity. If the predicted capacity is greater than the set alarm threshold, trigger an early warning notification; If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity remains stable within the set number of days after the current period, the frequency of early warning notifications is reduced; If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity does not remain stable within the set number of days after the current period, the frequency of early warning notifications is increased.

2. A method for collecting and warning file system capacity according to claim 1, characterized in that: The step 2 is specifically as follows: The file capacity prediction model includes a 30-day prediction model, a 7-day prediction model and a 24-hour prediction model; Select the collected data of the past 30 days, calculate the daily average used capacity of these 30 days, and obtain the used capacity sequence of 30 days; Select the collected data of the past 7 days, calculate the average hourly used capacity in these 7 days, and obtain the 7-day used capacity sequence; Select the collected data of the past 24 hours, calculate the average used capacity at fixed time intervals in minutes within each hour, and obtain the used capacity sequence of 24 hours; Use the STL seasonal trend decomposition algorithm to seasonally decompose the used capacity series of 30 days, 7 days and 24 hours; YT, YS, YR = STL (Y), where YT = [dt1, dt2...dti] is the trend term, YS = [ds1, ds2...dsi] is the seasonal term, and YR = [dr1, dr2...dri] is the random term; calculate the seasonal strength SS = max(0,1-(Var(YR) / Var(YS+YR))), Var represents the variance calculation formula; if the seasonal strength SS is greater than the preset threshold, the used capacity series of 30 days, 7 days or 24 hours is marked as seasonal, otherwise it is not seasonal, and s is 0; When the used capacity series of 30 days, 7 days or 24 hours is marked as seasonal, the ADF test is performed on the used capacity series of 30 days, 7 days and 24 hours; if it is not stationary, the used capacity series of 30 days, 7 days and 24 hours is differentiated once or twice to obtain Ydiff; seasonal differentiation is performed to obtain the sequence Y(d1,s)=Ydiff t -Ydiff t-s Repeat the ADF test and difference order adjustment to obtain the corresponding new used capacity sequence. If a stable sequence can be obtained, the stable sequence is expressed as Y d1,s , where s is the cycle length and d1 is the difference order; the seasonal autoregressive integrated moving average model SARIMA is used. The SARIMA model is expressed as SARIMA(p,d1,q)(P,D,Q)s, where the non-seasonal part parameters of the model parameters are: p is the autoregressive order, d1 is the difference order, and q is the moving average order; the seasonal part parameters are: P is the autoregressive order, D is the difference order, Q is the moving average order, and s is the cycle length; The data is selected from the stationary time series data Y after difference d1,s , the grid search method is used to determine the remaining SARIMA parameters and obtain the parameter values ​​with the best AIC value; The SARIMA model is fitted using the used capacity sequence of the file system to obtain a trained SARIMA file capacity prediction model; When the used capacity series of 30 days, 7 days or 24 hours is marked as non-seasonal, the ADF test is performed on the used capacity series of 30 days, 7 days and 24 hours; if it is not stable, the used capacity series of 30 days, 7 days and 24 hours is differentiated once or twice to obtain the sequence Y(d2)=Ydiff; repeat the ADF test and difference order adjustment to obtain the corresponding new used capacity series. If a stable series can be obtained, the stable series is expressed as Y d2 , where d2 is the difference order, the ARIMA model is used, and the ARIMA model is expressed as ARIMA(p,d2,q), where the model parameters are: p is the autoregressive order, d2 is the difference order, and q is the moving average order; the data is the stationary time series data Y after difference d2 , using the grid search method, determine the remaining ARIMA parameters and obtain the parameter values ​​with the best AIC value; Use the file system's used capacity sequence to fit the ARIMA model and obtain a trained ARIMA file capacity prediction model; If a stationary sequence cannot be obtained, proceed directly to step 3.

3. A method for collecting and warning file system capacity according to claim 2, characterized in that: The step 3 is specifically as follows: Use the trained 30-day forecast model to forecast capacity for 1 to 5 days after the current time, and obtain forecast values ​​of multiple 30-day forecast models; Use the trained 7-day forecast model to forecast capacity for 1 to 2 days after the current time, and obtain forecast values ​​of multiple 7-day forecast models; Use the 24-hour prediction model to predict 1 hour after the current time and obtain the prediction values ​​of multiple 24-hour prediction models; And obtain the real-time used capacity; define the current time as T, the forecast value from T to T+1 is the short-term forecast; the forecast value from T+2 to T+5 is the long-term forecast; Set the alarm threshold level: a -i ...a -1 ,a0,a1...a i , the larger i is, the higher the alarm level is; set i>=0 for severe alarm, i<0 for minor alarm, minor warning will send warning notification repeatedly according to the frequency set by the level, severe alarm will send warning notification directly, i is a positive integer; If the short-term forecast or real-time used capacity is greater than a0, an early warning notification is sent directly; If the short-term forecast is less than a0, and the long-term forecast remains stable within 5 days after the current period, the stability means that the corresponding forecast values ​​obtained by the 30-day forecast model, the 7-day forecast model, and the 24-hour forecast model are all less than the set threshold, and the current real-time used capacity is less than a0, and when the alarm threshold level of the short-term forecast is less than or equal to the alarm threshold level of the real-time used capacity, the set frequency is adjusted down by a set value, otherwise the corresponding notification frequency is set according to the short-term forecast being at the alarm threshold level; If the short-term forecast is less than a0, and the long-term forecast does not remain stable within 5 days after the current period, and the instability means that the corresponding forecast values ​​obtained by the 30-day forecast model, the 7-day forecast model, and the 24-hour forecast model are all greater than or equal to the set threshold, and the current real-time used capacity is less than a0, then according to the short-term forecast being at the alarm threshold level, the set frequency is increased by a set value to obtain an adjusted frequency, and an early warning notification is sent according to the adjusted frequency; If there is no predicted value and the current real-time used capacity is less than a0, the alarm notification frequency is set according to the capacity threshold level.

4. A method for collecting and warning file system capacity according to claim 2, characterized in that: The step 2 also includes: when the newly collected data is sufficient to calculate the set observation points, the newly collected data is processed as a new training sequence; the 30-day prediction model, the 7-day prediction model and the 24-hour prediction model are refitted using the new training sequence; and the corresponding latest prediction model is obtained.

5. A device for collecting and warning file system capacity, characterized in that: include: The data acquisition module acquires the collected data corresponding to all file systems in the device under test at a set time interval, wherein the collected data includes: used capacity, total capacity and acquisition time; The training model module extracts the corresponding collected data according to the set cycle length, and uses the STL seasonal trend decomposition algorithm to seasonally decompose the used capacity sequence of the file system to determine whether there is seasonality. If there is seasonality, the SARIMA model is used for training; if there is no seasonality, the ARIMA model is used for training to obtain the trained SARIMA file capacity prediction model and ARIMA file capacity prediction model respectively; The prediction module sets the alarm threshold level, obtains the real-time collected data, uses the file capacity prediction model to predict, and obtains the predicted capacity. If the predicted capacity is greater than the set alarm threshold, an early warning notification is triggered; If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity remains stable within the set number of days after the current period, the frequency of early warning notifications is reduced; If the predicted capacity is less than or equal to the set alarm threshold, and the predicted capacity does not remain stable within the set number of days after the current period, the frequency of early warning notifications is increased.

6. The device for collecting and warning file system capacity according to claim 5, characterized in that: The training model module is specifically: The file capacity prediction model includes a 30-day prediction model, a 7-day prediction model and a 24-hour prediction model; Select the collected data of the past 30 days, calculate the daily average used capacity of these 30 days, and obtain the used capacity sequence of 30 days; Select the collected data of the past 7 days, calculate the average hourly used capacity in these 7 days, and obtain the 7-day used capacity sequence; Select the collected data of the past 24 hours, calculate the average used capacity at fixed time intervals in minutes within each hour, and obtain the used capacity sequence of 24 hours; Use the STL seasonal trend decomposition algorithm to seasonally decompose the used capacity series of 30 days, 7 days and 24 hours; YT, YS, YR = STL (Y), where YT = [dt1, dt2...dti] is the trend term, YS = [ds1, ds2...dsi] is the seasonal term, and YR = [dr1, dr2...dri] is the random term; calculate the seasonal strength SS = max(0,1-(Var(YR) / Var(YS+YR))), Var represents the variance calculation formula; if the seasonal strength SS is greater than the preset threshold, the used capacity series of 30 days, 7 days or 24 hours is marked as seasonal, otherwise it is not seasonal, and s is 0; When the used capacity series of 30 days, 7 days or 24 hours is marked as seasonal, the ADF test is performed on the used capacity series of 30 days, 7 days and 24 hours; if it is not stationary, the used capacity series of 30 days, 7 days and 24 hours is differentiated once or twice to obtain Ydiff; seasonal differentiation is performed to obtain the sequence Y(d1,s)=Ydiff t -Ydiff t-s Repeat the ADF test and difference order adjustment to obtain the corresponding new used capacity sequence. If a stable sequence can be obtained, the stable sequence is expressed as Y d1,s , where s is the cycle length and d1 is the difference order; the seasonal autoregressive integrated moving average model SARIMA is used. The SARIMA model is expressed as SARIMA(p,d1,q)(P,D,Q)s, where the non-seasonal part parameters of the model parameters are: p is the autoregressive order, d1 is the difference order, and q is the moving average order; the seasonal part parameters are: P is the autoregressive order, D is the difference order, Q is the moving average order, and s is the cycle length; The data is selected from the stationary time series data Y after difference d1,s , the grid search method is used to determine the remaining SARIMA parameters and obtain the parameter values ​​with the best AIC value; The SARIMA model is fitted using the used capacity sequence of the file system to obtain a trained SARIMA file capacity prediction model; When the used capacity series of 30 days, 7 days or 24 hours is marked as non-seasonal, the ADF test is performed on the used capacity series of 30 days, 7 days and 24 hours; if it is not stable, the used capacity series of 30 days, 7 days and 24 hours is differentiated once or twice to obtain the sequence Y(d2)=Ydiff; repeat the ADF test and difference order adjustment to obtain the corresponding new used capacity series. If a stable series can be obtained, the stable series is expressed as Y d2 , where d2 is the difference order, the ARIMA model is used, and the ARIMA model is expressed as ARIMA(p,d2,q), where the model parameters are: p is the autoregressive order, d2 is the difference order, and q is the moving average order; the data is the stationary time series data Y after difference d2 , using the grid search method, determine the remaining ARIMA parameters and obtain the parameter values ​​with the best AIC value; Use the file system's used capacity sequence to fit the ARIMA model and obtain a trained ARIMA file capacity prediction model; If a stationary sequence cannot be obtained, it directly enters the prediction module.

7. A method for collecting and warning file system capacity according to claim 6, characterized in that: The prediction module is specifically: Use the trained 30-day forecast model to forecast capacity for 1 to 5 days after the current time, and obtain forecast values ​​of multiple 30-day forecast models; Use the trained 7-day forecast model to forecast capacity for 1 to 2 days after the current time, and obtain forecast values ​​of multiple 7-day forecast models; Use the 24-hour prediction model to predict 1 hour after the current time and obtain the prediction values ​​of multiple 24-hour prediction models; And obtain the real-time used capacity; define the current time as T, the forecast value from T to T+1 is the short-term forecast; the forecast value from T+2 to T+5 is the long-term forecast; Set the alarm threshold level: a -i ...a -1 ,a0,a1...a i , the larger i is, the higher the alarm level is; set i>=0 for severe alarm, i<0 for minor alarm, minor warning will send warning notification repeatedly according to the frequency set by the level, severe alarm will send warning notification directly, i is a positive integer; If the short-term forecast or real-time used capacity is greater than a0, an early warning notification is sent directly; If the short-term forecast is less than a0, and the long-term forecast remains stable within 5 days after the current period, the stability means that the corresponding forecast values ​​obtained by the 30-day forecast model, the 7-day forecast model, and the 24-hour forecast model are all less than the set threshold, and the current real-time used capacity is less than a0, and when the alarm threshold level of the short-term forecast is less than or equal to the alarm threshold level of the real-time used capacity, the set frequency is adjusted down by a set value, otherwise the corresponding notification frequency is set according to the short-term forecast being at the alarm threshold level; If the short-term forecast is less than a0, and the long-term forecast does not remain stable within 5 days after the current period, and the instability means that the corresponding forecast values ​​obtained by the 30-day forecast model, the 7-day forecast model, and the 24-hour forecast model are all greater than or equal to the set threshold, and the current real-time used capacity is less than a0, then according to the short-term forecast being at the alarm threshold level, the set frequency is increased by a set value to obtain an adjusted frequency, and an early warning notification is sent according to the adjusted frequency; If there is no predicted value and the current real-time used capacity is less than a0, the alarm notification frequency is set according to the capacity threshold level.

8. The method for collecting and warning the capacity of a file system according to claim 6, characterized in that: The training model module also includes: when the newly collected data is sufficient to calculate the set observation points, the newly collected data is processed as a new training sequence; the 30-day prediction model, the 7-day prediction model and the 24-hour prediction model are refitted using the new training sequence; and the corresponding latest prediction model is obtained.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images